Real-time video semantic segmentation system fusing edge size model and optimization method

Abstract

The invention relates to the technical field of digital image processing, in particular to a real-time video semantic segmentation system and optimization method fusing an edge size model, and the system comprises an edge end lightweight model belonging to terminal equipment, an edge server large model belonging to an edge server, a dynamic frame distribution module and a self-adaptive cache module. An optical flow change rate and scene complexity are analyzed through edge detection and entropy calculation by a dynamic frame distribution module, then a dynamic distribution strategy is output, the dynamic distribution strategy is formed by processing a low-change frame by a small model and processing a key frame by a large model, a high-confidence segmentation result of the large model is cached by a self-adaptive cache module, and a high-confidence segmentation result of the large model is obtained. The method can be used for reuse of small models in similar scenes so as to reduce calling frequency of large models, flexibly deploy resources of the large models and the small models during operation and solve the real-time performance-precision contradiction in edge computing scenes, and compared with a single model scheme, the method has the advantages that precision is improved, and load is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a real-time video semantic segmentation system and an optimization method for integrating an edge size model. Background Art

[0002] In existing real-time video semantic segmentation systems, although the use of lightweight models can achieve low latency, it is limited by the model capacity, and the segmentation accuracy in complex scenes is significantly reduced. Uploading the entire video stream to a large model for processing, although it can ensure high accuracy, is limited by latency and cannot meet real-time requirements.

[0003] Although the existing cascade model solution can allocate computing tasks, it still has the problem of being unable to adaptively adjust according to dynamic changes in the scene. Excessive calling of large models in static scenes causes waste of computing power and increased energy consumption. In dynamic and complex scenes, large model calls are delayed, and key frame processing delays cause segmentation errors to spread.

[0004] To this end, a real-time video semantic segmentation system and optimization method integrating edge size model are proposed. Summary of the Invention

[0005] In order to overcome the shortcomings of the prior art, the present invention provides a real-time video semantic segmentation system and an optimization method integrating an edge size model.

[0006] To solve the above technical problems, the present invention provides the following technical solutions: a real-time video semantic segmentation system integrating edge and large models, comprising an edge-end lightweight model deployed on a terminal device and an edge server large model deployed on an edge server, wherein the terminal device is used to receive video frames, the edge-end lightweight model adopts a lightweight architecture of MobileNetV3+DeepLabv3 for single-frame real-time inference, and the edge server large model adopts a high-precision architecture of Swin Transformer+Mask2Former for processing key frames and error-corrected frames, providing high-precision semantic segmentation results, and also includes a dynamic frame allocation module and an adaptive cache module. The video frames received by the terminal device are input into the dynamic frame allocation module. The dynamic frame allocation module analyzes the optical flow change rate and scene complexity through edge detection+entropy calculation, and then outputs a dynamic allocation strategy. The dynamic allocation strategy consists of a small model processing low-change frames and a large model processing key frames. The adaptive cache module caches the high-confidence segmentation results of the large model for reuse by the small model in similar scenarios to reduce the calling frequency of the large model.

[0007] The optimization method for real-time video semantic segmentation that integrates edge-to-edge large and small models includes asymmetric distillation: the large model on the edge server filters low-frequency noise through frequency domain decomposition, transfers high-frequency features to the lightweight model on the edge, distills the lightweight model on the edge, transfers real-time scene distribution data of the terminal device to the lightweight model on the edge, and reversely distills the large model on the edge server.

[0008] As a preferred technical solution of the present invention, it includes an adaptive caching method: the cache object is a semantic mask with a confidence level higher than a threshold output by a large model of an edge server, and lightweight features of each frame are extracted as a retrieval key; the storage structure includes a hot cache and a cold cache; the scene similarity matching algorithm is as follows: global feature extraction: HSV color histogram + image entropy value (measurement of texture complexity); local feature: block (4×4 grid) extraction of HOG directional gradient histogram; similarity calculation: S_global = 0.6·HSV_similarity + 0.4·entropy difference; where S_global is the global similarity; S_local = Σ(inter-block HOG cosine similarity) / total number of blocks; where S_local is the similarity; S_total = 0 .7·S_global+0.3·S_local; where S_total is the total similarity; the scene similarity matching threshold is: when S_total>0.85, it is judged to be a similar scene, triggering cache reuse; dynamic update: when the large model processes a new frame and the confidence level meets the standard, the features are extracted and added to the cache; old result refresh: if the timeliness of the cache hit item exceeds the threshold, the large model is forced to review and update; elimination mechanism: for cache items that have not been reused for a long time, its confidence weight is gradually reduced; if the cache hit rate of N consecutive frames is less than 20%, the cold cache is cleared and the strategy is reset; while reusing the cache results, the background asynchronously calls the edge server large model to process the current frame. If the difference between the asynchronous result and the cache exceeds the threshold, the cache strategy for subsequent frames is immediately corrected.

[0009] As an optimal technical solution of the present invention, it includes dynamic weight fusion: the weight result output of the fusion edge server large model and the edge lightweight model, the output formula is as follows: Output = α·SmallModel(X)+(1-α)·LargeModel(X); wherein SmallModel(X) is the output of the edge lightweight model, which has the characteristics of low precision and low latency; LargeModel(X) is the output of the edge server large model, which has the characteristics of high precision and high latency; a is a dynamic weight, and the value range is [0, 1]. When α=1, it completely relies on the edge lightweight model, and gives priority to real-time performance when resources are tight. When α=0, it completely relies on the edge server large model, which fits the high-precision requirement scenario. When 0<α<1, a mixed result is output to balance accuracy and speed. By adjusting the contribution ratio of the small model and the large model in real time, the real-time and accuracy requirements can be met on terminal devices with limited computing resources. The dynamic adjustment decision of the α value is jointly decided by multiple factors such as computing power reserve, battery status, optical flow change rate-scene complexity and network delay.

[0010] As an optimal technical solution of the present invention, it includes cross-frame propagation: constructing a spatiotemporal graph for consecutive frames, and propagating the processing results of the large model to adjacent frames through a graph neural network, including motion-aware interpolation: combining optical flow estimation and semantic segmentation results to achieve accuracy compensation between small model frames.

[0011] As an optimal technical solution of the present invention, it includes using hardware sensors for status monitoring, reading temperature, remaining power and computing power occupancy, calculating the mIoU and FLOPs of the current frame in software, and constructing a triangular optimization model of device status-model accuracy-energy consumption, the formula is as follows: min E = f(Accuracy) + g(Compute) + h(Thermal); where min E is the total loss; f(Accuracy) is the accuracy loss function; g(Compute) is the computing resource consumption function; h(Thermal) is the thermodynamic loss function; by adjusting the system parameters, min E is minimized, and weights w1, w2 and w3 are introduced, the formula is as follows: minE = w1·f(Accuracy)+w2·g(Compute)+w3·h(Thermal); the weights w1, w2 and w3 are dynamically adjusted according to the scenario.

[0012] Compared with the prior art, the present invention has the following beneficial effects:

[0013] 1. This application dynamically allocates computing tasks by integrating the large model of the edge server and the lightweight model of the edge end, flexibly allocates large and small model resources at runtime, and solves the "real-time-accuracy" contradiction in the edge computing scenario. Compared with the single model solution, the accuracy is improved and the load is reduced.

[0014] 2. This application constructs a core decision model for an adaptive edge computing system by jointly optimizing accuracy, computing power, and temperature, breaking through the limitations of traditional single-objective optimization. It directly links the physical device status and algorithm strategy, and can adjust the optimization weight according to real-time scenario requirements.

[0015] 3. The adaptive caching mechanism of this application significantly reduces resource consumption in edge computing scenarios through scenario feature analysis, dynamic update strategy and collaborative optimization, while ensuring result reliability through intelligent reuse. DETAILED DESCRIPTION

[0016] In order to make the technical means, creative features, purpose and efficacy of the present invention easy to understand, the present invention is further described below in conjunction with specific examples, but the following examples are only preferred embodiments of the present invention, not all. Based on the examples in the embodiments, other embodiments obtained by those skilled in the art without making creative work are all within the scope of protection of the present invention. The experimental methods in the following examples, unless otherwise specified, are all conventional methods.

[0017] Embodiment: A real-time video semantic segmentation system that integrates edge-size models includes an edge-end lightweight model deployed on a terminal device and an edge server large model deployed on an edge server. The terminal device includes a camera, and the terminal device is used to receive video frames. The edge-end lightweight model adopts a lightweight architecture of MobileNetV3+DeepLabv3 for single-frame real-time inference, and the edge server large model adopts a high-precision architecture of Swin Transformer+Mask2Former for processing key frames and error-corrected frames to provide high-precision semantic segmentation results.

[0018] It also includes a dynamic frame allocation module and an adaptive caching module. The video frames received by the terminal device are input into the dynamic frame allocation module. The dynamic frame allocation module analyzes the optical flow change rate and scene complexity through edge detection + entropy calculation, and then outputs a dynamic allocation strategy. The dynamic allocation strategy consists of a small model processing low-change frames and a large model processing key frames. The adaptive caching module caches the high-confidence segmentation results of the large model for the small model to reuse in similar scenarios to reduce the calling frequency of the large model.

[0019] The optimization method for real-time video semantic segmentation that integrates edge-to-edge large and small models includes asymmetric distillation: the large model on the edge server filters low-frequency noise through frequency domain decomposition, transfers high-frequency features to the lightweight model on the edge, distills the lightweight model on the edge, transfers real-time scene distribution data of the terminal device to the lightweight model on the edge, and reversely distills the large model on the edge server.

[0020] Adaptive caching method: The cached objects are semantic masks with confidence levels above a threshold output by the large model on the edge server. Lightweight features of each frame are extracted as retrieval keys. The storage structure includes a hot cache (which stores frequently accessed results in memory for high-speed response) and a cold cache (which transfers low-frequency results to the edge server's SSD for fast access via memory mapping). The scene similarity matching algorithm is as follows:

[0021] Global feature extraction: HSV color histogram + image entropy (measure of texture complexity).

[0022] Local features: Extract HOG oriented gradient histograms in blocks (4×4 grids).

[0023] Similarity calculation: S_global = 0.6·HSV_similarity + 0.4·entropy difference, where S_global is the global similarity.

[0024] S_local=Σ(inter-block HOG cosine similarity) / total number of blocks, where S_local is the similarity.

[0025] S_total = 0.7·S_global + 0.3·S_local, where S_total is the total similarity.

[0026] The scene similarity matching threshold is: when S_total>0.85, it is determined to be a similar scene, triggering cache reuse.

[0027] Dynamic update: When the large model processes a new frame and the confidence level meets the requirements, features are extracted and added to the cache; refresh of old results: if the timeliness of the cache hit item exceeds the threshold, the large model is forced to be reviewed and updated; elimination mechanism: for cache items that have not been reused for a long time, their confidence weight is gradually reduced; if the cache hit rate for N consecutive frames is less than 20%, the cold cache is cleared and the strategy is reset; while reusing the cached results, the background asynchronously calls the edge server large model to process the current frame. If the difference between the asynchronous result and the cache exceeds the threshold, the cache strategy for subsequent frames is immediately corrected.

[0028] Dynamic weight fusion: The weighted results of the fusion edge server large model and the edge lightweight model are output. The output formula is as follows:

[0029] Output=α·SmallModel(X)+(1-α)·LargeModel(X).

[0030] SmallModel(X) is the output of the lightweight model at the edge, which has the characteristics of low precision and low latency; LargeModel(X) is the output of the large model at the edge server, which has the characteristics of high precision and high latency; a is a dynamic weight with a value range of [0, 1].

[0031] When α=1, the system completely relies on the lightweight model on the edge, giving priority to ensuring real-time performance when resources are tight. When α=0, the system completely relies on the large model on the edge server, which is suitable for scenarios with high precision requirements. When 0<α<1, a mixed result is output, balancing precision and speed. By adjusting the contribution ratio of the small model to the large model in real time, the requirements of real-time performance and precision are met on terminal devices with limited computing resources. The dynamic adjustment decision of α is jointly determined by multiple factors such as computing power reserve, battery status, optical flow change rate-scene complexity, and network latency, as shown in the following table:

[0032] Decision Factors Logic of impact on α Hashrate margin High market share - α↑ Battery status The battery level is lower than the preset threshold -α↑ Optical flow change rate-scene complexity High complexity - α↓ Network latency Communication delay>50ms-α↑

[0033] In a specific embodiment of the smart traffic scenario, under normal road conditions (α=0.8), the lightweight model at the edge quickly segments the lane lines (low latency), and the large model on the edge server fine-tunes once every fixed number of frames. When there is a sudden obstacle (α=0.2), the optical flow detects a drastic change, which reduces the value of α. The large model on the edge server increases the frequency of intervention and corrects the obstacle boundary (high precision).

[0034] Cross-frame propagation: Construct a spatiotemporal graph for consecutive frames and propagate the processing results of the large model to adjacent frames through graph neural networks.

[0035] Motion-aware interpolation: Combines optical flow estimation and semantic segmentation results to achieve accuracy compensation between small model frames.

[0036] Hardware sensors are used for status monitoring to read temperature, remaining power, and computing power utilization. The software calculates the mIoU and FLOPs of the current frame and constructs a triangular optimization model of device status, model accuracy, and energy consumption. The formula is as follows:

[0037] min E=f(Accuracy)+g(Compute)+h(Thermal).

[0038] Where min E is the total loss; f(Accuracy) is the accuracy loss function, which quantifies the difference between the model output and the true label. For example, in the semantic segmentation task, f(Accuracy) = 1-mIoU, which calculates the intersection-over-union loss; g(Compute) is the computing resource consumption function, which quantifies the cost of computing resources. For example, in the energy consumption cost, g(Compute) = μW; h(Thermal) is the thermodynamic loss function, which quantifies the impact of device heating on system stability, such as h(Thermal) = κ·(T_current (current temperature) - T_safe (safe temperature threshold))^2.

[0039] By adjusting the system parameters to minimize min E, weights w1, w2, and w3 are introduced, and the formula is as follows:

[0040] min E=w1·f(Accuracy)+w2·g(Compute)+w3·h(Thermal).

[0041] The weights w1, w2, and w3 are dynamically adjusted according to the scenario. For example, in high-precision mode, w1↑, w2↓, w3↓; in high-temperature protection mode, w1↓, w2↓, w3↑.

[0042] Dynamically respond to status data for real-time control, switching to a low-bit quantization model at high temperatures, dominated by h (Thermal), and increasing w3; shutting down the edge server large model when power is low, and only using the edge lightweight model, dominated by g (Compute), and increasing w2; when a key object is detected, temporarily increasing the frequency of calling the large model, dominated by f (Accuracy), and increasing w1.

[0043] The above is a detailed description of the embodiments of the present invention, but the present invention is not limited thereto. Various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present invention.

Claims

1. A real-time video semantic segmentation system integrating edge size model, characterized by: It includes a lightweight edge model deployed on the terminal device and a large edge server model deployed on the edge server. The terminal device is used to receive video frames. The lightweight edge model adopts the lightweight architecture of MobileNetV3+DeepLabv3 for single-frame real-time inference. The large edge server model adopts the high-precision architecture of Swin Transformer+Mask2Former for processing key frames and error correction frames, providing high-precision semantic segmentation results.

2. The real-time video semantic segmentation system integrating edge size model according to claim 1, characterized in that: It also includes a dynamic frame allocation module and an adaptive caching module. The video frames received by the terminal device are input into the dynamic frame allocation module. The dynamic frame allocation module analyzes the optical flow change rate and scene complexity through edge detection + entropy calculation, and then outputs a dynamic allocation strategy. The dynamic allocation strategy consists of a small model processing low-change frames and a large model processing key frames. The adaptive caching module caches the high-confidence segmentation results of the large model for the small model to reuse in similar scenarios to reduce the calling frequency of the large model.

3. An optimization method for real-time video semantic segmentation integrating edge size model, characterized in that: Including asymmetric distillation: the edge server large model filters low-frequency noise through frequency domain decomposition, transmits high-frequency features to the edge lightweight model, distills the edge lightweight model, transmits real-time scene distribution data of the terminal device to the edge lightweight model, and reverse distills the edge server large model.

4. The optimization method for real-time video semantic segmentation integrating edge size model according to claim 3, characterized in that: This includes an adaptive caching method: the cached objects are semantic masks with confidence levels above a threshold output by the large model on the edge server, and lightweight features of each frame are extracted as retrieval keys; The storage structure includes hot cache and cold cache; The scene similarity matching algorithm is as follows: Global feature extraction: HSV color histogram + image entropy (measure of texture complexity); Local features: Extract HOG directional gradient histogram in blocks (4×4 grid); Similarity calculation: S_global = 0.6·HSV_similarity + 0.4·entropy difference; Where S_global is the global similarity; S_local = Σ(inter-block HOG cosine similarity) / total number of blocks; Among them, S_local is the similarity; S_total=0.7·S_global+0.3·S_local; Where S_total is the total similarity; The scene similarity matching threshold is: when S_total>0.85, it is determined to be a similar scene, triggering cache reuse; Dynamic update: When the large model processes a new frame and the confidence level meets the requirements, features are extracted and added to the cache; Refresh old results: If the timeliness of the cache hit item exceeds the threshold, the large model will be forced to review and update; Elimination mechanism: Gradually reduce the confidence weight of cache items that have not been reused for a long time; If the cache hit rate for N consecutive frames is less than 20%, clear the cold cache and reset the strategy; While reusing the cached results, the background asynchronously calls the edge server large model to process the current frame. If the difference between the asynchronous result and the cache exceeds the threshold, the caching strategy for subsequent frames is immediately corrected.

5. The optimization method for real-time video semantic segmentation integrating edge size model according to claim 4, characterized in that: Including dynamic weight fusion: the weight result output of the fusion edge server large model and the edge lightweight model is as follows: Output=α·SmallModel(X)+(1-α)·LargeModel(X); SmallModel(X) is the output of the lightweight model at the edge, which has the characteristics of low precision and low latency; LargeModel(X) is the output of the large model at the edge server, which has the characteristics of high precision and high latency; a is a dynamic weight with a value range of [0, 1]. When α=1, it completely relies on the lightweight model on the edge, giving priority to ensuring real-time performance when resources are tight. When α=0, it completely relies on the large model on the edge server, which is suitable for scenarios with high precision requirements. When 0<α<1, it outputs mixed results, balancing accuracy and speed. The dynamic adjustment decision of α is jointly determined by multiple factors such as computing power reserve, battery status, optical flow change rate-scene complexity and network latency.

6. The optimization method for real-time video semantic segmentation integrating edge size model according to claim 5, characterized in that: Including cross-frame propagation: constructing a spatiotemporal graph for consecutive frames, and propagating the processing results of the large model to adjacent frames through the graph neural network.

7. The optimization method for real-time video semantic segmentation integrating edge size model according to claim 6, characterized in that: Including motion-aware interpolation: combining optical flow estimation and semantic segmentation results to achieve accuracy compensation between small model frames.

8. The optimization method for real-time video semantic segmentation integrating edge size model according to claim 7, characterized in that: This involves using hardware sensors for status monitoring, reading temperature, remaining power, and computing power utilization. Software calculates the mIoU and FLOPs of the current frame, and constructs a triangular optimization model of device status, model accuracy, and energy consumption. The formula is as follows: min E=f(Accuracy)+g(Compute)+h(Thermal); Where min E is the total loss; f (Accuracy) is the accuracy loss function; g (Compute) is the computing resource consumption function; h (Thermal) is the thermodynamic loss function; By adjusting the system parameters to minimize min E, weights w1, w2, and w3 are introduced, and the formula is as follows: min E=w1·f(Accuracy)+w2·g(Compute)+w3·h(Thermal); The weights w1, w2, and w3 are dynamically adjusted according to the scenario.