A real-time video edge analysis method of a distributed vision controller

Abstract

The application discloses a kind of real-time video edge analysis methods of distributed visual controller, including, obtaining real-time video stream, and according to current computing resource and network state dynamic determination split strategy of deep neural network;Utilize the feature extraction part executed locally to video frame operation, and generate intermediate layer feature data;By combining pre-constructed channel sensitivity model and the feature activation intensity of current frame, the semantic importance parameter of each feature channel is determined;Based on the importance parameter, the semantic compression processing including channel screening and dynamic quantization is executed to feature data, and compressed feature data packet is generated;Finally, the data packet is sent to collaborative computing node to complete the final video analysis.The application can realize the adaptive allocation of end-edge computing power and intelligent data compression for machine vision tasks, while significantly reducing network bandwidth occupation, effectively guaranteeing the real-time performance and high precision of analysis task, improving the operation efficiency and robustness of the entire distributed system.

Description

Technical Field

[0001] This invention relates to the field of distributed computing and visual data processing technology, and in particular to a real-time video edge analysis method for a distributed visual controller. Background Technology

[0002] With the deep integration of the Internet of Things (IoT) and artificial intelligence (AI) technologies, distributed vision systems have been widely applied in smart cities, industrial quality inspection, and autonomous driving. In these scenarios, deep neural networks (DNNs) have become the mainstream algorithm for video analysis due to their superior feature extraction and pattern recognition capabilities. Traditional processing methods typically involve transmitting all collected video data back to a cloud server for centralized inference, or running lightweight models directly on edge devices. However, with the increase in video resolution (e.g., 4K / 8K) and the growing complexity of neural network models (e.g., ResNet, Transformer variants), single computing methods are no longer sufficient to meet real-time requirements. Cloud processing is limited by network bandwidth fluctuations, easily leading to uncontrollable transmission delays, and the transmission of massive amounts of video data incurs huge bandwidth costs. Meanwhile, edge devices (such as vision controllers) are limited by power consumption and chip computing power, making it difficult to independently handle high-precision, complex, large-scale model inference, resulting in decreased recognition accuracy.

[0003] To address the aforementioned computing power and bandwidth bottlenecks between the edge and cloud, current research has employed collaborative inference architectures based on intermediate layer feature transmission (such as the YOLOv8 object detection model). However, this collaborative inference architecture still has some shortcomings in application. First, most solutions typically select fixed model splitting points, failing to dynamically adjust the allocation strategy of computing tasks based on real-time fluctuations in network bandwidth and edge computing load, resulting in poor robustness of visual surveillance systems. Second, in the data transmission stage, existing technologies often directly transmit high-dimensional feature tensors or use traditional video coding standards optimized for the Human Visual System (HVS) (such as H.264 / AVC, HEVC / H.265, AV1, etc.) to compress feature maps. The former method still results in a massive amount of data, failing to effectively alleviate bandwidth pressure. The latter method introduces irreversible compression artifacts, destroying high-frequency edge features and texture details crucial for machine vision inference, easily leading to a significant decline in final inference accuracy. Summary of the Invention

[0004] In this section, as well as in the abstract and title of this application, some simplifications or omissions may be made to avoid obscuring the purpose of this section, the abstract, and the title of this application, and such simplifications or omissions shall not be used to limit the scope of the invention.

[0005] In view of the aforementioned existing problems, this invention is proposed. Therefore, this invention provides a real-time video edge analysis method using a distributed vision controller to solve the problems mentioned in the background art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: a real-time video edge analysis method for a distributed vision controller, the method being applied in a distributed vision controller, comprising:

[0007] The system acquires real-time video streams and, based on the current computing resource status and network transmission status, determines a network layer segmentation strategy for the deep neural network model. The segmentation strategy divides the model logic into a feature extraction part executed locally and an inference part executed by the collaborating end.

[0008] The feature extraction part is used to perform calculations on the current video frame to generate intermediate layer feature data;

[0009] The semantic importance parameter of each feature channel in the intermediate layer feature data is determined, and the semantic importance parameter characterizes the degree of contribution of the feature channel to the model inference result;

[0010] Based on the semantic importance parameter, semantic compression processing is performed on the intermediate layer feature data to generate a compressed feature data package. The semantic compression processing includes channel filtering based on importance and dynamic quantization operations.

[0011] The compressed feature data packet is sent to the collaborative computing node, so that the collaborative computing node can reconstruct the features of the compressed feature data packet and use the inference part to complete the video analysis.

[0012] As a preferred embodiment of the real-time video edge analysis method for the distributed vision controller described in this invention, wherein: the determination of the network layer segmentation strategy for the deep neural network model includes:

[0013] Calculate a comprehensive evaluation of the local computation cost and data transmission cost at multiple candidate segmentation locations;

[0014] Based on the comprehensive evaluation value, the candidate segmentation position that minimizes the overall system latency or energy consumption is selected as the current network layer segmentation point.

[0015] As a preferred embodiment of the real-time video edge analysis method for the distributed vision controller described in this invention, wherein: determining the semantic importance parameters of each feature channel in the intermediate layer feature data includes:

[0016] Obtain a pre-built channel sensitivity model, which records the correlation between feature channels and model loss function;

[0017] By combining the feature activation intensity of the current video frame with the channel sensitivity model, the semantic importance parameter of each feature channel is calculated.

[0018] As a preferred embodiment of the real-time video edge analysis method for the distributed vision controller described in this invention, the step of performing semantic compression processing on the intermediate layer feature data based on the semantic importance parameter includes:

[0019] The feature channels are sorted or classified according to the semantic importance parameter.

[0020] Retain feature channel data that meets the preset importance conditions, and generate mask information to indicate the location of the retained channels;

[0021] Feature channel data that does not meet the preset importance conditions are filtered out.

[0022] As a preferred embodiment of the real-time video edge analysis method of the distributed vision controller described in this invention, it further includes:

[0023] Establish a mapping relationship between semantic importance parameters and quantization accuracy;

[0024] Assign a first number of quantization bits to the feature channel of the first semantic importance parameter, and assign a second number of quantization bits to the feature channel of the second semantic importance parameter;

[0025] The floating-point data of the feature channel is mapped to discretized data using the allocated number of quantization bits.

[0026] As a preferred embodiment of the real-time video edge analysis method of the distributed vision controller described in this invention, the mapping relationship adopts nonlinear mapping logic to adapt to the long-tail distribution characteristics of the feature data of the intermediate layer of the deep neural network, and the compressed feature data packet includes a dynamic scaling factor for reverse data recovery.

[0027] As a preferred embodiment of the real-time video edge analysis method for the distributed vision controller described in this invention, the generation of compressed feature data packets includes:

[0028] Calculate the residual information of the feature data of the current frame relative to the feature data of historical frames;

[0029] The residual information is losslessly compressed and encoded to generate a compressed bitstream;

[0030] The compressed bitstream and auxiliary metadata used for feature reconstruction are encapsulated into a compressed feature data packet.

[0031] As a preferred embodiment of the real-time video edge analysis method of the distributed vision controller described in this invention, the method further includes a step of receiving feedback information from collaborative computing nodes and updating the compression strategy, the step comprising:

[0032] Receive parameter update instructions sent by collaborative computing nodes, wherein the parameter update instructions are calculated by the collaborative computing nodes based on the differences between the full feature inference results and the compressed feature inference results;

[0033] The weights or model parameters used to calculate semantic importance parameters are adjusted according to the parameter update instructions, and the semantic evaluation logic of the frame is corrected.

[0034] As a preferred embodiment of the real-time video edge analysis method of the distributed vision controller described in this invention, the feedback information is generated based on the gradient backpropagation mechanism and is used to dynamically calibrate the sensitivity distribution of feature channels under different scenarios.

[0035] As a preferred embodiment of the real-time video edge analysis method of the distributed vision controller described in this invention, the method further includes a multi-machine collaborative data alignment step, which includes:

[0036] When the distributed vision controller detects an area of ​​overlap with the field of view of a neighboring device, spatial transformation parameters are added to the compressed feature data packet.

[0037] The spatial transformation parameters are used to instruct the collaborative computing nodes to map the current feature data to a unified coordinate space for multi-view feature fusion.

[0038] Compared with existing technologies, the beneficial effects of this solution are:

[0039] 1. By dynamically determining candidate split points for the neural network model in real time by sensing local computing resources and network transmission status, this technology solves the problem of fixed split points being unable to adapt to dynamic environmental changes and exhibiting poor robustness in existing technologies. It enables adaptive load balancing between edge and collaborative ends, favoring local computation during network congestion and offloading resources when local computing power is strained. This ensures minimal end-to-end latency for the entire system under varying operating conditions, improving the real-time performance and reliability of analysis tasks.

[0040] 2. The contribution of a channel to the final task is evaluated by combining its static sensitivity and dynamic activation intensity, and channel selection and dynamic quantization are performed based on this evaluation. This invention abandons the traditional video coding approach that optimizes for human vision, instead preserving the semantic information most crucial for machine reasoning. This achieves a significantly higher compression rate and reduced bandwidth usage while effectively avoiding the destruction of key features, ensuring high accuracy in collaborative reasoning, and resolving the contradiction between accuracy and bandwidth in the feature compression stage of existing technologies.

[0041] 3. Furthermore, by receiving gradient feedback generated by collaborative computing nodes based on accuracy differences, this invention can adjust and optimize its semantic evaluation logic online, enabling the compression strategy to continuously adapt to specific scenarios and achieve autonomous performance improvement. Simultaneously, by embedding spatial transformation parameters in the data packets, it supports multi-viewpoint feature fusion at the collaborative end, breaking the limitations of single-device field of view and enabling advanced functions such as cross-camera tracking, thus enhancing the overall intelligence level and application scope of the distributed vision system. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0043] Figure 1 This is a flowchart illustrating the overall process of a real-time video edge analysis method using a distributed vision controller according to an embodiment of the present invention. Detailed Implementation

[0044] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0045] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0046] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0047] This invention is described in detail with reference to the schematic diagrams. When detailing the embodiments of this invention, for ease of explanation, the cross-sectional views illustrating the device structure may be partially enlarged, not adhering to the usual scale. Furthermore, the schematic diagrams are merely examples and should not be construed as limiting the scope of protection of this invention. In actual fabrication, the three-dimensional spatial dimensions of length, width, and depth should be included.

[0048] Furthermore, in the description of this invention, it should be noted that the terms "upper," "lower," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. These terms are used solely for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In addition, the terms "first," "second," or "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0049] Unless otherwise explicitly specified and limited, the terms "installation," "connection," and "joining" in this invention should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; similarly, they can refer to mechanical connections, electrical connections, or direct connections, or indirect connections through an intermediate medium, or internal connections between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0050] Example 1

[0051] Reference Figure 1 This is the first embodiment of the present invention, which provides a real-time video edge analysis method for a distributed vision controller. The method is applied in a distributed vision controller and includes:

[0052] S1. Acquire real-time video stream and, based on the current computing resource status and network transmission status, determine the network layer segmentation strategy of the deep neural network model. The segmentation strategy divides the model logic into a feature extraction part executed locally and an inference part executed by the collaborating end.

[0053] It should be noted that the purpose of this step is to achieve dynamic task partitioning between the edge devices (distributed vision controllers, which refer to edge computing devices deployed at the front end of data acquisition, including but not limited to: smart cameras with built-in processing units (such as SoCs), edge computing gateways connected to one or more network cameras, or industrial control computers) and collaborative computing nodes (collaborative computing nodes refer to computing units with stronger computing power that are connected to the distributed vision controller via a network, including but not limited to: edge servers, central cloud servers, or a group of other edge devices with stronger computing power). This is to adapt to the constantly changing load and network conditions of the distributed vision system, thereby ensuring low latency and high stability of the entire system.

[0054] Furthermore, before determining the segmentation strategy, the distributed vision controller first needs to perform a precise and real-time quantitative assessment of its own operating status and the external network environment. In this embodiment, the quantitative assessment of computing resource status and network transmission status is mainly considered.

[0055] Furthermore, for quantitative assessment of computing resource status, a distributed vision controller periodically monitors the real-time status of local processing units (such as CPU, GPU, or NPU). Key monitored metrics include:

[0056] Current computing power level: Measured in floating-point operations per second (FLOPS) or inference frames per second (FPS) for a specific model. It is important to note that this computing power level is not a fixed theoretical peak value, but rather changes dynamically based on factors such as the current load on the device (e.g., whether other processes are running) and temperature (whether throttling is occurring).

[0057] Memory usage and available bandwidth: Monitor RAM and video memory usage to ensure that there is enough space for intermediate layer feature data to be stored and processed.

[0058] Furthermore, for quantitative evaluation of network transmission status, the distributed vision controller assesses the network connection quality with collaborative computing nodes through active probing (such as sending ping packets or small data packets) or passive monitoring (such as analyzing TCP connection statistics). Correspondingly, key indicators for probing or monitoring include:

[0059] Effective network bandwidth: measured in bits per second (bps), representing the currently available data transmission rate.

[0060] Network round-trip delay: measured in seconds (s), it represents the round-trip time for a data packet to be sent from the local end to the cooperating end and back. Usually, half of this delay is taken as an estimate of one-way transmission delay.

[0061] It should be noted that deep neural network (DNN) models, such as ResNet and YOLOv8, are composed of a series of network layers (such as convolutional layers, pooling layers, and activation layers). Not every two layers is suitable as a split point. Therefore, the present invention predefines a set of candidate split points within the architecture of the model. These candidate split points are typically selected after key structural units of the model, such as the end of each residual block or after a downsampling layer.

[0062] Furthermore, for each candidate split point The system needs to evaluate the overall evaluation value resulting from splitting the model at this point. Specifically, this overall evaluation value mainly consists of local computation cost and data transmission cost. To achieve the goal of real-time video edge analysis, we usually quantify the "cost" as "time latency". Therefore, for any candidate split point... Its total delay cost This can be modeled as the sum of local computation latency and data transmission latency:

[0063]

[0064] in, It performs the process from the input layer to the candidate split point locally. The computational latency required for this part (i.e., the feature extraction part). It is to select candidate split points The network latency required for the output intermediate layer feature data to be transmitted to the collaborative computing nodes.

[0065] Furthermore, to make the model workable, we need to specify the total delay cost formula mentioned above.

[0066] Specifically, regarding local computing latency The computation time depends primarily on the time from the start of the model to the candidate split point. Required computation (in FLOPs) and the current actual computing power of the local device (in FLOPS). Based on this, the computational delay is... It can be represented as:

[0067]

[0068] It needs to be emphasized that here It is a pre-analyzed static value that is uniquely related to the model structure and the location of the candidate split point. It is a dynamically perceived variable. When the local device load is high, it affects the actual computing power. The latency of performing the same computational task during descent It will increase.

[0069] Specifically, regarding data transmission network latency The calculation involves the transmission delay, which comprises the time consumed by serialization and data transmission itself, as well as inherent network latency. This transmission delay depends on the time available at the candidate segmentation point. Size of the generated intermediate layer feature data (in bits) Current network effective bandwidth (in bps) and network latency Based on this, the data transmission network latency... It can be represented as:

[0070]

[0071] Among them, with the above Similarly, Also with candidate split points The relevant static value depends on the resolution, number of channels, and data type of the feature map output by the current layer of the model. and This is a dynamically perceived variable. When network congestion leads to bandwidth... Decline or delay As the network latency increases, the cost of data transmission will rise sharply.

[0072] Furthermore, considering both local computation latency and data transmission network latency, for each candidate split point... Its total delay cost function can be defined as:

[0073]

[0074] Specifically, in this formula, when a shallower candidate split point is selected ( (Forefront) Smaller size, faster local computation. However, the feature maps at this stage are typically high-resolution and high-dimensional, leading to... The data transmission network latency is high, resulting in significant overhead. When selecting a deeper candidate segmentation point ( (Later), after multiple convolutions and pooling, the feature map size The latency has been significantly reduced, resulting in low data transmission network latency costs. However, at this point... It is relatively large, requires high local computing power, and has a long computation latency.

[0075] Furthermore, the objective of this invention is to find an optimal candidate segmentation point. This makes the total delay cost Minimize. Therefore, this objective process can be represented as solving an optimization problem:

[0076]

[0077] Specifically, the process of solving this optimization problem is as follows: the distributed vision controller traverses all candidate split points in each decision cycle (e.g., every few seconds or when a significant change in the system state is detected). Substitute real-time perception and , Value, calculate the value for each point. and choose to make smallest As the segmentation point for the current frame or the next set of frames.

[0078] It should be noted that, by solving the optimization problem described above, this invention addresses the poor robustness of traditional fixed-point splitting methods that cannot adapt to environmental changes. When the network condition is good ( High) while local computing resources are scarce ( When the network conditions are low, by solving this optimization problem, the distributed vision controller will automatically select a shallower candidate segmentation point, offloading more computational tasks to the collaborating end, thereby avoiding local computational bottlenecks. Conversely, when network conditions are poor (…), When local computing power is sufficient (but low latency), the distributed vision controller tends to select a deeper candidate segmentation point and perform more computation locally to reduce reliance on high-latency, low-bandwidth networks, thereby minimizing latency caused by data transmission. This adaptive edge-cloud computing power allocation ensures that the overall end-to-end latency of the distributed vision system remains low under various complex and dynamically changing operating conditions, improving the real-time performance and reliability of distributed vision analysis tasks. In addition, this… It can also be replaced or extended to an energy consumption model, thereby achieving optimization with the goal of minimizing the energy consumption of distributed vision systems, to support battery-powered edge devices.

[0079] S2. The feature extraction part is used to perform calculations on the current video frame to generate intermediate layer feature data;

[0080] Furthermore, the distributed vision controller captures the current video frame to be processed from the real-time video stream, denoted as... ,in, Represents a timestamp. Before being fed into the model, the original frame typically undergoes a series of preprocessing operations, such as scaling it from its original resolution to the standard input size required by the model (e.g., ...). or (pixels), and perform normalization (e.g., change the pixel value from...) Range mapping to or (interval), to meet the requirements of the model input layer.

[0081] It should be noted that, since the optimal candidate network layer splitting point has already been determined in step S1, This split point divides the complete deep neural network model into two parts: one part is the feature extraction part executed locally. The other part is the inference part executed by the collaborative end. The core task of this step is to perform the forward propagation computation of the feature extraction part.

[0082] Specifically, the forward propagation calculation process can be represented as follows:

[0083]

[0084] in, It is the current video frame after preprocessing. This represents the distance from the model's input layer to the optimal candidate split point. This subnetwork comprises all the layers up to this point. It includes a series of operations such as convolution, activation functions (e.g., ReLU), pooling, and normalization. The forward propagation computation process essentially involves processing the input video frames. A series of nonlinear transformations were performed. This is the output result obtained from the forward propagation calculation, i.e., the intermediate layer feature data. This data is a high-dimensional tensor, and its specific shape is determined by the candidate split points. The design decisions of the network layer are usually expressed as , This represents the number of feature channels. Each channel can be understood as a specific "feature detector". For example, in shallow networks, some channels may be sensitive to edges, corners, or specific colors; while in deep networks, channels may respond to more complex patterns, such as parts of an object (wheels, faces) or textures. and These represent the height and width of the feature map, respectively. After passing through pooling layers or strided convolutions in a convolutional network, and It is usually smaller than the size of the original input image, thereby compressing the spatial dimension and increasing the receptive field, so that each feature point can "see" a larger area of ​​the original image.

[0085] It should be noted that this step allows a portion of the computationally intensive task to be performed at the edge. Typically, the first few layers of a deep neural network model need to process high-resolution input data, resulting in a massive computational burden. By performing this computation locally, the computing power of edge devices can be effectively utilized, avoiding the direct uploading of massive amounts of raw video data to the network and initially addressing the bandwidth bottleneck issue. In addition, the generated intermediate layer feature data... It is a highly condensed version of the original visual information. Compared to the original... , A large amount of redundant information has been removed. However, it is important to emphasize that, although compared to It is more advanced in semantics, but its data volume is still enormous. For example, the size of a feature tensor output from an intermediate layer might be... If stored as a 32-bit floating-point number, the data size will reach MB. For real-time video streaming (e.g., 30FPS), this would still result in a transmission demand of nearly 100MB / s, which is unsustainable for many wireless network environments (e.g., 4G / 5G or Wi-Fi). Therefore, it is necessary to... Intelligent compression for machine vision tasks, rather than direct transmission or the use of traditional video encoding, is the only way to ultimately achieve low-latency, high-precision collaborative reasoning.

[0086] S3. Determine the semantic importance parameter of each feature channel in the intermediate layer feature data. The semantic importance parameter characterizes the degree of contribution of the feature channel to the model inference result.

[0087] It should be noted that in step S2, we obtained the intermediate layer feature data. However, the data volume remains enormous, and direct transmission would cause unacceptable latency. Unlike traditional video coding methods used in existing technologies that are geared towards the human visual system (HVS), this invention recognizes that the intermediate layer feature data of a neural network ultimately serves machine vision tasks, not human viewing. Therefore, its compression should be based on its "contribution" to the final inference result rather than "visual fidelity." The purpose of this step is to calculate a quantified semantic importance parameter for each feature channel of the intermediate layer feature data, thereby providing guidance for subsequent targeted semantic compression processing.

[0088] Furthermore, the determination of this semantic importance parameter is not based on a single static or dynamic indicator, but rather combines the inherent sensitivity of the channel (static attribute) and the feature activation intensity of the current frame (dynamic attribute).

[0089] Furthermore, a pre-built channel sensitivity model is obtained. It is important to note that not all feature channels are equally important to the final inference task. Some channels may encode crucial discriminative information (such as object outlines or key components), while others may contain redundant or less useful background texture information. This inherent difference in importance is referred to as channel sensitivity in this invention.

[0090] Specifically, this channel sensitivity model This is built during the offline analysis phase before deploying the deep neural network model. In this embodiment, a representative validation dataset is used for each feature channel. ( Sensitivity analysis can be performed. Specifically, the final loss function of the channel sensitivity model can be calculated. Feature map of this channel The expected value of the partial derivatives. This measures how much a small change in the feature value of a channel will affect the final prediction result (i.e., the loss function value) of the deep neural network model. Therefore, its sensitivity parameter... It can be defined as:

[0091]

[0092] in, It is the first Sensitivity parameters for each channel. It is an offline validation dataset. It is an input sample in the validation set. It is the loss function of the neural network (for example, cross-entropy loss for classification tasks, and composite loss including classification and regression for object detection tasks). Input sample The first candidate split point generated Feature map of each channel. It is the gradient of the loss function on the feature map, representing the importance of this channel during backpropagation of information. This indicates taking the norm of the gradient (such as L1 or L2 norm) to obtain a scalar value. This means calculating the expected value over the entire validation set to obtain a stable and universally applicable static sensitivity score.

[0093] It should be noted that, through the above processing, we pre-obtain a static importance vector that represents the "talent" of each feature channel, independent of any specific input. .for The larger the channel value, the greater its impact on the final performance of the deep neural network model, making it a key "information hub" in the network.

[0094] Furthermore, even if a channel has high inherent sensitivity (e.g., it is an excellent "face" detector), if the current video frame... If no human face appears in the frame, the activation value of that channel will be very low, and its contribution to the analysis of the current frame will be negligible. Therefore, its immediate importance must be assessed in conjunction with real-time, content-related dynamic information.

[0095] Specifically, for the intermediate layer feature data generated in step S2... We calculate each feature channel Feature activation intensity This process can be achieved by aggregating the values ​​at all spatial locations of the channel feature map. In this embodiment, a robust aggregation method is used to calculate its L2 norm to simultaneously reflect the magnitude and breadth of the numerical values:

[0096]

[0097] in, Is the current frame in the 1st position? Feature activation intensity on each channel. It is the first The spatial location of each channel feature map The activation value at that location.

[0098] It should be noted that, It intuitively reflects the first The degree to which a feature detector is "lit up" in the current input frame. A high activation intensity means that the pattern corresponding to that channel can be significantly detected in the current frame.

[0099] Furthermore, by combining the static sensitivity (sensitivity parameter) of a channel with its dynamic activation intensity (feature activation intensity), a comprehensive semantic importance parameter for each feature channel in the current frame can be obtained. :

[0100]

[0101] Specifically, in this formula, for a channel to be considered important in the current frame, two conditions must be met simultaneously. The first condition: the channel itself must be important to the deep neural network model. The second condition is: the features detected by this channel are present and significant in the current frame. (High). If any of the above conditions are not met, the final importance score will be... They will all be very low. For example, a channel that is not important to the task ( Low), even when activated by noise ( Even if it's high, its ultimate importance is still not high. Conversely, a channel that is very important to the task ( (High), if it has no use in the current frame ( (If the value is low), its importance will also be suppressed.

[0102] It should be noted that by performing step S3, the present invention converts the intermediate layer feature data... It is parsed into a set of multiple channels with different semantic importance weights. The final output is a semantic importance parameter vector. This provides a quantifiable basis for subsequent compression. It enables distributed monitoring systems to perform data reduction and precision allocation in a "semantically aware" manner, prioritizing lossless or high-fidelity transmission of important information within a limited bandwidth budget, while heavily compressing less important information. This solves the problem of traditional encoding destroying key machine vision features and lays the foundation for achieving high-precision, low-latency collaborative reasoning in bandwidth-constrained edge computing scenarios.

[0103] S4. Perform semantic compression processing on the intermediate layer feature data based on the semantic importance parameter to generate a compressed feature data package. The semantic compression processing includes channel filtering based on importance and dynamic quantization operation.

[0104] It should be noted that this step aims to transform massive amounts of intermediate layer feature data into lightweight, transmittable data packets. In step S3, we have already assigned a quantized semantic importance parameter vector to each feature channel. Now, we will utilize this prior knowledge to analyze the intermediate layer feature data. A smart compression process is performed. Unlike traditional encoding, which strives to maintain overall visual quality, the semantic compression processing of this invention aims to achieve the maximum data compression rate at the cost of minimal loss of inference accuracy. This process mainly consists of importance-based channel filtering and dynamic quantization operations.

[0105] Furthermore, the core idea of ​​importance-based channel filtering is "focusing on the big picture and letting go of the small details," that is, directly eliminating feature channels that contribute very little to the final inference result, thereby achieving a significant initial reduction in data. The steps are as follows:

[0106] S401.1. Sort or classify feature channels according to semantic importance parameters: First, sort or classify the semantic importance parameter vector obtained in step S3. Sort the channels in descending order. Based on this, a channel index sequence from most important to least important can be obtained.

[0107] S401.2 Determine the preset importance condition for retention / filtering: Next, a threshold needs to be determined to decide which channels are retained and which are filtered. It is important to emphasize that this "preset importance condition" is not a fixed value, but can be dynamically adjusted according to the real-time needs of the system. In this embodiment, this condition is related to the target compression rate or available network bandwidth. Hooks. For example, the system can set a target data packet size. (represented as) Then, calculate the number of channels that need to be retained. Alternatively, a more direct approach is to set a retention rate. (For example, retain the most important 30% of the channels), that is .

[0108] S401.3, Retain, Filter, and Generate Mask Information: Based on the determined conditions, the system retains the first... Data for the most important feature channels. The rest... Data from each channel is completely filtered out and not further processed or transmitted. To ensure the collaborative computing nodes can correctly reconstruct the features, it's necessary to record which channels are retained. Therefore, this step also generates a binary channel mask. It can be represented as a variable with length and number of channels. Same vectors:

[0109]

[0110] Then, the mask It will be encapsulated as part of the metadata in the final compressed feature data packet.

[0111] It should be noted that channel filtering is a highly efficient compression method. For example, if there are a total of 512 channels, retaining only the most important 30% (approximately 154 channels) can directly achieve a compression rate of over 3 times. However, since the channels to be discarded are determined based on their actual contribution to the task (combining static sensitivity and dynamic activation), the filtering method of this invention can minimize the damage to critical information.

[0112] Furthermore, simply filtering channels is insufficient, as the retained channel data is typically still 32-bit floating-point numbers (FP32), and the data volume remains considerable. The purpose of introducing dynamic quantization is to further reduce the representation precision of each value while retaining the channels, thereby achieving a second data reduction. Its core lies in assigning different numbers of quantization bits to channels of varying importance. The steps are as follows:

[0113] S402.1 Establishing a nonlinear mapping relationship between semantic importance parameters and quantization accuracy: Considering that the importance of feature data in the intermediate layers of deep neural networks typically exhibits a long-tail distribution, meaning that a few channels have extremely high importance, while the importance of a large number of channels is at a medium to low level, using a linear mapping would waste valuable bit resources. Therefore, this invention employs nonlinear mapping logic to allocate sufficient bits (e.g., 8 bits or higher) to the most important channels to ensure their accuracy, while allocating fewer bits (e.g., 4 bits or 2 bits) to channels with lower importance. Specifically, it can be used for the first... Number of bits allocated to each reserved channel It can be determined using the following formula:

[0114]

[0115] in, This is a channel The number of quantization bits is dynamically allocated. and These are the preset minimum and maximum quantization bits (e.g., 2 and 8). It is a passage The semantic importance parameter. and These are the maximum and minimum importance parameters among all preserved channels in the current frame, used for normalization. It is a non-linear adjustment factor less than 1, used to adjust the curvature of the mapping curve. The smaller the value, the more convex the curve, meaning that the few channels with the highest importance parameters can be allocated more bit increments. This value can be obtained based on experience or by optimization on a specific dataset through offline experiments. In this embodiment, its initial value is 0.5. This indicates the rounding operation.

[0116] S402.2 Perform quantization and map floating-point data to discretized data: for each reserved channel Using the number of bits allocated in the previous step The data is then quantized. For this quantization, the present invention employs a uniform quantization method with a dynamic scaling factor: First, the maximum value of all floating-point values ​​within the channel is found. and minimum value Then, the dynamic scaling factor is calculated. and zero point :

[0117]

[0118] Finally, each floating-point value in the channel... Mapped to discretized integers :

[0119] in, The function ensures that the quantized value falls within the range of Within the range.

[0120] It is important to emphasize that the scaling factor is calculated dynamically. and zero point (or equivalent) Each channel is unique and will be packaged together with the channel mask as metadata necessary for data recovery.

[0121] Furthermore, the processed data and metadata are efficiently encapsulated together. The process is as follows:

[0122] S403.1 Residual Coding Using Temporal Redundancy: For video streams, the features between consecutive frames typically do not change significantly. To utilize this temporal redundancy, the present invention calculates the quantization features of the current frame. Reconstructed features from the previous frame residual information between At the same time, due to residuals It contains a large number of zeros, and its information entropy is much lower than that of the original quantized features, making it easier to compress.

[0123] S403.2 Lossless Compression: This involves compressing the calculated residual information. (Or for keyframes, directly quantize the features) The final compressed bitstream is generated using lossless compression algorithms such as entropy coding (e.g., Huffman coding, LZ series algorithms, or arithmetic coding).

[0124] S403.3 Encapsulation: The compressed bitstream and all metadata used for feature reconstruction are encapsulated into a unified compressed feature data packet. The structure of this data packet includes:

[0125] Packet header: frame index, timestamp, whether it is a keyframe, etc.

[0126] Metadata segment: Channel mask Number of bits allocated to each reserved channel The dynamic scaling factor is calculated for each reserved channel. and zero point .

[0127] Data payload segment: The bitstream that has undergone lossless compression.

[0128] It should be noted that the aforementioned channel filtering and dynamic quantization operations significantly reduce data volume. Furthermore, the entire compression process is "task-oriented" and "content-adaptive," ensuring that under any network conditions, the system prioritizes transmitting information that contributes most to the analysis task, thereby achieving an ultra-high compression rate while minimizing the impact on the final inference accuracy.

[0129] S5. The compressed feature data packet is sent to the collaborative computing node so that the collaborative computing node can reconstruct the features of the compressed feature data packet and complete the video analysis using the inference part.

[0130] It should be noted that this step aims to complete the formal handover of computing tasks from the edge device (distributed vision controller) to the collaborative computing nodes (such as edge servers or the cloud).

[0131] Furthermore, since the compressed feature data packet generated in step S4 has been optimized in size based on the current network bandwidth and data transmission latency, it can be transmitted in a way that meets real-time requirements. At this point, the distributed vision controller needs to send this data packet to the preset collaborative computing node via a network interface (such as Wi-Fi, 5G, or wired Ethernet).

[0132] Specifically, to minimize data transmission latency, this embodiment preferably uses UDP (User Datagram Protocol) instead of the traditional TCP. Although UDP cannot guarantee the arrival and order of data packets, in real-time video analytics scenarios, the impact of losing a single frame is usually less than the cumulative latency caused by TCP retransmission mechanisms. Therefore, the system can design a simple sequence number mechanism at the application layer to handle out-of-order delivery and detect packet loss.

[0133] Furthermore, after the collaborative computing node receives the data packet, it executes the inverse process of semantic compression in step S4, namely feature reconstruction. This process strictly follows the metadata decoding process, aiming to recover the semantic information intended to be transmitted by the edge as losslessly as possible. The process is as follows:

[0134] S501.1 Unpacking and Metadata Parsing: The collaborative computing node first parses the data packet, separating the data payload (compressed bitstream) and metadata segment.

[0135] S501.2, Inverse Lossless Compression and Residual Recovery: Perform inverse entropy encoding (such as Huffman decoding) on ​​the data load to obtain the quantized residual information. (or quantization features of keyframes) If it is residual information, then it is reconstructed from the previous frame cached by the collaborative computing nodes. Add them together to recover the quantization features of the current frame: .

[0136] S501.3, Inverse Dynamic Quantization: Collaborative computing nodes traverse all channels in the mask. Channel index marked with "1" For each such channel, utilize its unique metadata ( ), quantize integers Restore to floating-point value : .

[0137] It should be noted that this floating-point numerical formula is the inverse operation of the quantization process in step S4. Among them, The dynamic range of the value has been restored, and This corrects the offset. Since the numerical distribution and importance of each channel are different, matching is necessary. and This is the key to achieving high-precision recovery. If all channels share the same scaling factor, it will inevitably damage those channels with special numerical ranges but important semantics.

[0138] S501.4 Reconstructing the Complete Feature Tensor: After inverse quantization, we obtain high-precision floating-point data for all preserved channels. To feed this into the model's neural network, we need to reconstruct a tensor with the original number of channels. Complete feature tensor Specifically, the collaborative computing node creates an all-zero tensor and then, based on the channel mask... The recovered channel data is then filled into their corresponding positions. For those channels that were filtered out in step S4 ( Its position will remain zero. This has the original number of channels. Complete feature tensor It can be represented as:

[0139]

[0140] It should be noted that setting untransmitted channels to zero is based on the judgment of "semantic insignificance" in step S3. It needs to be explained that in the forward propagation of a neural network, zero input usually means that the feature detector is not activated, thus minimizing its impact on subsequent layers.

[0141] Furthermore, when the reconstructed feature tensor Once everything is ready, the inference portion allocated to the collaborating end in step S1 can be connected. It can be expressed by the formula:

[0142]

[0143] in, This refers to the final video analysis result, such as bounding box coordinates and class confidence in object detection tasks, or pixel-level masks in scene segmentation tasks. Because the solution of this invention preserves the feature information that contributes most to the task to the greatest extent through semantic compression, therefore... The result of the inference It can closely approximate the results obtained from the uncompressed original features in terms of accuracy, but the end-to-end latency of the entire process is greatly reduced.

[0144] Furthermore, to enable the system to learn adaptively, collaborative computing nodes can periodically perform a "precision calibration" (e.g., when computing resources are available). Specifically, uncompressed features can be obtained through some means (such as low-priority transmission or self-computation at the edge). The result of reasoning And compare it with the inference results based on compressed features. A comparison is made. The difference between the two (e.g., the difference in the loss function) can be used to generate parameter update instructions.

[0145] Specifically, in this embodiment, the parameter update instruction is generated based on the gradient backpropagation mechanism. That is, the cooperating nodes calculate the impact of the loss difference on the reconstructed features. gradient It needs to be explained that the significance of this gradient lies in: to reduce the loss of accuracy. Which channels and spatial locations should have their values ​​adjusted, and how? Then, this gradient is compressed and fed back to the distributed vision controller at the edge for dynamic calibration of the channel sensitivity model. This process can update the channel sensitivity parameters using gradient ascent. Implementation, that is , The learning rate used to update the channel sensitivity parameter determines the step size for each iteration. The initial value of this value can be selected within a typical range, such as between 1e-4 and 1e-6, and a dynamic learning rate adjustment strategy can be adopted, such as decaying as the number of updates increases.

[0146] It should be noted that by updating the parameters as described above, the compression strategy at the edge is no longer static but can be fine-tuned online based on the actual inference results. The advantage of this is that the "semantic evaluation logic" becomes increasingly adaptable to the specific scenario, thereby achieving continuous Pareto optimization of accuracy and compression ratio.

[0147] Furthermore, in large-scale deployment scenarios involving multiple vision controllers, if the data packet contains spatial transformation parameters (e.g., the homography matrix from the coordinate system of camera 2 to camera 1), If so, the collaborative computing node will perform an additional data alignment step after completing feature reconstruction.

[0148] Specifically, before performing multi-view feature fusion, this step uses the spatial transformation parameter to warp feature maps from different viewpoints into a unified coordinate space: After alignment, multiple feature maps This allows for splicing or element-level fusion at the channel dimension (such as taking the maximum value) to form a more information-rich and broader fused feature, which is then fed into the inference part.

[0149] It should be noted that this data alignment step can break the field of view limitations of a single camera, enabling the collaborative end to obtain a "third-person perspective," thereby achieving cross-camera target tracking, global scene understanding, and effective detection of targets occluded by a single camera, thus improving the overall intelligence level of the distributed vision system.

[0150] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of this application can be implemented using various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0151] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0152] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0153] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0154] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0155] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A real-time video edge analysis method for a distributed vision controller, characterized in that, The method is applied in a distributed vision controller, including: The system acquires real-time video streams and, based on the current computing resource status and network transmission status, determines a network layer segmentation strategy for the deep neural network model. The segmentation strategy divides the model logic into a feature extraction part executed locally and an inference part executed by the collaborating end. The feature extraction part is used to perform calculations on the current video frame to generate intermediate layer feature data; The semantic importance parameter of each feature channel in the intermediate layer feature data is determined, and the semantic importance parameter characterizes the degree of contribution of the feature channel to the model inference result; The determination of the semantic importance parameters of each feature channel in the intermediate layer feature data includes: Obtain a pre-built channel sensitivity model, which records the correlation between feature channels and model loss function; By combining the feature activation intensity of the current video frame with the channel sensitivity model, the semantic importance parameter of each feature channel is calculated; Based on the semantic importance parameter, semantic compression processing is performed on the intermediate layer feature data to generate a compressed feature data package. The semantic compression processing includes channel filtering based on importance and dynamic quantization operations. The compressed feature data packet is sent to the collaborative computing node so that the collaborative computing node can reconstruct the features of the compressed feature data packet and use the inference part to complete the video analysis; The method further includes a step of receiving feedback information from collaborative computing nodes and updating the compression strategy, which includes: Receive parameter update instructions sent by collaborative computing nodes, wherein the parameter update instructions are calculated by the collaborative computing nodes based on the differences between the full feature inference results and the compressed feature inference results; Adjust the weights or model parameters used to calculate semantic importance parameters according to the parameter update instructions, and correct the semantic evaluation logic of the frame. The method further includes a multi-machine collaborative data alignment step, which includes: When the distributed vision controller detects an area of ​​overlap with the field of view of a neighboring device, spatial transformation parameters are added to the compressed feature data packet. The spatial transformation parameters are used to instruct the collaborative computing nodes to map the current feature data to a unified coordinate space for multi-view feature fusion.

2. The real-time video edge analysis method of the distributed vision controller as described in claim 1, characterized in that, The network layer segmentation strategy for determining the deep neural network model includes: Calculate a comprehensive evaluation of the local computation cost and data transmission cost at multiple candidate segmentation locations; Based on the comprehensive evaluation value, the candidate segmentation position that minimizes the overall system latency or energy consumption is selected as the current network layer segmentation point.

3. The real-time video edge analysis method for a distributed vision controller as described in claim 1, characterized in that, The semantic compression processing of the intermediate layer feature data based on the semantic importance parameter includes: The feature channels are sorted or classified according to the semantic importance parameter. Retain feature channel data that meets the preset importance conditions, and generate mask information to indicate the location of the retained channels; Feature channel data that does not meet the preset importance conditions are filtered out.

4. The real-time video edge analysis method of the distributed vision controller as described in claim 3, characterized in that, Also includes: Establish a mapping relationship between semantic importance parameters and quantization accuracy; Assign a first number of quantization bits to the feature channel of the first semantic importance parameter, and assign a second number of quantization bits to the feature channel of the second semantic importance parameter; The floating-point data of the feature channel is mapped to discretized data using the allocated number of quantization bits.

5. The real-time video edge analysis method of the distributed vision controller as described in claim 4, characterized in that, The mapping relationship adopts nonlinear mapping logic to adapt to the long-tail distribution characteristics of the intermediate layer feature data of deep neural networks, and includes a dynamic scaling factor for reverse data recovery in the compressed feature data package.

6. The real-time video edge analysis method of the distributed vision controller as described in claim 1, characterized in that, The generation of compressed feature data packets includes: Calculate the residual information of the feature data of the current frame relative to the feature data of historical frames; The residual information is losslessly compressed and encoded to generate a compressed bitstream; The compressed bitstream and auxiliary metadata used for feature reconstruction are encapsulated into a compressed feature data packet.

7. The real-time video edge analysis method of the distributed vision controller as described in claim 1, characterized in that, The feedback information is generated based on the gradient backpropagation mechanism and is used to dynamically calibrate the sensitivity distribution of feature channels under different scenarios.

Images