A machine vision-oriented video transmission method and transmission system

Abstract

The application discloses a kind of machine vision-oriented video transmission method and transmission system, transmission method includes: video camera gathers video, determines current key frame, calculates the difference of next video frame and recent key frame, filters the video frame with difference less than filter threshold, and the video frame with difference not less than threshold is reserved as key frame;Any key frame in current time window is selected and divided into multiple regions with different importance levels and different levels of quantization parameters are assigned, and all key frames in the same time window are uploaded to the edge after being encoded according to the same encoding strategy;The edge decodes the encoded data and identifies the target, provides feedback information to the video camera in combination with the computing resources and the current bandwidth resources, and guides the video camera frame filtering and configuration selection strategy. By screening out redundant frames and setting time window, intra-frame block re-encoding is realized, which effectively reduces the transmission bandwidth and computing resource overhead while ensuring the accuracy of neural network inference.

Description

Technical Field

[0001] This invention belongs to the field of video analysis technology, and more specifically, relates to a video transmission method and transmission system for machine vision. Background Technology

[0002] With advancements in computer vision technology, particularly the development of high-precision target detection technology based on neural networks, video analytics has adapted to the era of big data. This has enabled the automation of common tasks such as traffic monitoring, target detection, and tracking, leading to widespread application and continuous empowerment of smart cities and the Internet of Things. Video analytics aims to utilize computer vision and even artificial intelligence technologies to process video streams captured by terminal devices such as cameras in real time and obtain accurate analysis results.

[0003] Due to insufficient computing power on edge devices, existing video analytics applications are often based on a centralized device-cloud architecture. Edge devices upload video streams to the cloud, which then analyzes the video using technologies such as deep neural network inference. Compared to the cloud, edge nodes are geographically closer to the edge devices, resulting in lower network traffic and faster response times for video analytics, thus gaining widespread application in this field. However, an edge node typically needs to communicate with multiple edge devices, requiring the transmission and processing of large amounts of video data. For edge nodes with limited bandwidth and computing resources, the proper handling of large volumes of video data is both a necessary requirement and a challenge to meet real-time requirements. Existing edge computing-based video analytics frameworks still employ traditional video transmission and encoding methods. While these methods effectively compress data volume, there is still room for improvement. This is because traditional video encoding algorithms are designed for human users, compressing data while ensuring human service needs are met. In automated video analytics scenarios, the video is no longer provided to humans but to computer vision algorithms and even neural networks, allowing for more aggressive compression of video data without compromising analytical accuracy. Adjacent frames often exhibit temporal correlation, leading algorithms like H.264 to divide video frames into I, B, and P frames for encoding video macroblock residuals. However, for neural networks, inputting similar frames will produce repetitive results, resulting in redundancy and wasted resources. H.264 and similar algorithms typically use consistent quantization parameters (CQP) for encoding the same frame, or adaptive quantization based on regional variations (e.g., CRF). However, within the same frame, the sensitivity of different regions to the neural network output depends primarily on the presence or absence of the target object, rather than the factors mentioned above. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a video transmission method and transmission system for machine vision, which aims to improve the accuracy and real-time performance of video transmission for machine vision.

[0005] To achieve the above objectives, according to one aspect of the present invention, a video transmission method for machine vision is provided, comprising:

[0006] The camera captures video, determines the current key frame, calculates the difference between the next video frame and the most recent key frame, filters video frames with a difference less than a filtering threshold, and retains video frames with a difference not less than the threshold as key frames.

[0007] Select any keyframe within the current time window and divide the frame into multiple regions with different importance levels according to their importance. Assign different levels of quantization parameters to regions with different importance levels. Encode all keyframes within the same time window according to the same encoding strategy and then upload them to the edge.

[0008] The edge decodes the encoded data and identifies the target. When the transmission delay exceeds expectations, the camera is controlled to increase the filtering threshold or decrease the encoding quality. When the analysis accuracy is lower than expected, the camera is controlled to decrease the filtering threshold or increase the encoding quality.

[0009] In one embodiment, when calculating the difference between the next video frame and the most recent keyframe, the calculation is based on video features selected from historical data analysis during offline processing. The video features selected from historical data analysis include:

[0010] Sampling of historical video data from the current camera to determine typical video scenes within the current camera's monitoring range;

[0011] Select the appropriate video features based on the processing task, accuracy, and latency requirements.

[0012] In one embodiment, depending on the processing task, accuracy, and latency requirements, corresponding video features are selected, including:

[0013] Initial selection of multiple video features;

[0014] Determine keyframes and obtain corresponding filter frames based on each video feature;

[0015] The accuracy of filtering is judged by calculating the F1 score of the filtered frames based on the keyframes using the Intersection over Union (IOU).

[0016] The video features with the highest accuracy are selected as the video features used for online video transmission.

[0017] In one embodiment, the frame is divided into multiple regions with different importance levels according to importance, including:

[0018] Determine the object's density function:

[0019]

[0020] Among them, b i For the i-th block, For block b i The object density, obj represents the target object detection box, S obj This represents the area of ​​the detection box for a given object. Represents block b i area, Represents block b i The area of ​​the union of all detection boxes within N i Represents block b i The number of inner detection boxes, N total This represents the total number of detection boxes identified in the corresponding frame, where a and b are both positive numbers representing weights;

[0021] The frame is divided into multiple regions and the object density function of each region is calculated. The higher the object density of a region, the higher its quality level and the higher the corresponding encoding quality requirement.

[0022] In one embodiment, the first keyframe within the current time window is selected to determine the encoding strategy, and other keyframes within the same time window are encoded with reference to the encoding strategy of the first keyframe.

[0023] In one embodiment, the keyframe is transmitted to the edge as an H.264 bitstream.

[0024] In one embodiment, after the edge decodes the encoded data, the decoded video frames are stored at the tail of the queue, and the video frames at the head of the queue are taken out sequentially for target recognition. The processing delay is estimated based on the length of the queue; the longer the queue, the more severe the delay.

[0025] In one embodiment, the initial value of the filtering threshold is 0.

[0026] According to another aspect of the present invention, a video transmission system for machine vision is provided, including a camera and an edge computing module. The camera is equipped with an inter-frame filtering module, an intra-frame configuration selection module, and an encoder. The edge computing module is equipped with a decoder, a video processing module, a resource awareness module, and an information feedback module.

[0027] The inter-frame filtering module is used to determine the current key frame, calculate the difference between the next video frame and the most recent key frame, filter video frames with a difference less than the filtering threshold, and retain video frames with a difference not less than the threshold as key frames.

[0028] The intra-frame configuration selection module is used to select any key frame within the current time window and divide the frame into multiple regions with different importance levels according to their importance.

[0029] The encoder is used to assign different levels of encoding configuration to regions of different importance levels of keyframes, and all keyframes within the same time window are encoded according to the same encoding strategy and uploaded to the edge.

[0030] The decoder is used to decode the encoded data;

[0031] The video processing module is used to perform target recognition on the decoded video frames;

[0032] The resource awareness module is used to determine whether the transmission delay exceeds expectations;

[0033] The information feedback module is used to control the camera to increase the filtering threshold or decrease the encoding quality when the transmission delay exceeds expectations, and to control the camera to decrease the filtering threshold or increase the encoding quality when the analysis accuracy is lower than expected.

[0034] In one embodiment, the intra-frame configuration selection module is equipped with a lightweight neural network. The intra-frame configuration selection module uses the lightweight neural network to identify the number and area of ​​targets within the current keyframe and based on an object density function.

[0035] Divide the current keyframe into multiple regions with different quality levels, where b i For the i-th block, For block b i The object density, obj represents the target object detection box, S obj This represents the area of ​​the detection box for a given object. Represents block b i area, Represents block b i The area of ​​the union of all detection boxes within N i Represents block b i The number of inner detection boxes, N total This represents the total number of detection boxes identified in the corresponding frame, where a and b are both positive numbers representing weights.

[0036] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects:

[0037] 1. This invention analyzes the characteristics of video transmission for machine vision and improves existing video coding methods. Existing video analysis systems typically use neural networks for automated image processing; however, for neural networks, inputting similar frames leads to repetitive results. This invention uses a custom inter-frame difference value based on image features to calculate the similarity between adjacent video frames, retaining keyframes and filtering out redundant frames similar to the keyframes. Only the selected keyframes are used for subsequent data analysis, saving bandwidth resources for data transmission and ensuring the accuracy of subsequent data analysis.

[0038] 2. This invention encodes regions with different object densities within the same frame using different quantization parameters, and further defines a method to characterize the importance of a block of target objects by density. By combining inter-frame filtering of redundant frames and intra-frame block coding, transmission bandwidth and computational resource overhead can be effectively reduced while ensuring the accuracy of neural network inference. Furthermore, considering the limited computing power of general-purpose cameras, this invention does not analyze and calculate for every frame division. Instead, based on spatiotemporal correlation, a time window is set, and only a specific frame within the same time window is analyzed to determine its encoding strategy. Other frames within the same time window directly refer to the encoding strategy of that frame for encoding, thereby saving computational resources on the camera, meeting real-time requirements, and reducing deployment costs.

[0039] 3. Considering the dynamic changes in video content and bandwidth resources, after the encoded data is transmitted to the edge, the edge will combine computing resources and current bandwidth resources to provide feedback information to the camera, guiding the camera's video frame filtering and configuration selection strategy. This achieves adaptive adjustment of video transmission and encoding configuration, enhancing the robustness of the system. Attached Figure Description

[0040] Figure 1 This is a flowchart of the steps of a video transmission method for machine vision according to an embodiment;

[0041] Figure 2 This is a structural block diagram of a machine vision-oriented video transmission system according to an embodiment. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0043] like Figure 1 The diagram shows a flowchart of a machine vision-oriented video transmission method according to one embodiment, which includes at least the following steps:

[0044] Step S100: The camera captures video, determines the current key frame, calculates the difference between the next video frame and the most recent key frame, filters video frames with a difference less than the filtering threshold, and retains video frames with a difference not less than the threshold as key frames.

[0045] Different video features have their applicable scenarios. In one embodiment, offline analysis is first performed to sample historical video data from the current camera to identify typical video scenes. The appropriate features are then selected based on the different processing tasks and requirements for accuracy and latency. For example, for tasks with lower accuracy requirements and higher latency requirements, low-level features such as pixel values, graphical contours, and corner points should be selected; for tasks with higher accuracy requirements and lower latency requirements, high-level feature description methods such as SIFT, HOG, ORB, and optical flow should be selected. Furthermore, different features represent frame differences in different ways and have different feature scales. Therefore, it is necessary to define an appropriate frame difference to describe the degree of difference between frame contents. For example, pixel features use the proportion of pixels whose pixel value changes exceed a certain value as the frame difference, while optical flow methods use the magnitude of the tracking vectors in the corresponding optical flow fields of two frames as the frame difference. In the offline analysis stage, after filtering the sampled video data using multiple features, the filtered redundant frames are used as a benchmark, and an F1 score is calculated based on the retained keyframes to measure the similarity between the two frames, evaluate the accuracy of the filtering, and select the feature with the highest accuracy as the filtering standard.

[0046] In this embodiment, based on historical video data collected by the camera, the content characteristics of the video stream are analyzed offline to select the most suitable video features for the scene corresponding to the camera, rather than the fixed features typically used by video analysis systems, thus achieving the best effect for the specific scene of the current camera. Traditional H.264 encoding methods use macroblock residuals to measure inter-frame differences, while neural networks extract features from images to generate outputs. Pixel value residuals are insufficient to characterize the differences in inference results obtained from input deep neural networks between different frames. In contrast, this embodiment, by customizing inter-frame difference values, can more accurately identify redundant information for the neural network. Furthermore, considering that in target detection scenarios, the neural network only needs to detect the presence of the target object in the same frame, redundant frames are not encoded using residuals with reference frames, but are directly filtered out (not participating in subsequent encoding), thereby further reducing the bandwidth required for transmission and lowering the computational resources required by edge nodes.

[0047] Initially, after selecting features, due to the unknown bandwidth and computing resource conditions, to minimize accuracy degradation, 0 or other small values ​​are chosen as the filtering threshold to begin filtering. If the threshold is too small, it will be dynamically adjusted based on feedback information received later.

[0048] Step S200: Select any keyframe within the current time window and divide the frame into multiple regions with different importance levels according to their importance. Assign different levels of quantization parameters to regions with different importance levels. Encode all keyframes within the same time window according to the same encoding strategy and then upload them to the edge.

[0049] In the temporal domain, the keyframes filtered in step S100 are divided into groups according to fixed time windows, and in the spatial domain, the first frame within a certain time window is divided into blocks of the same size. Lightweight neural network inference is run on this frame to obtain bounding box detection results. Since neural networks typically have lower accuracy in recognizing small and dense objects, but higher accuracy in recognizing large and sparse objects, the target object density is defined as...

[0050]

[0051] Among them, b i Let S be the i-th block, where obj represents the target object detection bounding box. obj This represents the area of ​​the detection box for a given object. This represents the area of ​​the i-th block. N represents the area of ​​the union of all detection boxes within the i-th block. i N represents the number of detection boxes within the i-th block. total This represents the total number of detection boxes identified in the frame, where 'a' and 'b' are both positive numbers representing weights. The first term on the right-hand side of the above equation reflects the size of objects in the block, while the second term reflects the number of objects in the block. For macroblocks with high object density, low quantization parameters are used for encoding, and the remaining frames within the same time window all use the same encoding strategy as this frame.

[0052] The frame sequence after inter-frame filtering is encoded into H.264 video according to the encoding strategy determined in the above steps, and then transmitted to the edge computing node via wireless network.

[0053] Traditional H.264 coding methods quantize the residuals obtained from intra-frame prediction using a discrete cosine transform and then apply a uniform quantization step size, or fine-tune them using an adaptive quantization algorithm. However, their adjustment principles do not consider the video accuracy requirements for target detection in machine vision scenarios. For example, video accuracy should be as high as possible if there are target object regions, and as low as possible if there are none. In contrast, this invention first divides the video frame into equal-length video segments in the temporal domain, and then divides the first frame of each segment into equally sized blocks, which are then input into a neural network for target detection to obtain the target object density of each block. The object density is calculated using the above formula; the higher the density value, the lower the quantization parameter, and the higher the coding quality level. Different regions within the same frame use different codes, and the same coding strategy is used within the same video segment. This further compresses the video data volume while ensuring latency and accuracy.

[0054] Step S300: The edge decodes the encoded data and identifies the target. When the transmission delay exceeds the expectation, the camera is controlled to increase the filtering threshold or decrease the encoding quality. When the analysis accuracy is lower than expected, the camera is controlled to decrease the filtering threshold or increase the encoding quality.

[0055] For each camera's video stream, the edge node maintains a synchronization queue: after receiving the H.264 video stream, the edge node decodes it and stores the resulting keyframes at the tail of the queue; the video processing module, located in another thread, continuously polls the head of the queue, retrieves the frames to be processed, sends them to the deep neural network for inference, and obtains the detection results. The queue length reflects the edge node's processing status of the camera's video data: when the queue length cumulatively increases, it indicates that the edge node is failing to process the video data in real time. To meet latency requirements, the filtering threshold at the camera end should be increased to reduce the amount of data to be processed. Similarly, when the resource awareness module detects a shortage of bandwidth resources, it should also increase the filtering threshold to reduce the amount of transmitted data to meet channel capacity limitations and thus meet latency requirements. In addition, increasing the filtering threshold may lead to erroneous filtering behavior, causing a decrease in analysis accuracy. Therefore, when the video processing module detects a decrease in accuracy, it should lower the filtering threshold. At regular intervals, the information feedback module will consider the above factors within that period, weigh accuracy against latency, and obtain a new filtering threshold and QP selection scheme, which will be transmitted to the corresponding camera as feedback information.

[0056] Considering the dynamic nature of video content and channel bandwidth, as well as the varying requirements of different applications for query accuracy and latency, the video transmission coding method provided in this invention simultaneously establishes a feedback data stream flowing from edge nodes back to the cameras. When the video processing module detects that the neural network analysis accuracy is lower than the accuracy target, it notifies the corresponding camera to lower the filtering frame difference threshold or reduce the candidate quantization parameter sequence through the information feedback module, while satisfying latency and bandwidth constraints. When the resource awareness module detects that the edge node's computing resources are insufficient, causing frame buffer blockage and the accumulated length to exceed a certain threshold, it notifies the corresponding camera to raise the filtering frame difference threshold through the information feedback module, while satisfying accuracy and bandwidth constraints. This feedback information dynamically guides the camera coding strategy, achieving adaptive adaptation to dynamic changes in video content and meeting the heterogeneity of different tasks.

[0057] Accordingly, the present invention also relates to a video transmission system for machine vision, including a camera and an edge, such as... Figure 2 As shown, the camera is equipped with an inter-frame filtering module, an intra-frame configuration selection module, and an encoder. At the edges, it is equipped with a decoder, a video processing module, a resource awareness module, and an information feedback module.

[0058] The inter-frame filtering module is used to determine the current keyframe, calculate the difference between the next video frame and the most recent keyframe, filter video frames with a difference less than the filtering threshold, and retain video frames with a difference not less than the threshold as keyframes.

[0059] The intra-frame configuration selection module is used to select any keyframe within the current time window and divide the frame into multiple regions with different importance levels according to their importance.

[0060] The encoder is used to assign different QP level encoding configurations to regions with different importance levels of keyframes, and all keyframes within the same time window are encoded according to the same encoding strategy and uploaded to the edge.

[0061] The decoder is used to decode encoded data;

[0062] The video processing module is used to perform target recognition on the decoded video frames;

[0063] The resource awareness module is used to determine whether the transmission delay exceeds expectations;

[0064] The information feedback module is used to control the camera to increase the filtering threshold or decrease the encoding quality when the latency exceeds expectations, and to control the camera to decrease the filtering threshold or increase the encoding quality when the analysis accuracy is lower than expected.

[0065] Specifically, the intra-frame configuration selection module is equipped with a lightweight neural network. This module uses the lightweight neural network to identify the number and area of ​​targets within the current keyframe and then selects targets based on an object density function. Divide the current keyframe into multiple regions with different quality levels, where b i For the i-th block, For block b i The object density, obj represents the target object detection box, S obj This represents the area of ​​the detection box for a given object. Represents block b i area, Represents block b i The area of ​​the union of all detection boxes within N i Represents block b i The number of inner detection boxes, N total This represents the total number of detection boxes identified in the corresponding frame, where a and b are both positive numbers representing weights.

[0066] The video transmission system for machine vision can transmit data according to the video transmission method for machine vision described above. Each functional unit is used to execute the relevant steps in the video transmission method. For details, please refer to the above description, which will not be repeated here.

[0067] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A video transmission method for machine vision, characterized in that, include: The camera captures video, determines the current key frame, calculates the difference between the next video frame and the most recent key frame, filters video frames with a difference less than a filtering threshold, and retains video frames with a difference not less than the threshold as key frames. Select any keyframe within the current time window and divide the frame into multiple regions with different importance levels according to their importance. Assign different levels of quantization parameters to regions with different importance levels. Encode all keyframes within the same time window according to the same encoding strategy and then upload them to the edge. The edge decodes the encoded data and identifies the target. When the transmission delay exceeds the expectation, the camera is controlled to increase the filtering threshold or reduce the encoding quality. When the analysis accuracy is lower than expected, the camera is controlled to decrease the filtering threshold or increase the encoding quality. Among them, the frame is divided into multiple regions with different importance levels according to its importance, including: Determine the object's density function: ; in, For the i-th block, For blocks The density of the object, Represents the target object detection box. This represents the area of ​​the detection box for a given object. Represents a block area, Represents a block The area of ​​the union of all detection boxes within the bounding box. Represents a block The number of inner detection boxes, This represents the total number of detection boxes identified in the corresponding frame, where a and b are both positive numbers representing weights; The frame is divided into multiple regions and the object density function of each region is calculated. The higher the object density of a region, the higher its quality level and the higher the corresponding encoding quality requirement.

2. The video transmission method for machine vision as described in claim 1, characterized in that, When calculating the difference between the next video frame and the most recent keyframe, the calculation is based on video features selected from historical data analysis during offline processing. These video features selected from historical data analysis include: Sampling of historical video data from the current camera to determine typical video scenes within the current camera's monitoring range; Select the appropriate video features based on the processing task, accuracy, and latency requirements.

3. The video transmission method for machine vision as described in claim 2, characterized in that, Depending on the processing task, accuracy, and latency requirements, select the appropriate video features, including: Initial selection of multiple video features; Determine keyframes and obtain corresponding filter frames based on each video feature; The F1 score of the filtered frames is calculated based on the Intersection over Union (IOU) ratio, with the keyframes as the benchmark, to evaluate the accuracy of the filtering. The video features with the highest accuracy are selected as the video features used for online video transmission.

4. The video transmission method for machine vision as described in claim 1, characterized in that, The first keyframe within the current time window is selected to determine the encoding strategy, and other keyframes within the same time window are encoded using the same encoding strategy as the first keyframe.

5. The video transmission method for machine vision as described in claim 1, characterized in that, The keyframes are transmitted to the edge in the form of an H.264 bitstream.

6. The video transmission method for machine vision as described in claim 1, characterized in that, After the edge decodes the encoded data, the decoded video frames are stored at the tail of the queue, and the video frames at the head of the queue are taken out in sequence for target recognition. The processing delay is estimated based on the length of the queue; the longer the queue, the more severe the delay.

7. The video transmission method for machine vision as described in claim 1, characterized in that, The initial value of the filtering threshold is 0.

8. A video transmission system for machine vision, comprising a camera and an edge, characterized in that, The camera is equipped with an inter-frame filtering module, an intra-frame configuration selection module, and an encoder. The edge is equipped with a decoder, a video processing module, a resource awareness module, and an information feedback module. The inter-frame filtering module is used to determine the current key frame, calculate the difference between the next video frame and the most recent key frame, filter video frames with a difference less than the filtering threshold, and retain video frames with a difference not less than the threshold as key frames. The intra-frame configuration selection module is used to select any key frame within the current time window and divide the frame into multiple regions with different importance levels according to their importance. The encoder is used to assign different levels of encoding configuration to regions of different importance levels of keyframes, and all keyframes within the same time window are encoded according to the same encoding strategy and uploaded to the edge. The decoder is used to decode the encoded data; The video processing module is used to perform target recognition on the decoded video frames; The resource awareness module is used to determine whether the transmission delay exceeds expectations; The information feedback module is used to control the camera to increase the filtering threshold or decrease the encoding quality when the transmission delay exceeds the expectation, and to control the camera to decrease the filtering threshold or increase the encoding quality when the analysis accuracy is lower than the expectation. The intra-frame configuration selection module is equipped with a lightweight neural network. This module uses the lightweight neural network to identify the number and area of ​​targets within the current keyframe and then uses an object density function as the basis for selection. The current keyframe is divided into multiple regions with different quality levels, among which... For the i-th block, For blocks The density of the object, Represents the target object detection box. This represents the area of ​​the detection box for a given object. Represents a block area, Represents a block The area of ​​the union of all detection boxes within the bounding box. Represents a block The number of inner detection boxes, This represents the total number of detection boxes identified in the corresponding frame, where a and b are both positive numbers representing weights.

Images