Ultra-low delay end-to-end real-time video stream intelligent analysis method, system, device, medium and product

Abstract

The invention discloses an ultra-low delay end-to-end real-time video stream intelligent analysis method, system and device, a medium and a product, and relates to the field of video data analysis, and the method comprises the steps: carrying out the real-time decoding of a video stream through employing a hardware decoding technology; an ADM-YOLOV11 network is adopted to carry out intelligent identification on the decoded video streams, and a TensorRT batch processing technology is combined to realize parallel processing of multiple paths of video streams; drawing and generating an image sequence based on the intelligent identification result of the ADM-YOLOV11 network; and encoding the image sequence by adopting a hardware encoding technology, and distributing the encoded image sequence. The method can meet the actual requirements of monitoring work in the field of natural resources.

Description

Technical Field

[0001] This application relates to the field of video data analysis, and in particular to an ultra-low latency end-to-end real-time intelligent analysis method, system, device, medium and product for video streams. Background Technology

[0002] In natural resource monitoring, target detection and segmentation technologies for real-time video streams face numerous pressing technical challenges. Firstly, in farmland monitoring, it's crucial to accurately identify illegal occupation of farmland for non-agricultural construction, distinguish between different crops and non-grain crops, and promptly obtain information on changes in farmland use. Secondly, for forest fire monitoring, it's essential to quickly and accurately detect fire sources and identify the fire's extent, providing reliable early warnings in the initial stages. Simultaneously, regardless of the monitoring scenario, the analyzed and processed real-time video stream must be rapidly and stably distributed to the monitoring terminals to ensure monitoring personnel can promptly grasp the situation and provide data support for decision-making.

[0003] The basic process of existing natural resource monitoring technologies in addressing the above problems is as follows:

[0004] (1) Decode the video stream into an image sequence;

[0005] (2) Apply relevant processing algorithms to the image sequence;

[0006] (3) Plot the processing results onto an image sequence;

[0007] (4) Encode the drawn image sequence

[0008] (5) Distribute the encoded video stream to the monitoring terminal.

[0009] The processing algorithms in step (2) above are mainly divided into two categories: one is the traditional image processing algorithm that is computed by a powerful CPU (Central Processing Unit); the other is the intelligent recognition algorithm based on deep learning that is computed by a powerful GPU (Graphics Processing Unit).

[0010] Based on the above description, some systems rely on traditional image processing algorithms, such as threshold-based segmentation and edge detection, for target detection and segmentation. These algorithms distinguish targets by setting fixed thresholds or extracting image edge features, but their adaptability is extremely poor in video data processing under complex environments. In farmland monitoring, due to factors such as different crop growth stages and varying lighting conditions, traditional algorithms struggle to accurately distinguish between different crops and non-grain crops. They also fail to accurately identify concealed non-agricultural construction activities, such as temporary structures. In forest fire monitoring, the dynamic changes in smoke and interference from complex terrain easily lead to false positives and false negatives in fire source detection. Furthermore, these algorithms are slow and lack robustness, failing to meet the timeliness requirements of real-time monitoring.

[0011] Other systems introduce deep learning algorithms for object detection and segmentation, which improves detection accuracy and versatility to some extent. However, in the video stream retrieval and distribution stages, CPU-based video encoding and decoding algorithms are often used, resulting in significant time consumption during the encoding and decoding phase. This leads to end-to-end latency (from video device to monitoring end) often exceeding 10-15 seconds. Secondly, video stream push often uses common network transmission protocols, such as HTTP (Hypertext Transfer Protocol) or common streaming media transmission protocols, without optimization tailored to specific application scenarios. In natural resource monitoring scenarios, monitoring areas are typically located in remote regions with complex and unstable network environments. Common transmission protocols lack effective adaptive mechanisms, and under conditions of fluctuating network bandwidth and weak signals, high video transmission latency and severe packet loss are likely to occur. This prevents the timely and complete delivery of analyzed real-time video to the monitoring center, further increasing end-to-end latency. After receiving an alarm about an anomaly, monitoring personnel often need to operate the surveillance camera to investigate and confirm the situation. However, the high latency of the video stream at this time makes it impossible for the camera to provide timely feedback after the situation has been controlled. As a result, it is difficult for monitoring personnel to grasp the situation in a timely manner and take timely countermeasures, which fails to meet the actual needs of monitoring work in the field of natural resources. Summary of the Invention

[0012] The purpose of this application is to provide an ultra-low latency end-to-end real-time video stream intelligent analysis method, system, device, medium and product that can meet the actual needs of monitoring work in the field of natural resources.

[0013] To achieve the above objectives, this application provides the following solution:

[0014] Firstly, this application provides an ultra-low latency end-to-end real-time video stream intelligent analysis method, including:

[0015] Hardware decoding technology is used to decode the video stream in real time;

[0016] The ADM-YOLOV11 network is used to intelligently identify the decoded video stream, and TensorRT batch processing technology is combined to achieve parallel processing of multiple video streams.

[0017] Image sequences are generated based on the intelligent recognition results of the ADM-YOLOV11 network;

[0018] The image sequence is encoded using hardware encoding technology, and the encoded image sequence is then distributed.

[0019] Secondly, this application provides an ultra-low latency end-to-end real-time video stream intelligent analysis system, comprising:

[0020] The video stream decoding module is used to decode the video stream in real time using hardware decoding technology;

[0021] The image processing module is used to intelligently identify the decoded video stream using the ADM-YOLOV11 network and combine it with TensorRT batch processing technology to achieve parallel processing of multiple video streams.

[0022] The image rendering module is used to render and generate image sequences based on the intelligent recognition results of the ADM-YOLOV11 network;

[0023] An image sequence encoding module is used to encode the image sequence using hardware encoding technology;

[0024] The video stream distribution module is used to distribute the encoded image sequence.

[0025] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the ultra-low latency end-to-end real-time video stream intelligent analysis method provided above.

[0026] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the ultra-low latency end-to-end real-time video stream intelligent analysis method described above.

[0027] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the ultra-low latency end-to-end real-time video stream intelligent analysis method described above.

[0028] According to the specific embodiments provided in this application, this application has the following technical effects:

[0029] This application provides an ultra-low latency end-to-end real-time intelligent video stream analysis method, system, device, medium, and product. By employing hardware acceleration technology (i.e., hardware decoding and encoding technology), processing speed can be significantly improved, unnecessary I / O operations and data transmission delays can be avoided, thereby solving the problems of high video transmission latency and severe packet loss. By using a highly real-time ADM-YOLOV11 network, targets in the video stream can be detected and segmented quickly and accurately, keeping pace with the processing speed of the decoded image data, ensuring the real-time performance of the entire intelligent video stream analysis, facilitating timely situation monitoring by monitoring personnel, and thus meeting the practical needs of monitoring work in the natural resources field. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart illustrating an ultra-low latency end-to-end real-time video stream intelligent analysis method provided in an embodiment of this application;

[0032] Figure 2 This is a schematic diagram of the CPU drawing process provided in an embodiment of this application;

[0033] Figure 3 This is a schematic diagram of the GPU (CUDA+OpenGL) rendering process provided in an embodiment of this application;

[0034] Figure 4 This is a schematic diagram of the ADM-YOLOv11 network structure provided in an embodiment of this application;

[0035] Figure 5 This is a schematic diagram of an AFE structure provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of the spatial context module structure provided in an embodiment of this application;

[0037] Figure 7 This is a schematic diagram of a feature refinement module provided in an embodiment of this application;

[0038] Figure 8 A schematic diagram of an EMA structure provided in an embodiment of this application;

[0039] Figure 9 This is a schematic diagram of the DySample structure provided in an embodiment of this application;

[0040] Figure 10 This is a schematic diagram of a detection process provided in an embodiment of this application;

[0041] Figure 11 A schematic diagram of the structure of an ultra-low latency end-to-end real-time video stream intelligent analysis system provided in an embodiment of this application;

[0042] Figure 12 This is a schematic diagram of a video stream decoding process provided in an embodiment of this application;

[0043] Figure 13 This is a schematic diagram of an image processing procedure provided in an embodiment of this application;

[0044] Figure 14 A schematic diagram illustrating the multi-stream asynchronous processing process using an event-driven mechanism, provided as an embodiment of this application;

[0045] Figure 15 A schematic diagram illustrating the detection results provided in one embodiment of this application;

[0046] Figure 16 This is a schematic diagram of image sequence encoding provided in an embodiment of this application;

[0047] Figure 17 This is a schematic diagram of video stream distribution provided in an embodiment of this application;

[0048] Figure 18 A flowchart illustrating the implementation of ultra-low latency end-to-end real-time video stream intelligent analysis according to an embodiment of this application;

[0049] Figure 19 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0052] In an exemplary embodiment, the ultra-low latency end-to-end real-time video stream intelligent analysis method provided in this application, such as... Figure 1 As shown, it includes:

[0053] Step 100: Use hardware decoding technology to decode the video stream in real time.

[0054] Step 101: Use the ADM-YOLOV11 network to intelligently identify the decoded video stream, and combine it with TensorRT batch processing technology to achieve parallel processing of multiple video streams.

[0055] Step 102: Generate an image sequence based on the intelligent recognition results of the ADM-YOLOV11 network.

[0056] Step 103: Encode the image sequence using hardware encoding technology and distribute the encoded image sequence.

[0057] By implementing steps 100 to 103 above, this application can significantly improve processing speed, avoid unnecessary IO operations and data transmission delays, thereby solving the problems of high video transmission delays and severe packet loss. It can also quickly and accurately detect and segment targets in the video stream, keep up with the processing speed of the decoded image data, ensure the real-time performance of the entire video stream intelligent analysis, facilitate monitoring personnel to grasp the situation in a timely manner, and thus meet the actual needs of monitoring work in the field of natural resources.

[0058] In another exemplary embodiment of this application, in order to achieve real-time decoding and make the latency at least less than 5ms, in step 100 above, hardware decoding technology based on NVIDIA GPU can be used to decode the video stream in real time.

[0059] For example, compared with traditional CPU decoding, taking the decoding of a 2560x1440x25 FPS real-time H265 video stream, with hardware using an RTX 4060 graphics card and an i7-13700KF CPU, as an example, the comparison results are shown in Table 1.

[0060] Table 1 Decoding Results

[0061]

[0062] In another exemplary embodiment of this application, addressing the technical bottlenecks such as insufficient multi-scale target detection accuracy and low small target recognition rate in dynamic scenes with wide viewing angles in tower videos, this application proposes a target detection algorithm based on multi-module collaborative optimization (i.e., an ADM-YOLOV11 network). This ADM-YOLOV11 network is obtained by improving the YOLOV11 network, such as... Figure 4 As shown, Figure 4In this model, Backbone represents the main network, and Head represents the detection head. The improvements are as follows: First, an Adaptive Feature Enhancement (AFE) module is embedded into the backbone network (i.e., the Backbone network) to deeply reconstruct the C3K2 module. Through spatial context awareness and feature refinement mechanisms, the network's feature extraction capability for complex scenes is significantly improved. Second, an ultra-lightweight dynamic upsampler, DySample (DynamicUpsample), is introduced into the Neck structure to replace the traditional upsampling layer, optimizing the detailed representation of multi-scale features and the efficiency of semantic fusion. Third, an Efficient Multi-scale Attention (EMA) module is integrated into the C3K2 module of the Neck structure to enhance the model's robustness to multi-scale target detection. Finally, an EMASlideLoss classification loss function, which combines exponential moving average and sliding window mechanisms, is used. A dynamic weighting strategy suppresses gradient shift caused by data imbalance, effectively improving the model's generalization performance. Experimental results show that the ADM-YOLOV11 network provided in this application can improve the mAP50-95 from 74.8% of the baseline model to 82.6% (+7.8%), with the mAP50 improving to 96.6% (+4.1%). It is evident that the ADM-YOLOV11 network (hereinafter referred to as the model) can significantly improve the multi-scale target detection accuracy in dynamic scenes of tower videos. Furthermore, the specific improvement methods described above are as follows:

[0063] (1) Improve the backbone network.

[0064] To effectively improve the semantic understanding accuracy of deep learning models in complex scenes and reduce the impact of interference factors such as cluttered backgrounds, dynamic blur, and scale changes on model performance, this application proposes a C3K2_AFE module. This module deeply fuses AFE modules (such as...) Figure 5 As shown in the diagram, the C3K2 model architecture significantly enhances the backbone network's ability to extract and represent multi-scale features. For example, Figure 5 As shown, the dimension is H i ×W i ×C i In the feature map, H i W represents the number of pixels (or feature units) in the vertical direction (height direction) of the input feature map. i C represents the number of pixels (or feature units) in the horizontal (width) direction of the input feature map. i This indicates the number of channels in the input feature map.

[0065] The AFE module achieves performance optimization through two core mechanisms: spatial context awareness and feature refinement (e.g., ... Figure 6 and Figure 7 (As shown). Based on this, the spatial context awareness mechanism is implemented using a Spatial Context Module (SCM). SCM, based on deep mining of image spatial structure information, constructs cross-regional feature dependencies, improving the model's multi-scale detection capability. Its specific data processing flow is as follows:

[0066] 1) First, the input features are processed through a convolutional (Conv) layer to perform preliminary feature extraction and transformation, and adjust the number of channels or extract basic spatial features F.

[0067] 2) Max pooling and average pooling are used to construct a dual-branch structure to mine spatial context at different scales.

[0068] 3) The output features of max pooling and average pooling are obtained by element-wise multiplication / addition (corresponding to...) Figure 6 The feature obtained by fusing multi-scale context is obtained by using the "·" symbol.

[0069] 4) The fused features are activated by the corresponding activation function (corresponding to...) Figure 6 (Activate) to generate attention weights.

[0070] 5) The attention weights are multiplied element-wise with the base spatial features F (or the original input branch features) (corresponding to...). Figure 6 middle The features are weighted and enhanced, so that the features in important regions (high weight) are strengthened and the features in minor regions (low weight) are weakened, resulting in the attention-enhanced feature F′.

[0071] 6) Feature F′ and original input branch ( Figure 6 The characteristics of the direct connection path in the middle and upper part are used to sum the residuals (corresponding to...). Figure 6 middle The components are then combined to obtain the final output.

[0072] Furthermore, a Feature Refinement Module (FRM) is employed to implement a feature refinement mechanism. This module leverages a semantic cue capture mechanism to enhance the feature differences between foreground targets and background regions, effectively improving the expressive power of object detail features. Its specific data processing flow is as follows:

[0073] 1) The basic spatial feature F is processed by variable separable convolution to obtain feature P.

[0074] 2) Upsample feature P to obtain feature Q.

[0075] 3) Obtain feature R directly from the basic spatial feature F through the residual path, and then use variable separation for feature extraction.

[0076] 4) The basic spatial features F are cross-operated to obtain features S, and then variable separation is used for feature extraction.

[0077] 5) The residual branch output, upsampling branch output, and cross branch output are concatenated / elementally added (corresponding to...). Figure 7 The "C" in the figure is used to fuse multi-path features to obtain feature T.

[0078] 6) Process the feature T using a 1×1 convolution (Conv1×1) to obtain the final output feature.

[0079] Based on the above description, the problem of insufficient feature representation in complex scenes of the C3K2 model can be significantly improved, the model's ability to perceive multi-scale targets can be greatly enhanced in the target detection task, and the robustness of the model to background noise can be strengthened.

[0080] (2) Improve the Neck structure.

[0081] To enhance the ADM-YOLOV11 network's ability to perceive multi-scale contextual information, optimize multi-scale feature detection performance, and improve computational efficiency, this application proposes a C3K2_EMA module. For example... Figure 8 As shown, the EMA module is embedded into the C3K2 module within the Neck structure. The EMA module, through a parallelized multi-scale feature processing mechanism, simultaneously captures local details and global semantic features of the image, effectively enhancing the model's robustness to complex scenes. At the network architecture design level, lightweight convolution strategies such as grouped convolution and depthwise separable convolution are employed, significantly reducing the model's floating-point operations (FLOPs) and parameter count. The C3K2_EMA module effectively improves the model's detection accuracy while maintaining computational efficiency.

[0082] The specific data processing flow in the EMA (Efficient Multi-scale Attention) module is as follows:

[0083] 1) Receive a feature map with dimensions h×w×c (h height, w width, c number of channels) as the basic feature to be enhanced.

[0084] 2) Divide the number of channels c into g groups (i.e., the number of channels in each group is c / g) to obtain g sub-feature maps (dimensions h×w×(c / g)).

[0085] 3) Establish pooling branches: X-average pooling, Y-average pooling, and convolution branches (3x3). For the grouped sub-feature maps, perform average pooling along the horizontal (X) and vertical (Y) directions respectively to capture the global context in different directions. For the grouped sub-feature maps, use 3×3 convolution to extract local features and capture small-scale details (such as the edges of objects, texture details, etc.).

[0086] 4) The outputs of pooling and convolution branches are concatenated and then adjusted using a 1×1 convolution. Finally, attention weights are generated by activating the sigmoid function.

[0087] 5) Use the Softmax function to normalize the features, implement attention weighting through matrix multiplication (Matmul), and then stabilize the feature distribution through group normalization (Group Norm).

[0088] 6) Features after multi-branch processing are connected via residuals (corresponding to...) Figure 8 The "+" in the text is related to attention weighting (corresponding to...). Figure 8 The features are integrated using the asterisk (*), and the weight distribution is adjusted using the Sigmoid function to finally output the enhanced features.

[0089] (3) Improve the upsampling layer.

[0090] YOLOv11 employs the traditional nearest-neighbor interpolation upsampling method, which has significant limitations. In complex dynamic scenes, it cannot dynamically adjust the sampling position based on the content of the input feature map, only obtaining fixed sampling point coordinates. This makes it difficult for the model to accurately capture dynamic changes and complex features in the scene. This application innovatively introduces the dynamic upsampling mechanism DySample (i.e., using an ultra-lightweight dynamic upsampling device), which adaptively generates upsampling coordinates based on the content of the input feature map. By accurately calculating the sampling coordinates, the feature map resolution is restored more accurately, effectively recovering detailed information, making the model perform better when processing complex background images. For multi-scale input features at higher levels, the model further enhances the multi-scale detail recovery capability by concatenating or weighted fusing features from different levels.

[0091] Among them, such as Figure 9 As shown, the specific data processing flow of the ultralight dynamic upsampler is as follows:

[0092] 1) For the feature x of input dimension H×W×C, perform channel transformation using a linear layer (corresponding to...) Figure 9 (linear transformation in the model), output dimension H×W×2s 2 Its characteristics.

[0093] 2) For the features after linear transformation (i.e., dimensions H×W×2s) 2The features are numerically scaled (multiplied by 0.25), and pixel rearrangement is used to move the information from the channel dimension to the spatial dimension, outputting the feature O of dimension sH×sW×2 (s is the scaling factor, which controls the change of spatial scale).

[0094] 3) Introduce an additional feature g (dimensions sH×sW×2), and add it element-wise to feature O (corresponding to...). Figure 9 The "+" sign in the figure is used to obtain the fused feature S (dimensions sH×sW×2).

[0095] 4) Using the fused feature S as the sampling grid, perform grid sampling on feature x, and output feature X with dimensions sH×sW×C.

[0096] (4) Optimize the loss function.

[0097] In the construction of the intelligent detection dataset for tower videos, a severe class imbalance problem arose due to differences in sample acquisition difficulty and uneven distribution. During model training, traditional loss functions are easily dominated by majority class samples, resulting in a significant decline in the classification performance of minority class targets. To address this issue, this application adopts the EMASlideLoss classification loss function, which combines the Exponential Moving Average (EMA) and Slide Window mechanisms. The Slide Window mechanism can capture recent loss data in real time, accurately reflecting the dynamic changes in data distribution. The EMA mechanism, through weighted averaging, achieves adaptive adjustment of the loss function, enabling the model to quickly adapt to the dynamic evolution of data distribution. The EMASlideLoss classification loss function differentiates the weights of loss values ​​for different classes, especially increasing the EMA weights for rare classes, prompting the model to focus on difficult-to-classify samples.

[0098] Based on the above description, the detection process of the finally constructed ADM-YOLOV11 network is as follows: Figure 10 As shown. Among them, Figure 10 The standard box in the lower left corner is for illustrative purposes only, and the content above the detection box is not limited.

[0099] In another exemplary embodiment of this application, to further improve detection efficiency and reduce memory usage, TensorRT batch processing technology is used to achieve deep optimization of the ADM-YOLOV11 network for NVIDIA GPUs. Based on this, step 101, which combines TensorRT batch processing technology to achieve parallel processing of multiple video streams, includes:

[0100] 1) When there are n or more video streams, a dynamic batching strategy is used to automatically aggregate multiple video streams into one batch, and TensorRT batch processing technology is used for single batch processing.

[0101] For example, a dynamic batching strategy is adopted. When there are 4 or more input video streams, 4 frames are automatically aggregated into a batch (i.e., BatchSize=4). By utilizing TensorRT's batch processing acceleration capability, the inference time of a single batch is 48ms (equivalent to 12ms per stream), which is 33% more efficient than single-stream processing.

[0102] 2) When the number of video streams is less than n, a buffering and waiting mechanism is adopted to aggregate the video streams within a set time into a batch, and then process them in a single batch using TensorRT batch processing technology.

[0103] For example, for low-concurrency scenarios (less than 4 paths), a caching waiting mechanism (e.g., a timeout threshold of 5ms) is adopted to avoid wasting computing power caused by small batches.

[0104] By combining multi-stream load balancing and using CUDA Stream to achieve parallel processing of multiple batches, and avoiding resource conflicts between streams through event synchronization (CUDAEvent), the end-to-end processing latency of 8 2K streams can be ≤20ms / stream.

[0105] In another exemplary embodiment of this application, to avoid the time consumed by GPU and CPU device I / O and further reduce CPU resource consumption, step 102, the process of drawing and generating an image sequence based on the intelligent recognition results of the ADM-YOLOv11 network, is executed in the image processor. The CPU drawing process and the GPU drawing process are as follows: Figure 2 and Figure 3 As shown in the figure, the drawing results are shown in Table 2.

[0106] Table 2 shows the plotting results.

[0107]

[0108]

[0109] In another exemplary embodiment of this application, to further reduce latency, step 103 can employ NVIDIA GPU-based hardware encoding technology to achieve real-time encoding of the image sequence. Compared to traditional CPU encoding, this example uses encoding a 2560x1440x25 FPS real-time H.265 video stream, with hardware including an RTX 4060 graphics card and an i7-13700KF CPU. The encoding comparison results are shown in Table 3.

[0110] Table 3 Encoding Results

[0111]

[0112] In another exemplary embodiment of this application, to further reduce latency, step 103 can use the WebSocket protocol to actively push the encoded video stream data to the established monitoring end in real time. The monitoring end then plays the received data as an image directly on the web interface. Compared with traditional network transmission protocols, the WebSocket protocol has lower latency: RTMP is approximately 500ms to 1s, HLS / DASH can have a latency of 2s to 10s due to its slicing mechanism, while WebSocket latency is within 200ms. The comparison results are shown in Table 4.

[0113] Table 4 Protocol Transmission Results

[0114]

[0115] Based on the same inventive concept, this application also provides an ultra-low latency end-to-end real-time video stream intelligent analysis system for implementing the ultra-low latency end-to-end real-time video stream intelligent analysis method described above. The solution provided by this system is similar to the implementation scheme described in the above method. Therefore, the specific limitations of one or more ultra-low latency end-to-end real-time video stream intelligent analysis system embodiments provided below can be found in the limitations of the ultra-low latency end-to-end real-time video stream intelligent analysis method described above, and will not be repeated here.

[0116] In one exemplary embodiment, such as Figure 11 As shown, an ultra-low latency end-to-end real-time video stream intelligent analysis system is provided, comprising: a video stream decoding module, an image processing module, an image rendering module, an image sequence encoding module, and a video stream distribution module.

[0117] The video stream decoding module uses hardware decoding technology to decode the video stream in real time. The image processing module uses an ADM-YOLOv11 network to intelligently identify the decoded video stream and combines it with TensorRT batch processing technology to achieve parallel processing of multiple video streams. The image rendering module generates image sequences based on the intelligent identification results of the ADM-YOLOv11 network. The image sequence encoding module encodes the image sequences using hardware encoding technology. The video stream distribution module distributes the encoded image sequences. These modules work collaboratively to achieve ultra-low latency end-to-end real-time intelligent video stream analysis. Compared to traditional CPU-based processing systems, the end-to-end latency of the system provided in this application is reduced from 5-10 seconds to less than 500ms.

[0118] Specifically, the video stream decoding module receives the input video stream, performs real-time decoding based on NVIDIA GPU hardware decoding technology, and transmits the decoded image data to the image processing module. Utilizing the GPU's parallel processing units, it handles a large number of video data decoding tasks simultaneously, significantly improving decoding speed compared to traditional CPU software decoding. The image processing module receives the decoded image data from the video stream decoding module, uses the ADM-YOLOv11 network to process the image to detect and segment target objects, and inputs the detection results to the image rendering module. The image rendering module receives the detection results output by the image processing module, then marks and draws the detected target objects on the image, and transmits the rendered image sequence to the image sequence encoding module.

[0119] As an optional implementation, the process of marking and drawing the detected target objects on the image includes: (1) dividing the 2560×1440 image into 32×32 grids (a total of 80×45 grids), allocating one thread block to each grid, and each thread block contains 4 threads. (2) Each thread is responsible for drawing the detection box of a 16×16 pixel area, rendering the category text, or filling the mask. The category text can be the category text corresponding to the detected target object, used to identify the type of the detected target, such as: car, pedestrian, bicycle, etc. (3) using Billboard technology, pre-rendering the text as a texture map, and mapping it directly to the target position through the GPU, with a single frame text rendering time of <1ms. Billboard technology is a very mature and widely used technology in computer graphics. In mainstream graphics interfaces such as OpenGL and DirectX, Billboard can be directly implemented through shaders or API interfaces, and there are a large number of open source libraries (such as Unity and Unreal Engine) that encapsulate related functions. (4) The detection box drawing result texture or semantic segmentation mask texture is mixed with the original image texture and rendered by the fragment shader to synthesize the result frame. The obtained result frame is obtained by superimposing the drawn result texture onto the corresponding image sequence, and the detection result is drawn onto the original image sequence through texture blending technology.

[0120] The image sequence encoding module receives the drawn image sequence and then encodes it in real time using NVIDIA GPU hardware encoding technology, converting it into a video stream format suitable for transmission. The encoded video stream is then transmitted to the video stream distribution module. The video stream distribution module receives the encoded video stream and then actively pushes the video stream data to the established monitoring terminal using the WebSocket protocol in real time.

[0121] As an optional implementation, in conjunction with the above-described method provided in this application, the working process of the ultra-low latency end-to-end real-time video stream intelligent analysis system (hereinafter referred to as the system) provided in this application includes multiple stages, wherein:

[0122] (I) Video stream decoding stage.

[0123] Working process: When a video stream is input into the system, it first enters the video stream decoding module. This module uses hardware decoding technology based on NVIDIA GPUs to perform real-time decoding of the video stream. The entire video stream decoding process is as follows: Figure 12 As shown.

[0124] Structural Function: NVIDIA GPUs possess powerful parallel computing capabilities, and their hardware decoding architecture can fully utilize the GPU's parallel processing units to handle large amounts of video data decoding tasks simultaneously. Compared to traditional CPU software decoding, this hardware decoding method significantly improves decoding speed.

[0125] Advantages: This hardware decoding technology enables real-time decoding of video streams with a decoding latency of less than 5ms, laying the foundation for the ultra-low latency characteristics of the entire system.

[0126] (II) Image processing stage.

[0127] Working Process: The decoded image queue is then transmitted to the image processing module. In this module, the ADM-YOLOV11 network is used to process the images to detect and segment target objects. Specifically: 1) TensorRT is used to implement the ADM-YOLOV11 network, deeply optimized for NVIDIA GPUs, further improving detection efficiency and reducing memory usage. 2) A dynamic batching strategy is adopted. When the input video stream is greater than or equal to 4 channels (i.e., 4 frames), 4 frames are automatically aggregated into a batch (Batch Size = 4). Utilizing TensorRT's batch processing acceleration capability, the inference time per batch is 48ms (equivalent to 12ms per channel), a 33% improvement in efficiency compared to single-channel processing. For low-concurrency scenarios (less than 4 channels), a caching mechanism (timeout threshold 5ms) is used to avoid wasting computing power due to small batches. 3) Combining multi-stream load balancing, multi-batch parallel processing is achieved through CUDA Streams. Resource conflicts are avoided between streams through event synchronization (CUDAEvent), achieving an end-to-end processing latency of ≤20ms / stream for 8 2K streams. The specific implementation process of steps 1) to 3) is as follows: Figure 13 As shown.

[0128] Structural Role: The lightweight design of the model-engine collaborative architecture—ADM-YOLOv11 network—provides an efficient computational graph for TensorRT batch processing. TensorRT further releases model performance through layer fusion and accuracy optimization, forming a closed loop of algorithm optimization and hardware acceleration. A dynamic batch pipeline is established, automatically adjusting the batch size based on the number of input streams (e.g., buffering for streams below 4, batch aggregation for streams above 4), and implementing asynchronous multi-stream processing through an event-driven mechanism to avoid data blocking.

[0129] Advantages: By combining model optimization and batch processing acceleration, the proportion of total latency in the image processing stage to end-to-end latency is reduced from 30-40% in traditional solutions to 15-20% (total processing latency ≤160ms when 2K streams and 8 concurrent connections are used).

[0130] Based on the above description, the entire image processing flow is as follows: Figure 14 As shown.

[0131] (III) Stage of drawing test results.

[0132] Working Process: The detection results output by the image processing module are transmitted to the image rendering module. In the image rendering module, the detection results from the ADM-YOLOV11 network are rendered on the GPU, that is, the detected target objects are marked and drawn on the image. The implementation process is as follows: 1) The 2560×1440 image is divided into 32×32 grids (80×45 grids in total), with one thread block assigned to each grid. 2) Each thread is responsible for drawing the detection bounding box, rendering text, or filling a mask in a 16×16 pixel area. 3) Using Billboard technology, the text is pre-rendered as a texture map and directly mapped to the target location via the GPU, with a single-frame text rendering time of less than 1ms. 4) The fragment shader blends the texture of the detection bounding box drawing result or the semantic segmentation mask texture with the original image texture to create a composite frame.

[0133] Structural Function: Utilizing CUDA's thread-level parallel architecture, a 2K image is divided into 32×32 pixel grid cells. Each grid cell is allocated an independent thread block, enabling pixel-level parallel processing of bounding box drawing, text rendering, and semantic mask filling. The normalized coordinates output by the object detection model are converted to the original image size. Based on the preset object category name, combined with the detection result category and object confidence score, target category and confidence score text are generated to complete text rendering. A single rendering function can handle the text rendering. Confidence score represents the probability of a detection result being a target of that category. Displayed as text, it provides decision-making reference for managers, distinguishing different targets and indicating the confidence level of each target category. When subsequently pushed to the client, it shows the user the model's confidence level in the current detection result, helping users quickly assess the reliability of the detection. Different color indicators are assigned based on target type (e.g., arable land crops, fire areas) to distinguish different types of target areas, helping users quickly differentiate target categories (e.g., red for "people," green for "vehicles"), especially reducing confusion in multi-target dense scenes. The original image, detection boxes, and semantic masks are each bound to an independent texture. A fragment shader is used to blend them by weights (e.g., 90% original image, 10% detection boxes) to preserve image details while maintaining visual quality. Leveraging OpenGL's hardware acceleration capabilities, intermediate results from CUDA rendering are quickly composited into a complete frame, avoiding latency caused by CPU intervention. The specific implementation of this process is as follows... Figure 15 As shown.

[0134] Advantages: Ultra-low latency and high-efficiency parallel processing, with a single frame processing time of 4-8ms (approximately 5ms for 50 targets), reducing processing time by more than 60% compared to CPU rendering (12-20ms), and completely eliminating IO latency caused by PCIe transmission (accounting for 60-70% of CPU rendering time). It possesses multi-stream processing capabilities; a single RTX 4060 supports 8-10 parallel rendering channels of 2K streams, while a CPU can only handle 3-4 channels. This application can meet the concurrency requirements of large-scale monitoring scenarios.

[0135] (iv) Image sequence encoding stage.

[0136] Working process: The drawn image sequence enters the image sequence encoding module. This module employs NVIDIA GPU-based hardware encoding technology to encode the image sequence in real time, converting it into a video stream format suitable for transmission. The specific implementation process of this encoding is as follows: Figure 16 As shown.

[0137] Function of the structure: NVIDIA GPU hardware encoding technology utilizes its dedicated encoding units to efficiently compress and encode image sequences. This hardware encoding method can complete the encoding process quickly while ensuring video quality.

[0138] Advantages: It enables real-time encoding of image sequences, further reducing latency and allowing the entire system to proceed quickly and smoothly from decoding to encoding, ensuring the real-time performance of the video stream.

[0139] (v) Video stream distribution stage.

[0140] Working process: The encoded video stream is actively pushed in real time to the established monitoring terminal via the video stream distribution module using the WebSocket protocol. Upon receiving the data, the monitoring terminal directly displays it as an image on a web page for playback. This implementation process is as follows: Figure 17 As shown.

[0141] Function and Structure: The WebSocket protocol is a bidirectional communication protocol that enables the establishment of a persistent connection between the client and server, allowing for real-time bidirectional data transmission. This protocol is well-suited for transmitting real-time video streams, ensuring fast and stable transmission of video stream data to the monitoring end.

[0142] Advantages: By using the WebSocket protocol to achieve real-time active streaming of encoded video stream data, the monitoring end can receive and play the video stream data in a timely manner, avoiding the delay and stuttering phenomena in the data transmission process of traditional communication protocols, and further ensuring the ultra-low latency characteristics of the entire system.

[0143] Based on the above description, the overall process of implementing the ultra-low latency end-to-end real-time video stream intelligent analysis method provided in this application using the ultra-low latency end-to-end real-time video stream intelligent analysis system is as follows: Figure 18 As shown. Compared with the prior art, the solution provided in this application has the following advantages:

[0144] 1. Ultra-low latency.

[0145] (1) Full-process hardware acceleration: Each module of the system provided in this application adopts hardware acceleration technology based on NVIDIA GPU, such as hardware decoding and hardware encoding. By utilizing the powerful parallel computing capabilities and dedicated hardware units of the GPU, the processing speed can be greatly improved, and unnecessary IO operations and data transmission delays can be avoided.

[0146] (2) Algorithm model optimization: The ADM-YOLOV11 network has strong real-time performance, which can quickly and accurately detect and segment targets in the video stream, keep up with the processing speed of the decoded image data, and thus ensure the real-time performance of the entire system.

[0147] (3) Significantly reduced end-to-end latency: Compared with traditional CPU-based processing systems, the end-to-end latency of this application is reduced from 5-10 seconds to less than 500ms, which can meet the needs of application scenarios with extremely high real-time requirements, such as early warning and timely decision-making in natural resource monitoring.

[0148] II. Efficient resource utilization.

[0149] (1) Rational allocation of CPU and GPU resources: During the image rendering stage, this application places the rendering work on the GPU, reducing the occupation of CPU resources, so that the CPU and GPU can perform their respective functions and give full play to their advantages. The CPU can be used to handle other more important tasks, such as system scheduling and data management, thereby improving the overall resource utilization of the system.

[0150] (2) Full utilization of GPU resources: The GPU's decoding, detection, rendering, and encoding units work independently or collaboratively, which can avoid resource idleness and waste, and achieve efficient utilization of resources while ensuring high-performance processing.

[0151] III. Powerful real-time processing capabilities.

[0152] (1) High-precision real-time detection and segmentation: The ADM-YOLOV11 network, combined with the powerful computing capabilities of the GPU, can quickly and accurately detect and segment targets in the video stream. In farmland monitoring, it can accurately identify different crops and non-grain crops, as well as illegal non-agricultural construction activities. In forest fire monitoring, it can quickly and accurately detect fire sources and identify the fire range, ensuring the real-time performance and accuracy of video analysis.

[0153] (2) Multi-stream parallel processing capability: Each module is based on GPU hardware acceleration technology and has powerful multi-stream processing capability. For example, a single RTX4060 can simultaneously decode and encode 6-8 channels of 2K / 25FPSH.265 streams, which can meet the needs of multi-channel video monitoring and real-time analysis, and is suitable for large-scale natural resource monitoring scenarios.

[0154] IV. Stable video transmission.

[0155] (1) Advantages of WebSocket Protocol: This application uses the WebSocket protocol for video stream distribution, which can realize real-time active data push. This protocol has lower latency, and can customize retransmission and forward error correction (FEC) in weak network environments. It can adapt to complex and unstable network environments, ensuring that the video stream can be stably and quickly transmitted to the monitoring end, improving the reliability of the system and the user experience.

[0156] (2) Low and stable transmission latency: Compared with traditional network transmission protocols, the WebSocket protocol used in this application has a latency of less than 200ms, which enables monitoring personnel to receive alarms and monitoring screens of abnormal situations in a timely manner, so as to grasp the situation in a timely manner and take countermeasures.

[0157] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 19 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores ultra-low latency end-to-end real-time video stream intelligent analysis data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an ultra-low latency end-to-end real-time video stream intelligent analysis method.

[0158] Those skilled in the art will understand that Figure 19 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0159] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0160] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0161] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0162] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0163] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Furthermore, any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory.

[0164] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0165] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0166] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for intelligent analysis of ultra-low latency end-to-end real-time video streams, characterized in that, include: Hardware decoding technology is used to decode the video stream in real time; The ADM-YOLOV11 network is used to intelligently identify the decoded video stream, and TensorRT batch processing technology is combined to achieve parallel processing of multiple video streams. Image sequences are generated based on the intelligent recognition results of the ADM-YOLOV11 network; The image sequence is encoded using hardware encoding technology, and the encoded image sequence is then distributed.

2. The ultra-low latency end-to-end real-time video stream intelligent analysis method according to claim 1, characterized in that, The hardware decoding technology is based on NVIDIA GPUs.

3. The ultra-low latency end-to-end real-time video stream intelligent analysis method according to claim 1, characterized in that, The ADM-YOLOV11 network is obtained by improving the YOLOV11 network, including: An adaptive feature enhancement module is embedded in the backbone network, and the C3K2 module is deeply reconstructed through spatial context awareness and feature refinement mechanisms. In the Neck structure, an ultra-lightweight dynamic upsampling layer is used to replace the upsampling layer, and a multi-scale attention module is integrated into the C3K2 module of the Neck structure. The loss function is the EMASlideLoss classification loss function, which is a loss function that combines the exponential moving average mechanism and the sliding window mechanism.

4. The ultra-low latency end-to-end real-time video stream intelligent analysis method according to claim 1, characterized in that, In the process of achieving parallel processing of multiple video streams by combining TensorRT batch processing technology, the following are included: When there are n or more video streams, a dynamic batching strategy is used to automatically aggregate multiple video streams into one batch, and TensorRT batch processing technology is used for single batch processing. When the number of video streams is less than n, a buffering and waiting mechanism is adopted to aggregate the video streams within a set time into a batch, and then process them in a single batch using TensorRT batch processing technology.

5. The ultra-low latency end-to-end real-time video stream intelligent analysis method according to claim 1, characterized in that, In the image processor, the intelligent recognition results based on the ADM-YOLOV11 network are used to generate image sequences.

6. The ultra-low latency end-to-end real-time video stream intelligent analysis method according to claim 1, characterized in that, The encoded image sequence is distributed using the WebSocket protocol.

7. An ultra-low latency end-to-end real-time video stream intelligent analysis system, characterized in that, include: The video stream decoding module is used to decode the video stream in real time using hardware decoding technology; The image processing module is used to intelligently identify the decoded video stream using the ADM-YOLOV11 network and combine it with TensorRT batch processing technology to achieve parallel processing of multiple video streams. The image rendering module is used to render and generate image sequences based on the intelligent recognition results of the ADM-YOLOV11 network; An image sequence encoding module is used to encode the image sequence using hardware encoding technology; The video stream distribution module is used to distribute the encoded image sequence.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that the processor executes the computer program to implement the ultra-low latency end-to-end real-time video stream intelligent analysis method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the ultra-low latency end-to-end real-time video stream intelligent analysis method as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the ultra-low latency end-to-end real-time video stream intelligent analysis method as described in any one of claims 1-6.

Images