Video stream cloud-edge collaborative analysis method and device

Abstract

This application proposes a video stream cloud-edge collaborative analysis method and device. The method includes: an FPGA edge computing device acquiring a video stream captured by a camera device; the FPGA edge computing device performing real-time key region detection on the video stream using an internally deployed lightweight neural network model to obtain key region detection results and extracting low-resolution feature vectors of the key regions; the FPGA edge computing device, through its built-in video encoding and decoding unit, performing lossless encoding and high bit rate compression on the key region video stream based on the key region detection results, and performing Gaussian blur preprocessing and high quantization parameter compression on the non-key region video stream to obtain compressed video data; and uploading the compressed video data, key region location metadata, and low-resolution feature vectors to a cloud GPU server. This can improve real-time performance and network efficiency.

Description

Technical Field

[0001] This application relates to the fields of cloud computing and edge computing technology, and in particular to a video stream cloud-edge collaborative analysis method and device. Background Technology

[0002] In the context of intelligent upgrading in the coal industry, underground video monitoring systems have become a core infrastructure for safety management. These systems can improve mine safety control by monitoring the working environment, equipment status, and personnel behavior in real time. However, with the large-scale deployment of high-definition cameras, the amount of underground video data is growing exponentially, leading to severe bandwidth pressure on video transmission channels and affecting the monitoring efficiency of the video monitoring system. Summary of the Invention

[0003] This application aims to at least partially address one of the technical problems in the related art.

[0004] The first aspect of this application proposes a video stream cloud-edge collaborative analysis method, applied to a video stream cloud-edge collaborative analysis system, wherein the video stream cloud-edge collaborative analysis system includes an FPGA edge computing device and a cloud GPU server; the video stream cloud-edge collaborative analysis method includes: The FPGA edge computing device acquires the video stream captured by the camera device; The FPGA edge computing device uses a lightweight neural network model deployed internally to perform real-time key region detection on the video stream, obtain key region detection results, and extract low-resolution feature vectors of the key regions. The FPGA edge computing device, through its built-in video encoding and decoding unit, performs lossless encoding and high bit rate compression on the video stream of the key area based on the key area detection results, and performs Gaussian blur preprocessing and high quantization parameter compression on the video stream of the non-key area to obtain compressed video data. The compressed video data, the location metadata of the key area, and the low-resolution feature vector are uploaded to the cloud GPU server in a coordinated manner, so that the cloud GPU server can perform data analysis based on the compressed video data, the location metadata of the key area, and the low-resolution feature vector.

[0005] A second aspect of this application provides a video stream cloud-edge collaborative analysis device, comprising: The data acquisition module is used by the FPGA edge computing device to acquire the video stream captured by the camera device; The detection module is used by the FPGA edge computing device to perform real-time key region detection on the video stream through an internally deployed lightweight neural network model, obtain key region detection results, and extract low-resolution feature vectors of the key regions. The compression module is used by the FPGA edge computing device to compress the video stream of the key area using lossless encoding mode and high bit rate according to the key area detection results through the built-in video encoding and decoding unit, and to preprocess the video stream of the non-key area using Gaussian blur training and high quantization parameters to obtain compressed video data. The collaborative upload module is used to collaboratively upload the compressed video data, the location metadata of the key area, and the low-resolution feature vector to the cloud GPU server, so that the cloud GPU server can perform data analysis based on the compressed video data, the location metadata of the key area, and the low-resolution feature vector.

[0006] A third aspect of this application provides a video stream cloud-edge collaborative analysis device, comprising: a processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any of the first aspects above.

[0007] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method described in any of the first aspects above.

[0008] A fifth aspect of this application provides a computer program product including a computer program that, when executed by a processor, implements the method as described in any of the first aspects above.

[0009] In the embodiments of this application, the video stream captured by the camera device is acquired through the FPGA edge computing device; the FPGA edge computing device performs real-time key region detection on the video stream using an internally deployed lightweight neural network model, obtains key region detection results, and extracts low-resolution feature vectors of the key regions; the FPGA edge computing device, through its built-in video encoding and decoding unit, performs lossless encoding and high bit rate compression on the key region video stream based on the key region detection results, and performs Gaussian blur preprocessing on the non-key region video stream and compression using high quantization parameters to obtain compressed video data; the compressed video data, the location metadata of the key regions, and the low-resolution feature vectors are collaboratively uploaded to the cloud GPU server, so that the cloud GPU server can perform data analysis based on the compressed video data, the location metadata of the key regions, and the low-resolution feature vectors. In this way, by performing real-time key area detection and video encoding / decoding on FPGA edge computing devices, and combining the powerful computing capabilities of cloud GPU servers for in-depth data analysis, efficient and low-latency video stream processing and analysis can be achieved. At the same time, while ensuring the high fidelity of the video stream in key areas, the data transmission bandwidth requirements can be significantly reduced, improving the overall performance and reliability of the system. This can effectively improve the real-time performance and network efficiency of the system while ensuring the accuracy of mine video monitoring and analysis.

[0010] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0011] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 A flowchart illustrating a video stream cloud-edge collaborative analysis method provided in an embodiment of this application; Figure 2 This application provides a flowchart illustrating a video stream cloud-edge collaborative analysis system as an embodiment of the present application. Figure 3 This is a schematic diagram of the structure of a video stream cloud-edge collaborative analysis device provided in an embodiment of this application. Detailed Implementation

[0012] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0013] In the process of intelligent upgrading of the coal industry, underground video monitoring systems have become a core infrastructure for safety production management. By monitoring the working environment, equipment status, and personnel behavior in real time, they effectively improve the level of mine safety management. However, with the large-scale deployment of high-definition cameras, the amount of underground video data is growing exponentially, leading to severe bandwidth pressure on existing transmission channels. Traditional video coding technologies (such as H.264 / H.265) have significant performance bottlenecks when directly processing such massive amounts of data: they struggle to achieve high compression ratios while maintaining acceptable visual quality, and strong compression easily causes distortion of details in critical areas (such as equipment and instrument readings, and personnel posture), failing to meet the dual requirements of real-time transmission and accurate analysis.

[0014] To overcome resource constraints, relevant technologies mainly include two approaches: enhanced coding architecture and adaptive bitrate mechanisms. Enhanced coding architecture improves compression efficiency by refining rate-distortion optimization models or introducing deep learning modules (such as inter-frame prediction and transform-domain quantization based on CNNs). While this method achieves breakthroughs in rate-distortion performance, its reliance on complex neural network structures (such as 3D-CNNs and Transformers) leads to a surge in computational complexity, making real-time encoding difficult to achieve with limited computing resources at the mine edge. Adaptive bitrate mechanisms use machine learning algorithms to identify key semantic regions in the video (such as moving targets and dangerous areas), and reduce the coding precision of non-ROI (Region of Interest) regions (key regions) through non-uniform bit allocation strategies to save bitrate. However, this approach suffers from insufficient ROI detection accuracy and low energy efficiency of the deployment platform. Alternatively, in related technologies, methods based on motion vectors or manual features have poor robustness in low-light and multi-obstructed environments underground, resulting in a high rate of missed detections in critical areas; while mainstream computing solutions rely on GPU (Graphics Processing Unit) platforms to run neural network models, whose high power consumption (usually >150W), large size and heat dissipation requirements seriously conflict with the stringent deployment conditions of edge devices in explosion-proof and confined spaces in mines.

[0015] In related technologies, to address the issue of excessively high power consumption of computing units that does not meet the requirements of equipment used in underground mines, FPGAs (Field-Programmable Gate Arrays) can be used to overcome the power consumption limitations of computing devices. FPGA hardware devices, with their programmability, high parallel computing capabilities, and ultra-low power consumption (typically 5-30W), exhibit unique advantages in edge computing scenarios. However, FPGA-based video encoding and inference schemes still have limitations. Most directly replace a single encoder module with a neural network (such as intra-frame prediction or entropy coding). To achieve compression performance exceeding traditional algorithms, computationally intensive network structures are forced to be designed, exceeding the FPGA computing power limits of edge mining equipment. Furthermore, there is a lack of hardware-software synergy in the ecosystem; this FPGA solution does not construct a complete "ROI detection-adaptive encoding-image quality enhancement" closed loop, failing to guarantee visual recognizability of critical areas at low bitrates.

[0016] In summary, video stream compression and inference technologies for mining scenarios face three major contradictions: 1. The contradiction between the demand for high compression ratios and limited computing power: making it difficult to deploy complex neural network models in real time at the edge. 2. The contradiction between accurate ROI identification and environmental interference: complex lighting and dust interference in underground environments hinder traditional detection methods. 3. The contradiction between advanced algorithms and hardware adaptability: the high power consumption and large size of GPUs cannot meet the physical constraints of mining edge devices.

[0017] Based on this, embodiments of this application provide a cloud-edge collaborative analysis method and device for video streams. Addressing the problems of huge transmission bandwidth requirements, high end-to-end latency, and the limitations of pure edge computing in achieving high-precision inference due to device computing power constraints caused by pure cloud-based video analysis, a cloud-edge collaborative processing architecture is constructed. At the edge, an efficient video processing pipeline is built using an FPGA edge computing device. Its integrated Video Coding Unit (VCU) performs hardware-level real-time processing on the video stream captured by the original camera (video recording device) and deploys a lightweight neural network model for preliminary inference analysis. Simultaneously, the inference results of the lightweight model (keyframe index and low-resolution features) and the video data efficiently compressed by the VCU are collaboratively uploaded to the cloud. In the cloud, the powerful computing power of the cloud GPU server is used to perform high-precision, high-quality deep inference analysis on the received compressed video stream and execute super-resolution reconstruction to improve the visual effect of the video. Through the aforementioned collaborative mechanism, while significantly reducing the amount of data transmitted between the cloud and the edge and the subsequent video distribution traffic, it can achieve inference capabilities superior to pure edge computing, effectively overcoming the high latency bottleneck of pure cloud processing. Thus, it can effectively improve the real-time performance and network efficiency of the system while ensuring the accuracy of mine video monitoring and analysis.

[0018] The video stream cloud-edge collaborative analysis method and device according to embodiments of this application are described below with reference to the accompanying drawings.

[0019] Figure 1 This is a flowchart illustrating a video stream cloud-edge collaborative analysis method provided in an embodiment of this application. The method is applied to a video stream cloud-edge collaborative analysis system, which includes an FPGA edge computing device and a cloud GPU server. Figure 1 As shown, the video stream cloud-edge collaborative analysis method includes the following steps: S101, the FPGA edge computing device acquires the video stream captured by the camera device.

[0020] In embodiments of this application, the FPGA edge computing device can receive real-time video streams captured by a camera device (e.g., a webcam) through its input interface. For example, the FPGA edge computing device can receive raw mine camera video streams via a PCIe interface (PCI Express Interface), ensuring high-speed and stable data transmission.

[0021] The S102 FPGA edge computing device uses an internally deployed lightweight neural network model to perform real-time key region detection on the video stream, obtain key region detection results, and extract low-resolution feature vectors of the key regions.

[0022] In the embodiments of this application, a lightweight neural network model is deployed inside the FPGA edge computing device. This lightweight neural network model can adopt the simplified MobileNetV3 architecture. This lightweight neural network model can process video streams in real time, detect key regions in the video (such as equipment and instrument areas, personnel posture areas, etc.), and obtain key region detection results. At the same time, the lightweight neural network model can also extract low-resolution feature vectors of these key regions for subsequent analysis and processing.

[0023] In the embodiments of this application, the lightweight neural network model can adopt the MobileNetV3 simplified architecture and can be trained using adversarial sample augmentation strategies. Furthermore, the inference engine of the lightweight neural network model can employ HLS parallel computing units, and the computation process is controlled through inter-layer pipelines and data reuse mechanisms. For example, the MobileNetV3 simplified architecture is a deep learning architecture optimized for mobile and edge devices. It can significantly reduce the computational load and number of parameters of the model while maintaining high accuracy by using techniques such as depthwise separable convolutions, thereby improving the running efficiency of the model (lightweight neural network model) on resource-constrained edge devices. During training, adversarial sample augmentation strategies can be introduced. By generating adversarial samples and incorporating them into the training dataset, the model can learn more robust feature representations, thereby improving its adaptability to complex environments and potential interference, and enhancing the stability and accuracy of the model in practical applications. Meanwhile, the inference engine can utilize parallel computing units implemented with HLS (High-Level Synthesis) technology. HLS technology allows the use of high-level languages ​​to describe hardware logic, thereby simplifying the hardware design process and improving development efficiency. Parallel computing units can handle multiple computational tasks simultaneously, significantly improving inference speed and meeting real-time requirements. Furthermore, to further optimize computational efficiency, the inference engine can employ inter-layer pipelines and data reuse mechanisms. Inter-layer pipelines can distribute the operations of different layers of the neural network across multiple processing stages, enabling parallel execution of each stage and reducing overall computation time. Data reuse mechanisms can reduce hardware resource consumption by rationally arranging data storage and transmission, avoiding redundant computations and unnecessary data movement.

[0024] The S103 FPGA edge computing device uses its built-in video encoding and decoding unit to perform lossless encoding and allocate a high bit rate to the video stream of the key area based on the key area detection results. It also performs Gaussian blur preprocessing on the video stream of the non-key area and high quantization parameter compression processing to obtain compressed video data.

[0025] In embodiments of this application, the FPGA edge computing device can also perform differential processing on the video stream based on the key region detection results using its built-in video codec unit (VCU). For example, differential processing of the video stream may include: for the video stream in the key region, using a lossless encoding mode and allocating a high bit rate for compression processing to ensure high fidelity of the key information; for the video stream in the non-key region, Gaussian blur preprocessing and high quantization parameters can be used for compression processing to reduce data volume and transmission bandwidth requirements.

[0026] S104 uploads the compressed video data, key area location metadata, and low-resolution feature vectors to the cloud GPU server so that the cloud GPU server can perform data analysis based on the compressed video data, key area location metadata, and low-resolution feature vectors.

[0027] In the embodiments of this application, the FPGA edge computing device can upload processed compressed video data, key area location metadata, and low-resolution feature vectors to a cloud GPU server via a network. After receiving this data, the cloud GPU server can perform further data analysis based on the compressed video data, key area location metadata, and low-resolution feature vectors, such as deep inference and event recognition, to extract more valuable information.

[0028] In the embodiments of this application, the video stream captured by the camera device is acquired through the FPGA edge computing device; the FPGA edge computing device performs real-time key region detection on the video stream using an internally deployed lightweight neural network model, obtains key region detection results, and extracts low-resolution feature vectors of the key regions; the FPGA edge computing device, through its built-in video encoding and decoding unit, performs lossless encoding and high bit rate compression on the key region video stream based on the key region detection results, and performs Gaussian blur preprocessing on the non-key region video stream and compression using high quantization parameters to obtain compressed video data; the compressed video data, the location metadata of the key regions, and the low-resolution feature vectors are collaboratively uploaded to the cloud GPU server, so that the cloud GPU server can perform data analysis based on the compressed video data, the location metadata of the key regions, and the low-resolution feature vectors. In this way, by performing real-time key area detection and video encoding / decoding on FPGA edge computing devices, and combining the powerful computing capabilities of cloud GPU servers for in-depth data analysis, efficient and low-latency video stream processing and analysis can be achieved. At the same time, while ensuring the high fidelity of the video stream in key areas, the data transmission bandwidth requirements can be significantly reduced, improving the overall performance and reliability of the system. This can effectively improve the real-time performance and network efficiency of the system while ensuring the accuracy of mine video monitoring and analysis.

[0029] In some possible implementations, the key region detection results are output to the video encoding / decoding unit in the form of a binary mask.

[0030] In the embodiments of this application, a lightweight neural network model processes the input video stream in real time, identifies key region detection results in the video, and outputs them in the form of a binary mask. For example, the binary mask is a two-dimensional array with the same resolution as the input video stream, where each pixel has a value of 0 or 1. For instance, a pixel with a value of 1 indicates that the location belongs to a key region, and a pixel with a value of 0 indicates that the location does not belong to a key region. This form of output is concise and clear, facilitating subsequent processing by the video encoding / decoding unit.

[0031] In some possible implementations, when the video stream of the key area is compressed using a lossless coding mode and a high bit rate, the quantization parameter is less than or equal to a first threshold. When training a Gaussian blur preprocessing of a video stream in a non-critical area and then compressing it with high quantization parameters, the quantization parameters are greater than or equal to the second threshold; the first threshold is less than the second threshold.

[0032] In the embodiments of this application, when processing video streams of key regions, in order to ensure the high fidelity of the video stream information in the key regions, a lossless encoding mode and a higher bit rate can be used for compression processing. The quantization parameter (QP) is an important parameter that controls the degree of data compression during the encoding process. A lower quantization parameter value means less data compression and higher video quality. Therefore, the quantization parameter of the video stream in the key regions is set to be less than or equal to a lower first threshold (e.g., 22) to ensure the video quality of the key regions. When compressing video streams in non-critical areas, Gaussian blur preprocessing (e.g., using a 3×3 convolution kernel) can first be performed to reduce detail and noise in these areas, thus reducing the data volume. Then, higher quantization parameters can be used for compression. Higher quantization parameter values ​​mean more data compression and lower video quality, significantly reducing the data volume in non-critical areas. Therefore, the quantization parameter for non-critical area video streams is set to be greater than or equal to a higher second threshold (e.g., 45). This differentiated processing significantly reduces the data volume of non-critical area video streams while maintaining high fidelity in critical area video streams. This not only reduces transmission bandwidth requirements but also improves the overall system performance and reliability, effectively balancing video quality and data transmission efficiency to ensure accurate transmission and processing of critical information.

[0033] In some possible implementations, before the FPGA edge computing device performs real-time key region detection on the video stream using an internally deployed lightweight neural network model, obtains the key region detection results, and extracts the low-resolution feature vectors of the key regions, it further includes: The FPGA edge computing device integrates a dynamic bit rate control mechanism to receive network load data fed back from the cloud GPU server; When network load data indicates network congestion, expand the scope of non-critical areas, increase quantization parameters, and enhance Gaussian blur intensity. When network load data indicates that the network has returned to stability, shrink the boundaries of critical areas and reduce quantization parameters.

[0034] In embodiments of this application, the FPGA edge computing device can also integrate a dynamic bitrate adjustment mechanism. This mechanism can receive network load data from a cloud GPU server, including but not limited to bandwidth usage, packet loss rate, and latency, for real-time monitoring of network status. When network load data indicates network congestion, the dynamic bitrate adjustment mechanism automatically adjusts the encoding parameters of the video stream. For example, it can expand the range of non-critical areas, increase quantization parameters (increase compression), and enhance Gaussian blur intensity to reduce the amount of data in non-critical areas, thereby reducing the overall transmission bitrate and alleviating network congestion. When network load data indicates that the network status has stabilized, the dynamic bitrate adjustment mechanism adjusts the encoding parameters to optimize video quality. For example, it can shrink the boundaries of critical areas, reduce the range of non-critical areas, and reduce quantization parameters (reduce compression) to improve the video quality of critical areas. In this way, by integrating a dynamic bitrate adjustment mechanism, the encoding parameters of the video stream can be dynamically adjusted according to the real-time network status, thereby not only improving the adaptability and stability of the system but also ensuring high-fidelity transmission of critical information under different network conditions.

[0035] In some possible implementations, after the compressed video data, key region location metadata, and low-resolution feature vectors are collaboratively uploaded to a cloud GPU server, the process also includes: The cloud-based GPU server uses the parallel computing capabilities of the GPU to perform hardware decoding on the compressed video data, resulting in decoded video frames. The decoded video frames are processed by region segmentation based on the metadata of the key regions. The region segmentation process includes: reconstructing the key regions using a super-resolution reconstruction model to obtain key region images; and reconstructing the non-key regions using the original compressed resolution to obtain non-key region images. The super-resolution reconstruction model is implemented based on a generative adversarial network.

[0036] In the embodiments of this application, the cloud GPU server can utilize its powerful parallel computing capabilities to perform hardware decoding of compressed video data uploaded by the FPGA edge computing device. For example, this process can leverage the multi-core architecture of the GPU to efficiently decode the compressed video data into original video frames, providing a foundation for subsequent analysis and processing. The decoded video frames can be processed by region based on the location metadata of key regions. This key region metadata provides specific location information of the key regions, enabling the cloud GPU server to accurately identify and distinguish between key and non-key regions. For example, for key regions, the cloud GPU server can use a super-resolution reconstruction model to reconstruct the key region image. The super-resolution reconstruction model can be implemented based on a generative adversarial network (GAN), capable of reconstructing low-resolution key region images into high-resolution images, thereby improving the visual quality and detail of the key regions. For non-key regions, the cloud GPU server can use the original compressed resolution for reconstruction, reconstructing the non-key region image. Since non-key regions are of lower importance, high-resolution reconstruction is unnecessary, thus saving computing resources and processing time. In this way, by performing hardware decoding, regional processing, and super-resolution reconstruction on the cloud GPU server, compressed video data uploaded by FPGA edge computing devices can be processed efficiently. This collaborative processing mechanism of regional processing can not only improve the processing quality of video data, but also significantly improve the real-time performance and reliability of the system.

[0037] In some possible implementations, it also includes: The cloud-based GPU server uses a deep inference model to perform fusion analysis based on key region images, low-resolution feature vectors, and fine-grained features extracted by the deep inference model to obtain structured analysis results. The structured analysis results include at least one of the following: event type, location coordinates, and confidence level. Output the structured analysis results.

[0038] In the embodiments of this application, the cloud GPU server can also utilize a deep inference model to perform fusion analysis on key area images, low-resolution feature vectors (such as target contours and motion trends), and fine-grained features extracted by the model itself (such as instrument pointer angles and personnel protective equipment status). Through fusion analysis, the deep inference model can generate structured analysis results. The structured analysis results can be presented in a standardized and easy-to-understand format for easy further processing and application. The structured analysis results can include key information such as event type, location coordinates, and confidence level, which can directly reflect the important features and events of the video content. Finally, the structured analysis results can be output, for example, in the form of data files, real-time alarms, or visualization interfaces, depending on the needs of the application scenario. In this way, by performing deep inference and structured analysis on the cloud GPU server, this method can efficiently process and analyze video data uploaded from FPGA edge computing devices. Fusion analysis can ensure the accuracy and comprehensiveness of the analysis results, while the generation of structured analysis results allows this information to be directly applied to practical applications. Thus, not only can the efficiency and accuracy of video analysis be improved, but the real-time performance and reliability of the system can also be enhanced.

[0039] In some possible implementations, at least one of the following is also included: When a high-risk event is detected in consecutive video frames, the FPGA edge computing device increases the transmission priority of the critical area associated with the high-risk event. In the event of network congestion, FPGA edge computing devices temporarily amplify the blur intensity and quantization parameters of non-critical areas; The cloud-based GPU server continuously adjusts the model parameters of the lightweight neural network model of the FPGA edge computing device through a reinforcement learning framework, and then sends the model parameters to the FPGA edge computing device.

[0040] In the embodiments of this application, when the FPGA edge computing device detects a high-risk event in consecutive video frames using a lightweight neural network model, it can automatically increase the transmission priority of key areas related to the high-risk event. This ensures that data in key areas related to the high-risk event is processed and transmitted preferentially, guaranteeing timely identification and response to the high-risk event. When network congestion occurs, the FPGA edge computing device temporarily adjusts the encoding parameters of non-critical areas. Specifically, this may include increasing the intensity of Gaussian blur and increasing quantization parameters. These adjustments reduce the amount of data in non-critical areas, thereby freeing up network bandwidth and ensuring smooth transmission of data in critical areas, alleviating network congestion. The cloud GPU server can also utilize a reinforcement learning framework to continuously adjust the model parameters of the lightweight neural network model deployed on the FPGA edge computing device based on feedback from network status and inference quality. The adjusted model parameters can be sent back to the FPGA edge computing device via the network to optimize model performance and improve the accuracy and efficiency of key area detection.

[0041] In some possible implementations, at least one of the following is also included: The cloud-based GPU server captures end-to-end transmission status from the network interface card (NIC) hardware layer and combines this with the analysis quality metrics output by the deep inference module of the cloud-based GPU server to construct a multi-dimensional evaluation model. This multi-dimensional evaluation model employs a weighted decision-making mechanism to ensure lossless encoded stream transmission in critical areas during network congestion periods, while suspending the upload of non-critical background frames. When a high-risk event is identified, the FPGA edge computing device is forced to increase the encoding priority of the critical areas associated with the high-risk event to the highest level and enables forward error correction redundancy protection. The analysis quality metrics are output by the cloud-based server's deep inference module after performing deep analysis on the compressed video stream uploaded by the FPGA edge computing device, key area location metadata, and low-resolution feature vectors. The FPGA edge computing device deploys reinforcement learning algorithms, and uses the network state matrix and inference quality decay rate as the environment state, the coding parameter adjustment range as the action space, and trains the optimal control strategy through a long-term reward function to realize the coding parameter adjustment process of the video codec unit.

[0042] In the embodiments of this application, the cloud GPU server can capture end-to-end transmission status through the network card hardware layer, including key indicators such as bandwidth utilization, packet loss rate, and latency. Simultaneously, it can combine the analysis quality indicators output by the deep inference module, such as the confidence level of key target identification and the missed detection rate of consecutive frame events, to construct a multi-dimensional evaluation model. This model can comprehensively evaluate the current network status and video stream quality, providing a basis for subsequent decision-making. For example, the multi-dimensional evaluation model can employ a weighted decision mechanism, dynamically adjusting encoding and transmission strategies based on different network statuses and video stream quality indicators. For instance, during network congestion, priority is given to ensuring the transmission of lossless encoded streams in critical areas to guarantee the high fidelity of critical information; simultaneously, the uploading of non-critical background frames is suspended to reduce unnecessary data transmission and alleviate network congestion. When the deep inference module identifies high-risk events, the system forces the FPGA edge computing device to increase the encoding priority of critical areas related to these high-risk events to the highest level. At the same time, forward error correction (FEC) redundancy protection processing is enabled to ensure that even in the event of network packet loss, data in critical areas can be accurately recovered, improving the robustness of the system. The quality metrics can be generated by performing deep analysis on the compressed video stream, key region location metadata, and low-resolution feature vectors uploaded by the FPGA edge computing device using the deep inference module on a cloud server. These metrics reflect the quality of the video stream and the completeness of key information, providing important input data for the multi-dimensional evaluation model.

[0043] FPGA edge computing devices can also deploy reinforcement learning algorithms. These algorithms use the network state matrix and inference quality decay rate as the environment state, and the adjustment range of encoding parameters as the action space. Through a long-term reward function, the algorithm trains the optimal control strategy, dynamically adjusting the encoding parameters of the video codec unit to adapt to different network conditions and video content changes, ensuring the overall performance and efficiency of the system.

[0044] In this way, through advanced collaborative optimization of cloud-based GPU servers and FPGA edge computing devices, the system can dynamically adapt to complex network conditions and changes in video content. Multi-dimensional evaluation models and weighted decision-making mechanisms ensure high-fidelity transmission and timely processing of critical information, while reinforcement learning algorithms further optimize the encoding parameter adjustment process, improving the system's adaptability and robustness. Thus, this collaborative optimization mechanism can significantly improve the overall performance and reliability of the video stream cloud-edge collaborative analysis system.

[0045] To make the method provided in the embodiments of this application clearer, the following is combined with Figure 2 The flowchart shown is used to illustrate the process of the video stream cloud-edge collaborative analysis system.

[0046] In coal mine scenarios, where video quality, transmission bandwidth, and processing latency are critical and resources are relatively limited, a cloud-edge collaborative approach is needed to address the challenges of balancing compression ratio and distortion rate, as well as resource constraints and computational efficiency in video acquisition, encoding, and transmission systems. This application's embodiments address the collaborative optimization needs of high compression ratio, low distortion rate, and resource constraints in underground coal mine video surveillance scenarios by constructing an intelligent video encoding and transmission system based on FPGA and cloud-edge collaboration (a video stream cloud-edge collaborative analysis system). See also... Figure 2 On the edge side, a hardware acceleration pipeline is deployed on the Xilinx KV260 FPGA platform. A lightweight neural network model is used to identify key regions of interest (ROI) in video in real time, and the built-in VCU codec unit is linked to implement differentiated compression: lossless encoding is used for ROI regions to ensure information integrity, and Gaussian blur and high quantization parameter compression are applied to non-ROI regions to reduce bandwidth consumption. On the cloud side, the GPU server performs ROI region super-resolution reconstruction on the received compressed stream to improve visual quality. At the same time, network load monitoring data and model inference results are integrated to generate bitrate adjustment instructions and feed them back to the edge. Through a dual-end joint learning mechanism, the cloud periodically sends reinforcement learning optimized model parameters to the edge FPGA to fine-tune the lightweight network, and the edge dynamically adjusts the encoding parameters (such as quantization step size and ROI threshold) according to the cloud feedback, forming a closed-loop optimization system of "precise edge compression - cloud super-resolution enhancement - network adaptive adjustment", which significantly reduces transmission bandwidth costs and end-to-end latency, while meeting the low power consumption constraints of intrinsically safe equipment in mines. The patent comprises four parts: a deep learning network model inference module and a video encoding / decoding / compression module on the FPGA edge computing device kv260 side; a super-resolution reconstruction and model inference module on the cloud GPU server side; and a network load and inference result feedback module. Among them: (1) FPGA-side inference and encoding / decoding module.

[0047] The FPGA-side inference and encoding / decoding module leverages the heterogeneous computing architecture of the Xilinx KV260 platform to construct a hardware-level video processing pipeline. This module first receives raw video streams from mine cameras via a PCIe interface and then deploys lightweight convolutional neural networks (such as the simplified MobileNetV3 architecture) using FPGA programmable logic resources for real-time Region of Interest (ROI) detection. To address the complex environment of low light and dust interference underground, the network incorporates adversarial sample augmentation strategies during the training phase to enhance its robustness in recognizing key targets such as equipment instruments and personnel postures. The inference engine employs HLS (High-Level Synthesis) optimized parallel computing units, controlling computational latency to within 33ms / frame through inter-layer pipelines and data reuse mechanisms, meeting the real-time requirements of mine monitoring.

[0048] The ROI detection results are output as a binary mask to the built-in video codec unit (VCU). The VCU uses a hard-core H.265 encoder and is directly connected to the neural network inference module via the AXI-Stream interface to achieve pixel-level collaboration. For ROI areas marked by the mask (such as equipment operation status areas and personnel activity areas), the VCU enables lossless encoding mode and allocates a high bit rate (QP≤22) to ensure zero distortion of key information. For non-ROI background areas (such as static tunnel walls), Gaussian blur preprocessing (3×3 convolution kernel) is dynamically applied and high quantization parameters (QP≥45) are used for compression, significantly reducing the bit rate. This process achieves pixel-level strategy switching through FPGA logic resources, avoiding the data transfer overhead of traditional CPU solutions.

[0049] Meanwhile, this module integrates a dynamic bitrate control mechanism: after receiving network load data (bandwidth, packet loss rate) and model optimization instructions from the cloud, it adjusts the encoding parameters through the ARM Cortex-A53 processor. For example, it automatically expands the non-ROI area and increases the QP value when the network is congested, and shrinks the ROI area boundary to improve the overall image quality when bandwidth is sufficient. All processing is completed on the FPGA hardware pipeline, with typical power consumption below 25W, which meets the intrinsic safety constraints of mining equipment. The final output is a composite data packet consisting of compressed video stream, ROI location metadata, and low-resolution feature vectors extracted by the lightweight model, which is uploaded to the cloud via gigabit industrial Ethernet.

[0050] This module achieves integrated "detection-compression-regulation" processing with limited power consumption through hardware and software co-design. Compared with traditional GPU edge methods, it can reduce bandwidth usage by more than 40% while ensuring the integrity of information in critical areas.

[0051] (2) Super-resolution reconstruction and model inference module on the cloud GPU server side.

[0052] The super-resolution reconstruction and model inference module on the cloud GPU server side receives a composite data stream uploaded from the edge FPGA device. This data stream includes a video bitstream efficiently compressed by the VCU, key region (ROI) location metadata, and low-resolution feature vectors extracted by a lightweight model. The server (cloud GPU server) first utilizes the parallel computing power of the NVIDIA GPU (A100) to perform H.265 hardware decoding on the compressed video stream. The decoded video frames undergo region processing based on the ROI metadata: for regions marked as ROIs (such as equipment dashboards and personnel workstations), a deep learning-based super-resolution reconstruction model is used, leveraging its generative adversarial network characteristics to recover details lost due to low lighting and high compression in the mine, upscaling the resolution to the original high-definition level (e.g., 720P→1080P), significantly enhancing the visual recognizability of key information; for non-ROI background areas, the original compressed resolution is maintained to save computing resources. Simultaneously with the super-resolution reconstruction, a high-precision deep inference model (ResNet-50) deployed in the cloud analyzes the video content. This model integrates two input sources: the super-resolution ROI region image and the low-resolution feature vector uploaded from the edge. By employing feature concatenation technology, semantic features (such as target contours and movement trends) extracted by the lightweight edge model are fused with fine-grained features (such as instrument pointer angles and personnel protective equipment status) extracted by the cloud model across resolutions, significantly improving the accuracy of identifying underground safety hazards (such as equipment malfunctions and violations). The model outputs structured analysis results (including event type, location coordinates, and confidence level) and pushes them to the mine safety monitoring platform for alarms.

[0053] This module can also integrate a dynamic optimization mechanism: based on real-time network load monitoring data (bandwidth fluctuations, packet loss rate) and quality assessment indicators of deep inference results (such as key target recognition rate), it generates bitrate adjustment instructions. For example, when a high-risk event (such as an area associated with abnormal gas concentration) is identified in consecutive video frames, the subsequent transmission priority of that ROI is automatically increased; during periods of network congestion, the edge end is instructed to temporarily increase the blur intensity and quantization parameters of non-ROI areas. Simultaneously, the cloud continuously optimizes the parameters of the lightweight edge model through a reinforcement learning framework (such as the PPO algorithm), periodically sending the fine-tuned model weights to the FPGA device, forming a collaborative optimization link of "initial edge perception - precise cloud decision-making - closed-loop model evolution." Finally, the video stream, after super-resolution enhancement and deep analysis, is distributed to end users by the cloud streaming media server according to an adaptive bitrate strategy (DASH), ensuring that high-value monitoring information can be obtained under different network conditions.

[0054] (3) Cloud-edge collaborative network load and inference optimization module.

[0055] The cloud-edge collaborative network load and inference optimization module constructs a dynamic closed-loop control system. By monitoring network transmission quality (including bandwidth fluctuations, end-to-end latency, and packet loss rate) and the validity of inference results in real time, it drives the adaptive allocation of edge and cloud resources. On the edge side, the FPGA's ARM processor periodically injects lightweight network probe data packets (such as 8-byte UDP packets), and obtains real-time link layer indicators by combining the hardware counter (Xilinx AXI Ethernet Subsystem register) of the gigabit industrial Ethernet interface: when a sudden drop in bandwidth (e.g., <50Mbps) or an increase in packet loss rate (e.g., >5%) is detected, a local bitrate adjustment strategy is immediately triggered—instructions are sent to the VCU encoder through the AXI-Lite interface to dynamically expand the range of non-ROI areas and increase quantization parameters (QP value increases by 10-15), while simultaneously enhancing the Gaussian blur intensity to reduce the instantaneous bitrate; conversely, when the network recovers stability, the ROI boundary is shrunk and the QP value is reduced to improve image quality.

[0056] The cloud server (cloud GPU server) can capture end-to-end transmission status (TCP retransmission rate and RTT jitter) from the network card hardware layer and combine it with the analysis quality indicators output by the deep inference module (such as decreased confidence in key target identification and increased missed detection rate of consecutive frame events) to build a multi-dimensional evaluation model. This model employs a weighted decision-making mechanism: during network congestion (bandwidth utilization > 85%), it prioritizes lossless encoded stream transmission in the ROI region and suspends the uploading of non-critical background frames; when a high-risk event is identified (such as device anomaly confidence > 90%), it forces the edge to increase the encoding priority of that ROI region to the highest level, and even in the face of packet loss, it enables forward error correction (FEC) redundancy protection. All policy instructions are serialized via Protobuf and fed back to the ARM core of the edge FPGA through a low-overhead control channel.

[0057] The core innovation of this module lies in its dual-end joint learning mechanism: a reinforcement learning algorithm (Q-learning algorithm) is deployed at the edge, using the network state matrix (bandwidth, latency, packet loss) and inference quality degradation rate as the environment state, and the adjustment range of encoding parameters (such as QP offset, ROI area ratio) as the action space. The optimal control strategy is trained through a long-term reward function (such as bandwidth saving rate × event recognition accuracy). This closed-loop system reduces image quality loss caused by network fluctuations by 37% and maintains a critical event recognition rate >92% in a typical weak network environment in a mine (RTT ≥ 200ms).

[0058] In summary, the mine video optimization technology based on FPGA and cloud-edge collaboration (video stream cloud-edge collaborative analysis method) provided in this embodiment identifies key regions of interest (ROIs) in real time by deploying lightweight neural networks at the edge, and implements differentiated compression using an FPGA hardware-level VCU codec: lossless encoding is used for ROIs to ensure information integrity, while Gaussian blurring and high-quantization parameter compression are applied to non-ROIs to reduce bandwidth consumption. The cloud utilizes GPU computing power to perform super-resolution reconstruction of ROIs to improve visual quality, and integrates low-resolution features uploaded from the edge with high-resolution images for high-precision security analysis. By monitoring network load (bandwidth, latency, packet loss rate) and inference quality in real time, bitrate adjustment commands are dynamically generated and fed back to the edge to adjust encoding parameters; simultaneously, reinforcement learning is used to optimize edge model parameters, forming a closed loop of "precise edge compression - cloud super-resolution enhancement - network adaptive adjustment". Thus, while meeting the low-power constraints of intrinsically safe underground equipment, it can significantly reduce transmission bandwidth by more than 40% and maintain a critical event recognition rate of >92%.

[0059] To implement the above embodiments, this application also proposes a video stream cloud-edge collaborative analysis device.

[0060] Figure 3 This is a schematic diagram of the structure of a video stream cloud-edge collaborative analysis device provided in an embodiment of this application.

[0061] like Figure 3 As shown, the video stream cloud-edge collaborative analysis device 300 includes: Data acquisition module 310 is used by the FPGA edge computing device to acquire the video stream captured by the camera device; The detection module 320 is used by the FPGA edge computing device to perform real-time key region detection on the video stream through an internally deployed lightweight neural network model, obtain key region detection results, and extract low-resolution feature vectors of the key regions. The compression module 330 is used by the FPGA edge computing device to compress the video stream of the key area using a lossless encoding mode and allocating a high bit rate according to the key area detection results through the built-in video encoding and decoding unit, and to preprocess the video stream of the non-key area using Gaussian blur training and high quantization parameters, so as to obtain compressed video data. The collaborative upload module 340 is used to collaboratively upload the compressed video data, the location metadata of the key area, and the low-resolution feature vector to the cloud GPU server, so that the cloud GPU server can perform data analysis based on the compressed video data, the location metadata of the key area, and the low-resolution feature vector.

[0062] The specific implementation and technical effects of each module in this embodiment are similar to those in the above method embodiments, and will not be repeated here.

[0063] To implement the above embodiments, this application also proposes a video stream cloud-edge collaborative analysis device, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method provided in the foregoing embodiments.

[0064] To implement the above embodiments, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided in the foregoing embodiments.

[0065] To implement the above embodiments, this application also proposes a computer program product, including a computer program that, when executed by a processor, implements the methods provided in the foregoing embodiments.

[0066] It should be noted that personal information collected from users should be used for legitimate and reasonable purposes and should not be shared or sold outside of these legitimate uses. Furthermore, such collection / sharing should only be conducted after receiving the user's informed consent, including but not limited to notifying the user to read the user agreement / user notice and sign an agreement / authorization that includes authorization of relevant user information before the user uses the function. In addition, any necessary steps must be taken to protect and safeguard access to such personal information data and ensure that others with access to personal information data comply with their privacy policies and procedures.

[0067] This application is intended to provide an implementation scheme for users to selectively prevent the use or access to their personal information data. Specifically, this application is intended to provide hardware and / or software to prevent or block access to such personal information data. Once personal information data is no longer needed, risks can be minimized by restricting data collection and deleting data. Furthermore, where applicable, such personal information is de-identified to protect user privacy.

[0068] In the foregoing descriptions of the embodiments, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0069] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0070] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0071] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0072] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0073] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0074] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0075] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A video stream cloud-edge collaborative analysis method, characterized in that, An application is made in a video stream cloud-edge collaborative analysis system, the video stream cloud-edge collaborative analysis system comprising an FPGA edge computing device and a cloud GPU server; the video stream cloud-edge collaborative analysis method comprises: The FPGA edge computing device acquires the video stream captured by the camera device; The FPGA edge computing device uses a lightweight neural network model deployed internally to perform real-time key region detection on the video stream, obtain key region detection results, and extract low-resolution feature vectors of the key regions. The FPGA edge computing device, through its built-in video encoding and decoding unit, performs lossless encoding and high bit rate compression on the video stream of the key area based on the key area detection results, and performs Gaussian blur preprocessing and high quantization parameter compression on the video stream of the non-key area to obtain compressed video data. The compressed video data, the location metadata of the key area, and the low-resolution feature vector are uploaded to the cloud GPU server in a coordinated manner, so that the cloud GPU server can perform data analysis based on the compressed video data, the location metadata of the key area, and the low-resolution feature vector.

2. The video stream cloud-edge collaborative analysis method according to claim 1, characterized in that, The lightweight neural network model adopts the simplified MobileNetV3 architecture and is trained using an adversarial sample augmentation strategy. Furthermore, the inference engine of the lightweight neural network model uses the parallel computing unit of HLS and controls the computing process through inter-layer pipelines and data reuse mechanisms.

3. The video stream cloud-edge collaborative analysis method according to claim 1, characterized in that, The detection results of the key region are output to the video encoding and decoding unit in the form of a binary mask.

4. The video stream cloud-edge collaborative analysis method according to claim 1, characterized in that, When the video stream of the key area is compressed using a lossless coding mode and a high bit rate, the quantization parameter is less than or equal to the first threshold. When performing Gaussian blur preprocessing on the video stream training of non-critical areas and compression processing with high quantization parameters, the quantization parameters are greater than or equal to the second threshold. The first threshold is less than the second threshold.

5. The video stream cloud-edge collaborative analysis method according to claim 4, characterized in that, Before the FPGA edge computing device performs real-time key region detection on the video stream using an internally deployed lightweight neural network model, obtains the key region detection results, and extracts the low-resolution feature vectors of the key regions, it also includes: The FPGA edge computing device integrates a dynamic bit rate control mechanism to receive network load data fed back by the cloud GPU server; When the network load data indicates network congestion, the range of non-critical areas is expanded, the quantization parameters are increased, and the Gaussian blur intensity is enhanced. When the network load data indicates that the network has returned to stability, the boundary of the critical region is contracted and the quantization parameter is reduced.

6. The video stream cloud-edge collaborative analysis method according to claim 1, characterized in that, After uploading the compressed video data, the location metadata of the key region, and the low-resolution feature vector to the cloud GPU server, the process further includes: The cloud GPU server uses the parallel computing capabilities of the GPU to perform hardware decoding on the compressed video data to obtain decoded video frames. The decoded video frame is processed by region segmentation based on key region metadata; wherein, the region segmentation includes: reconstructing key regions using a super-resolution reconstruction model to obtain key region images; and reconstructing non-key regions using the original compressed resolution to obtain non-key region images; the super-resolution reconstruction model is implemented based on a generative adversarial network.

7. The video stream cloud-edge collaborative analysis method according to claim 6, characterized in that, Also includes: The cloud GPU server performs a fusion analysis based on the key region image, the low-resolution feature vector, and the fine-grained features extracted by the deep inference model to obtain a structured analysis result; wherein, the structured analysis result includes at least one of event type, location coordinates, and confidence level; Output the structured analysis results.

8. The video stream cloud-edge collaborative analysis method according to any one of claims 1-7, characterized in that, It also includes at least one of the following: When a high-risk event is detected in consecutive video frames, the FPGA edge computing device increases the transmission priority of the critical area associated with the high-risk event; In the event of network congestion, the FPGA edge computing device temporarily amplifies the blur intensity and quantization parameters of the non-critical areas; The cloud GPU server continuously adjusts the model parameters of the lightweight neural network model of the FPGA edge computing device through a reinforcement learning framework, and sends the model parameters to the FPGA edge computing device.

9. The video stream cloud-edge collaborative analysis method according to any one of claims 1-7, characterized in that, It also includes at least one of the following: The cloud-based GPU server captures end-to-end transmission status from the network interface card (NIC) hardware layer and, combined with the analysis quality metrics output by the deep inference module of the cloud-based GPU server, constructs a multi-dimensional evaluation model. This multi-dimensional evaluation model employs a weighted decision-making mechanism to ensure lossless encoded stream transmission in critical areas during network congestion periods, while suspending the upload of non-critical background frames. When a high-risk event is identified, the FPGA edge computing device is forced to increase the encoding priority of the critical areas associated with the high-risk event to the highest level and enables forward error correction redundancy protection processing. The analysis quality metrics are output by the deep inference module of the cloud-based server after performing deep analysis on the compressed video stream uploaded by the FPGA edge computing device, key area location metadata, and low-resolution feature vectors. The FPGA edge computing device deploys a reinforcement learning algorithm, and uses the network state matrix and inference quality decay rate as the environment state, the encoding parameter adjustment range as the action space, and trains the optimal control strategy through a long-term reward function to realize the encoding parameter adjustment process of the video encoding and decoding unit.

10. A video stream cloud-edge collaborative analysis device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

Images