A video analytics system, method, device, and medium

Abstract

The application provides a video analysis system, method, device and medium, relates to the technical field of video analysis processing, and comprises a server and multiple edge nodes; the server comprises a top agent network and function modules of different video processing functions, each function module is configured with a sub-agent network; each function module is used for receiving real-time video streams sent by each edge node, and processing the real-time video streams based on processing parameters in the corresponding sub-agent network; each sub-agent network is used for monitoring the current processing state of the corresponding function module when processing the real-time video streams in real time, and sending the current processing state to the top agent network; the top agent network is used for acquiring a server state of the server; based on the server state and different current processing states, a resource allocation scheme is determined, and the resource allocation scheme is sent to each sub-agent network and the server. The application can improve the resource utilization rate of video analysis and reduce processing delay.

Description

Technical Field

[0001] This application relates to the field of video analysis and processing technology, and more specifically, to a video analysis system, method, device, and medium. Background Technology

[0002] Video analytics leverages artificial intelligence to intelligently process and analyze video data, aiming to extract valuable information, identify specific targets, and understand scenes and behaviors, thereby achieving in-depth insights and accurate understanding of video content. This task is widely applied in numerous fields such as security monitoring, intelligent transportation, industrial quality inspection, and medical image analysis, playing a crucial role in improving efficiency, ensuring safety, and optimizing services across various industries.

[0003] Current video analytics technologies primarily employ a pre-allocation combined with threshold triggering model for resource configuration. This not only results in low resource utilization and high processing latency, but also fails to dynamically adjust based on the workload, priority, and real-time requirements of processing tasks. Furthermore, it cannot achieve dynamic collaborative allocation of resources across multiple modules or adapt to dynamic changes in the scenario. Summary of the Invention

[0004] The purpose of this application is to provide a video analysis system, method, device, and medium to solve the above-mentioned problems existing in the prior art, improve resource utilization, reduce processing latency, and realize intelligent resource scheduling and parameter optimization across modules.

[0005] In a first aspect, a video analysis system is provided, which may include: a server and multiple edge nodes; the server includes a top-level proxy network and functional modules with different video processing functions, each functional module being configured with a sub-proxy network;

[0006] Each edge node is used to acquire real-time video streams and send the real-time video streams to the corresponding functional modules;

[0007] Each functional module is used to receive the real-time video stream and process the real-time video stream based on the processing parameters in the corresponding sub-proxy network;

[0008] Each sub-agent network is used to monitor the current processing status of the corresponding functional module when processing the real-time video stream in real time, and send the current processing status to the top-level agent network;

[0009] The top-level proxy network is used to obtain the server status of the server; based on the server status and different current processing states, it determines a resource allocation scheme and sends the resource allocation scheme to each sub-proxy network and server, so that the server allocates resources to each functional module according to the resource allocation scheme.

[0010] In an optional implementation, the resource allocation scheme includes the amount of resources to be allocated for different functional modules;

[0011] Each sub-agent network is also used for:

[0012] Receive the resource allocation scheme issued by the top-level proxy network;

[0013] The configured baseline processing parameters, the current processing status of the corresponding functional modules, and the amount of resources to be allocated are input into the corresponding parameter decision model based on residual attention mechanism to determine the processing parameters of the corresponding processing modules.

[0014] The processing parameters are sent to the corresponding processing modules so that the corresponding processing modules can process the real-time video stream based on the processing parameters.

[0015] In an optional implementation, the top-level proxy network is specifically used for:

[0016] The different current processing states sent by each sub-agent network are integrated into a global state vector;

[0017] The server state and the global state vector are input into a pre-trained global resource allocation model to obtain the resource allocation scheme.

[0018] In an optional implementation, the sub-agent networks are further used for:

[0019] Based on the current processing state of the corresponding functional module at the current moment and the historical processing state at the previous moment adjacent to the current moment, calculate the first reward value of the corresponding functional module at the current moment.

[0020] The first reward value is sent to the top-level agent network, and the parameter decision model based on the residual attention mechanism is updated using the first reward value.

[0021] In an optional implementation, the top-level proxy network is also used for:

[0022] Based on the first reward value sent by each sub-agent network, the server status of the server at the current moment, and the historical server status at the previous moment adjacent to the current moment, the second reward value of the corresponding functional module at the current moment is determined.

[0023] The global resource allocation model is updated using the second reward value.

[0024] In an optional implementation, the sub-agent networks are further used for:

[0025] Based on the processing status of the corresponding functional modules at multiple consecutive moments, predict the predicted processing status at a future preset moment, and send the predicted processing status to the top-level proxy network.

[0026] The top-level proxy network is further used for:

[0027] The different prediction processing states sent by each sub-agent network are integrated into a global prediction state vector;

[0028] The server state, the global state vector, and the global predicted state vector are input into a pre-trained global resource allocation model to obtain a resource allocation scheme for a future preset time.

[0029] In an optional implementation, the functional modules include: a read stream module, an inference module, a tracking module, a behavior analysis module, a rendering and image storage module, and a message sending module;

[0030] The reading stream module is used to receive the real-time video stream, decode the real-time video stream based on the reading stream parameters in the corresponding reading stream sub-agent network to obtain a decoded video stream, and send the decoded video stream to the inference module.

[0031] The inference module is used to receive the decoded video stream sent by the reading module, perform target detection and classification on the decoded video stream based on the inference parameters in the corresponding inference sub-agent network, and obtain the target detection result; after binding the target detection result with the corresponding decoded video stream, it is sent to the tracking module.

[0032] The tracking module is used to receive the target detection result and the decoded video stream sent by the inference module; perform cross-frame target association on the target detection result based on the tracking parameters in the corresponding tracking sub-agent network, assign a unique ID to each target detected in the target detection result and track the trajectory to obtain the trajectory data of each target; and send the trajectory data to the behavior analysis module.

[0033] The behavior analysis module is used to receive trajectory data sent by the tracking module, determine whether the trajectory data of each target conforms to the configured event rules based on the analysis parameters in the corresponding behavior analysis sub-agent network, and obtain the event detection result; send the event detection result to the message sending module, and annotate the video frame data that conforms to the event rules and send it to the rendering and image storage module.

[0034] The rendering and image storage module is used to receive the video frame data sent by the behavior analysis module, render the video frame data based on the rendering parameters in the corresponding rendering sub-proxy network, and then store it.

[0035] The message sending module is used to receive the event detection results sent by the behavior analysis module, generate alarm information based on the alarm parameters in the corresponding message sub-agent network and the event detection results, and send the alarm information.

[0036] In a second aspect, a video analysis method is provided, wherein in a server using any of the video analysis systems described in the first aspect above, the apparatus may include:

[0037] Each functional module is controlled to receive real-time video streams sent by each edge node, and to process the real-time video streams based on the processing parameters in the corresponding sub-agent network.

[0038] The system controls each sub-agent network to monitor the current processing status of the corresponding functional modules when processing the real-time video stream, and sends the current processing status to the top-level agent network.

[0039] The top-level proxy network is controlled to obtain the server status of the server, and based on the server status and different current processing states, a resource allocation scheme is determined and the resource allocation scheme is sent to each sub-proxy network and server.

[0040] After receiving the resource allocation scheme of the top-level proxy network, resources are allocated to each functional module according to the resource allocation scheme.

[0041] Thirdly, an electronic device is provided, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0042] Memory, used to store computer programs;

[0043] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect above.

[0044] Fourthly, a computer-readable storage medium is provided, wherein a computer program is stored therein, and when executed by a processor, the computer program implements the steps of any of the methods described in the first aspect above.

[0045] This application adopts a two-level optimization structure of top-level agent and module-level sub-agent to achieve coordination between global resource quota decision-making and local parameter adjustment; the top-level agent generates a resource allocation scheme based on server status and global status; the sub-agent dynamically adjusts local processing parameters according to the resource allocation scheme.

[0046] This application achieves dynamic optimization of cross-module resources through a layered RL framework, reducing redundant resource waste; reduces end-to-end latency through the linkage of global decision-making and local adjustment; and reduces performance fluctuations by pre-trained baseline parameters (such as default parameters in standard scenarios) through a layered structure.

[0047] This application significantly improves resource utilization: the average utilization of GPU and CPU is increased by 25%–40%, effectively reducing resource idleness and waste, and supporting higher-density concurrent task processing; the application significantly reduces system latency: end-to-end processing latency is reduced by 15%–30%, significantly enhancing the real-time performance of the system and meeting the needs of high-time-sensitivity application scenarios; the application's strategy operation is more stable: during multi-scenario switching and strategy iteration, system performance fluctuations are controlled within 5%, effectively avoiding "catastrophic forgetting" and ensuring long-term operational reliability; the application's event response capability is greatly optimized: for critical events such as traffic accidents and abnormal behavior, the system can complete resource reallocation and parameter adjustment within milliseconds, achieving rapid response and accurate processing. Attached Figure Description

[0048] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 An architecture diagram of a video analysis system provided in this application embodiment;

[0050] Figure 2 A flowchart illustrating a video analysis method provided in an embodiment of this application;

[0051] Figure 3 A flowchart illustrating another video analysis method provided in an embodiment of this application;

[0052] Figure 4 A flowchart illustrating another video analysis method provided in an embodiment of this application;

[0053] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise defined, the technical or scientific terms used in this application should have the ordinary meaning understood by those skilled in the art. The words "first," "second," and similar terms used in this application do not indicate any order, quantity, or importance, but are only used to distinguish different components. The words "comprising" or "including," etc., mean that the element or object preceding the word covers the element or object listed after the word and its equivalents, but do not exclude other elements or objects. The words "connected," "coupled," or "connected," etc., are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up," "down," "left," "right," etc., are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0055] Reinforcement learning (RL) is a machine learning method that learns the optimal policy through "trial and error" by having an agent interact with the environment, with the goal of maximizing cumulative reward.

[0056] A hierarchical RL agent is a hierarchical structure consisting of a top-level agent and sub-agents. The top-level agent is responsible for cross-module resource allocation decisions, while the sub-agents are responsible for optimizing the internal parameters of a single module (such as frame rate and model resolution).

[0057] The residual-attention module is a mechanism that integrates pre-trained knowledge with real-time learning. It retains the core parameters of the model during resource adjustment and reduces "catastrophic forgetting" in online learning (i.e., new strategies learn to cover historically valid knowledge).

[0058] LSTM (Long Short-Term Memory) is a type of recurrent neural network that excels at processing time-series data. It can predict future trends based on historical states and is used to model the temporal dependencies between modules in video analytics.

[0059] Model inference refers to the process of using deep convolutional neural networks to detect and classify targets in image frames, with typical network models (such as the YOLO series, ResNet network, etc.).

[0060] The video analysis method provided in this application embodiment can be applied to... Figure 1 In the system architecture shown, such as Figure 1 As shown, the system may include: a server and multiple edge nodes; wherein, the server includes a top-level proxy network and functional modules with different video processing functions, and each functional module is configured with a sub-proxy network;

[0061] Each edge node is used to acquire real-time video streams and send them to the corresponding functional modules; each edge node can be a camera, and each edge node is connected using the RTSP / GB28181 protocol.

[0062] Each functional module is used to receive real-time video streams and process the real-time video streams based on the processing parameters in the corresponding sub-agent network;

[0063] Each sub-agent network is used to monitor the current processing status of the corresponding functional module when processing real-time video streams, and send the current processing status to the top-level agent network;

[0064] The top-level proxy network is used to obtain the server status; based on the server status and different current processing states, it determines the resource allocation scheme and sends the resource allocation scheme to each sub-proxy network and server; the resource allocation scheme includes the amount of resources to be allocated for different functional modules;

[0065] The server is used to execute the video analysis method provided in the embodiments of this application and to allocate resources to each functional module according to the resource allocation scheme.

[0066] In one embodiment of this application, each sub-agent network is further used for:

[0067] Receive resource allocation plans issued by the top-level proxy network;

[0068] The configured baseline processing parameters, the current processing status of the corresponding functional modules, and the amount of resources to be allocated are input into the corresponding parameter decision model based on residual attention mechanism to determine the processing parameters of the corresponding processing modules.

[0069] The processing parameters are sent to the corresponding processing modules so that the modules can process the real-time video stream based on these parameters.

[0070] In another embodiment of this application, the top-level proxy network is specifically used for:

[0071] The different current processing states sent by each sub-agent network are integrated into a global state vector;

[0072] The server state and global state vector are input into a pre-trained global resource allocation model to obtain a resource allocation scheme.

[0073] In yet another embodiment of this application, each sub-agent network is further configured to:

[0074] Based on the current processing state of the corresponding functional module at the current moment and the historical processing state at the previous moment adjacent to the current moment, calculate the first reward value of the corresponding functional module at the current moment.

[0075] The first reward value is sent to the top-level agent network, and the parameter decision model based on the residual attention mechanism is updated using the first reward value.

[0076] In another embodiment of this application, the top-level proxy network is further used for:

[0077] Based on the first reward value sent by each sub-agent network, the server status at the current moment, and the historical server status at the previous moment adjacent to the current moment, determine the second reward value of the corresponding functional module at the current moment.

[0078] Update the global resource allocation model using the second reward value.

[0079] In yet another embodiment of this application, each sub-agent network is further configured to:

[0080] Based on the processing status of the corresponding functional modules at multiple consecutive times, predict the predicted processing status at a future preset time, and send the predicted processing status to the top-level agent network.

[0081] The top-level proxy network is further used for:

[0082] The different prediction processing states sent by each sub-agent network are integrated into a global prediction state vector;

[0083] The server state, global state vector, and global predicted state vector are input into a pre-trained global resource allocation model to obtain a resource allocation scheme for a future preset time.

[0084] In another embodiment of this application, the functional modules include: a reading stream module, an inference module, a tracking module, a behavior analysis module, a rendering and image storage module, and a message sending module;

[0085] The read stream module is used to receive real-time video streams, decode the real-time video streams based on the read stream parameters in the corresponding read stream sub-agent network to obtain decoded video streams, and send the decoded video streams to the inference module.

[0086] The inference module receives the decoded video stream sent by the reading module, performs target detection and classification on the decoded video stream based on the inference parameters in the corresponding inference sub-agent network, and obtains the target detection result; after binding the target detection result with the corresponding decoded video stream, it sends it to the tracking module.

[0087] The tracking module receives the target detection results and decoded video stream sent by the inference module; performs cross-frame target association on the target detection results based on the tracking parameters in the corresponding tracking sub-agent network, assigns a unique ID to each target detected in the target detection results and tracks its trajectory to obtain the trajectory data of each target; and sends the trajectory data to the behavior analysis module.

[0088] The behavior analysis module receives trajectory data sent by the tracking module, determines whether the trajectory data of each target conforms to the configured event rules based on the analysis parameters in the corresponding behavior analysis sub-agent network, and obtains the event detection results; it sends the event detection results to the message sending module, and annotates the video frame data that conforms to the event rules and sends it to the rendering and image storage module.

[0089] The rendering and image storage module receives video frame data sent by the behavior analysis module, renders the video frame data based on the rendering parameters in the corresponding rendering sub-agent network, and then stores it.

[0090] The message sending module is used to receive the event detection results sent by the behavior analysis module, generate alarm information based on the alarm parameters in the corresponding message sub-agent network and the event detection results, and then send the alarm information.

[0091] In another embodiment of this application, the sub-agent network includes: a read stream sub-agent, an inference sub-agent, a tracking sub-agent, a behavior analysis sub-agent, a rendering and storage sub-agent, and a message sending sub-agent.

[0092] In practical applications, edge nodes and servers can be connected directly or indirectly through wired or wireless communication methods, which is not limited in this application.

[0093] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0094] Figure 2 This is a flowchart illustrating a video analysis method provided in an embodiment of this application. Figure 2 As shown, the method may include:

[0095] Step S210: Control each functional module to receive the real-time video stream sent by each edge node, and process the real-time video stream based on the processing parameters in the corresponding sub-agent network.

[0096] In practice, the control reading module receives the real-time video stream and decodes it based on the reading parameters in the corresponding reading sub-agent network to obtain the decoded video stream; the decoded video stream is then sent to the inference module.

[0097] The control inference module receives the decoded video stream sent by the read stream module, performs target detection and classification on the decoded video stream based on the inference parameters in the corresponding inference sub-agent network, and obtains the target detection result; after binding the target detection result with the corresponding decoded video stream, it sends it to the tracking module.

[0098] The control and tracking module receives the target detection results and decoded video stream sent by the inference module; it performs cross-frame target association on the target detection results based on the tracking parameters in the corresponding tracking sub-agent network, assigns a unique ID to each target detected in the target detection results and tracks its trajectory to obtain the trajectory data of each target; and sends the trajectory data to the behavior analysis module.

[0099] The control behavior analysis module receives trajectory data sent by the tracking module, and based on the analysis parameters in the corresponding behavior analysis sub-agent network, determines whether the trajectory data of each target conforms to the configured event rules, and obtains the event detection results; the event detection results are sent to the message sending module, and the video frame data that conforms to the event rules are labeled and sent to the rendering and image storage module.

[0100] The control rendering and image storage module receives video frame data sent by the behavior analysis module, renders the video frame data based on the rendering parameters in the corresponding rendering sub-proxy network, and then stores it.

[0101] The control message sending module receives the event detection results sent by the behavior analysis module, generates alarm information based on the alarm parameters in the corresponding message sub-agent network and the event detection results, and sends the alarm information.

[0102] In practical applications, the reading module sends the decoded video frame to the inference module for target detection and classification after decoding each video frame in the video stream, and obtains the target detection result corresponding to the video frame. The inference module sends the video frame and the corresponding target detection result to the tracking module after obtaining the target detection result corresponding to each video frame. The tracking module performs cross-frame target association based on multiple consecutive video frames and the corresponding target detection results.

[0103] Step S220: Control each sub-agent network to monitor the current processing status of the corresponding functional modules when processing real-time video streams, and send the current processing status to the top-level agent network.

[0104] In practice, the control stream sub-agent detects the current stream status of the stream module when decoding the real-time video stream. The current stream status includes: decoding latency, resource (GPU / CPU) usage, output queue length, and output frame rate.

[0105] The control inference sub-agent detects the current inference state of the inference module when performing target detection and classification on the decoded video stream. The current inference state includes: inference latency, resource (GPU / CPU) usage, output queue length, number of targets, and average target confidence.

[0106] The current tracking status of the real-time detection and tracking module of the control tracking sub-agent when performing cross-frame target association on the target detection results includes: detection latency, resource (GPU / CPU) usage, output queue length, average trajectory loss rate, and trajectory ID switching count.

[0107] The control behavior analysis sub-agent monitors the current behavior analysis status of the behavior analysis module when it is performing event detection. The current behavior analysis status includes: analysis latency, resource (GPU / CPU) usage, output queue length, number of event response timeouts, event recognition confidence, and event alarm frequency (times / minute).

[0108] The control rendering and image storage sub-agent can detect the current rendering status of the rendering and image storage module when rendering video frame data in real time. The current rendering status includes: rendering latency, resource (GPU / CPU) usage, output queue length, and rendering performance indicators.

[0109] The message sending sub-agent controls the real-time detection of the current alarm status of the message sending module when generating and sending alarm information. The current alarm status includes: alarm latency, resource (GPU / CPU) usage, output queue length, and alarm performance.

[0110] Each sub-agent is controlled to periodically (e.g., every second) report its current processing status to the top-level agent network via an efficient message queue (e.g., Kafka).

[0111] In one embodiment of this application, the processing parameters in each sub-agent network are determined by each sub-agent network based on the amount of resources to be allocated to the corresponding functional module and the baseline processing parameters in the resource allocation scheme received at the current time.

[0112] Step S230: Control the top-level proxy network to obtain the server status of the server, determine the resource allocation scheme based on the server status and different current processing states, and send the resource allocation scheme to each sub-proxy network and server.

[0113] The server status includes: overall GPU utilization, overall CPU utilization, memory usage, end-to-end processing latency, and video stream frame rate; the end-to-end processing latency and video stream frame rate are determined based on the different current processing statuses sent by each sub-agent network.

[0114] In practice, the top-level agent network integrates the different current processing states sent by each sub-agent network into a global state vector. The server state and the global state vector are input into a pre-trained global resource allocation model to obtain a resource allocation scheme containing the amount of resources to be allocated for different functional modules, and then distributed to each sub-agent network and server. The global resource allocation model uses a continuous decision space to allocate the maximum available resource quota for each functional module. The total GPU computing power quota is allocated to the inference module, for example, accounting for 30%-60% of the total GPU resources of the node; the total CPU core quota is allocated to the read stream module, for example, accounting for 10%-30% of the total CPU resources of the node; and the remaining CPU resource quota is allocated to other functional modules.

[0115] Step S240: After receiving the resource allocation scheme of the top-level proxy network, allocate resources to each functional module according to the resource allocation scheme.

[0116] In practice, after receiving the resource allocation plan, the server allocates corresponding resources to each functional module according to the amount of resources to be allocated to different functional modules in the resource allocation plan, so that each functional module can call the corresponding amount of resources to perform corresponding processing.

[0117] In one embodiment of this application, after receiving the resource allocation scheme from the top-level agent network, each sub-agent network inputs the configured baseline processing parameters, the current processing status of the corresponding functional module, and the amount of resources to be allocated into the corresponding parameter decision model based on the residual attention mechanism to determine the processing parameters of the corresponding processing module. Specifically, corresponding baseline processing parameters are pre-configured for different functional modules, such as a frame rate of 15fps and an inference resolution of 640×640 in a standard daytime scene. The processing parameters are then sent to the corresponding processing modules so that they can process the real-time video stream based on these parameters. For example, if the resource allocation scheme allocates 60% of the GPU computing power to the inference module, the inference sub-agent can adjust the resolution to 960×720.

[0118] Specifically, the parameter decision model based on residual attention mechanism is used to determine the processing parameters of the corresponding functional modules; the read stream sub-agent is used to determine the frame extraction rate for decoding; the inference sub-agent is used to dynamically adjust the concurrency of model instances and batch size through the API of the TritonServer model inference framework; the tracking sub-agent is used to determine the target association threshold and the number of trajectory retention frames; the parameter decision model based on residual attention mechanism internally integrates a residual attention module, which is used to perform attention calculation on the current state of the corresponding processing module and an embedding vector that can be learned to represent the pre-trained benchmark processing parameters, so as to determine the current processing parameters according to the deviation between the current processing state and the benchmark processing parameters.

[0119] The parameter decision model based on residual attention mechanism in this application retains the baseline parameters of the pre-trained scene (such as a frame rate of 15fps and a resolution of 640×640 in a standard scene) and only optimizes the dynamic adjustment part to avoid performance loss in policy updates. At the same time, by dynamically adjusting the residual part, it can quickly adapt to new scenes (such as rainy days and nights) without affecting the baseline performance.

[0120] In one embodiment of this application, after each sub-agent network determines the processing parameters of the corresponding processing module according to the resource allocation scheme, the method may further include:

[0121] Each sub-agent network calculates the first reward value for its corresponding functional module at the current moment based on the module's current processing state and its historical processing state at the previous moment adjacent to the current moment. The first reward value is calculated by each sub-agent network according to a reward function. ; Indicates the weighting coefficient; This represents the baseline latency value, which is set based on the maximum reasonable processing latency configured or the maximum latency statistical value in the training data. This indicates the actual processing delay, i.e., the time consumed by each functional module in processing data; This represents the target recall rate, which is configured based on business needs. Indicates the weighting coefficient; Indicates the actual amount of resources used; This represents the resource allocation quota; given the resource quota, the faster and more accurately the data is processed, the higher the first reward value.

[0122] Each sub-agent network is controlled to send the first reward value to the top-level agent network, and the parameter decision model based on the residual attention mechanism is updated using the first reward value;

[0123] The top-level agent network determines the second reward value of the corresponding functional module at the current moment based on the first reward value sent by each sub-agent network, the server status at the current moment, and the historical server status at the previous moment adjacent to the current moment; and updates the global resource allocation model using the second reward value; wherein, the second reward value of the top-level agent network is calculated based on the configured reward function, which is a bi-objective optimization function aimed at encouraging high efficiency and low latency; Where w1, w2, w3, and w4 are weight coefficients, which are automatically tuned and determined during training; S represents the normalization function. Indicates average resource utilization rate; This represents the baseline value for resource utilization. Indicates the actual end-to-end delay; Indicates the end-to-end delay reference value; Indicates the trajectory loss rate; This indicates the number of event response timeouts; average resource utilization, end-to-end latency, trajectory loss rate, and number of event response timeouts are extracted based on the server state and historical processing status of the previous moment, as well as the server state and current processing status of the current moment; reward maximization means achieving the lowest latency and the best business performance with the highest possible resource utilization.

[0124] In yet another embodiment of this application, the method may further include:

[0125] Using a pre-trained LSTM prediction network, based on the processing status of the corresponding functional modules at multiple consecutive historical moments, the prediction processing status at a future preset moment is predicted, and the prediction processing status is sent to the top-level agent network.

[0126] The top-level agent network integrates the different prediction processing states sent by each sub-agent network into a global prediction state vector. The server state, global state vector, and global prediction state vector are input into a pre-trained global resource allocation model to obtain a resource allocation scheme at a preset time in the future. The system repeatedly allocates resources and processing parameters according to the resource allocation scheme and performs video processing based on the processing parameters and the allocated resources.

[0127] This application utilizes an LSTM model to predict future resource requirements (such as the number of targets and the frequency of events), and combines this with an event-triggered mechanism (such as traffic accident detection) to adjust parameters in a coordinated manner, which effectively improves GPU utilization. Through event-triggered coordinated adjustments (such as switching the inference module to a high-precision model and increasing the frame rate to 30fps), the response time is greatly shortened.

[0128] In another embodiment of this application, both the global resource allocation model and the parameter decision model adopt DRL algorithms suitable for continuous action spaces, such as DDPG (Deep Deterministic Policy Gradient) or TD3 (Twin Delayed DDPG); both the global resource allocation model and the parameter decision model include an Actor network for outputting actions and a Critic network for generating energy values.

[0129] like Figure 3 As shown, the video analysis method may also include:

[0130] Step 1: Each camera sends a video stream to the server's streaming module;

[0131] Step 2: The reading stream module decodes the video stream according to the processing parameters in the reading stream sub-agent and sends it to the inference module; at the same time, the reading stream sub-agent collects the reading stream status during the decoding process of the reading stream module and reports it to the top-level agent network.

[0132] Step 3: The inference module performs target detection on the decoded video stream according to the processing parameters in the inference sub-agent and sends the results to the inference module; at the same time, the inference sub-agent collects the inference status of the inference module during the inference process and reports it to the top-level agent network.

[0133] Step 4: The tracking module tracks the detected target according to the processing parameters in the tracking sub-agent and sends the result to the tracking module; at the same time, the tracking sub-agent collects the tracking status of the tracking module during the tracking process and reports it to the top-level agent network.

[0134] Step 5: The behavior analysis module performs behavior analysis on the tracked target based on the processing parameters in the behavior analysis sub-agent, and sends the results to the rendering and image storage module and the message sending module for further processing; at the same time, the behavior analysis sub-agent collects the analysis status of the behavior analysis module during the analysis process and reports it to the top-level agent network.

[0135] Step 6: The top-level proxy network determines the resource allocation plan based on the current processing status and server status reported by each proxy sub-network, and distributes it to the server and each sub-proxy network.

[0136] Step 7: The server allocates resources to each functional module according to the resource allocation scheme, and each sub-agent network generates new processing parameters for each functional module according to the resource allocation scheme. Then, it returns to execute step 2 until all video frames in the video stream have been processed.

[0137] like Figure 4As shown, the video analysis method may also include: when the behavior analysis module is running, it collects the number of historical targets and the frequency of historical events, such as the number of targets = 300 and the event frequency = 2 times / minute in the past 5 minutes; based on the number of historical targets and the frequency of historical events, it predicts the resource requirements for the next 5 seconds as the number of targets = 400 and the event frequency = 3 times / minute; when the top-level agent network performs resource pre-allocation, it increases the GPU computing power of the tracking module to 25% and suspends low-priority tasks (such as regular traffic statistics); when a traffic accident is detected, it triggers event linkage: triggers parameter adjustment of the associated module: the inference module switches to a high-precision model (such as switching YOLOv8s to YOLOv8l), and the frame rate is increased to 25fps.

[0138] This application also provides an electronic device, such as... Figure 5 As shown, it includes a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540.

[0139] Memory 530 is used to store computer programs;

[0140] When the processor 510 executes the program stored in the memory 530, it performs the following steps:

[0141] Each functional module receives real-time video streams sent by each edge node and processes the real-time video streams based on the processing parameters in the corresponding sub-agent network.

[0142] Control each sub-agent network to monitor the current processing status of the corresponding functional modules when processing real-time video streams, and send the current processing status to the top-level agent network;

[0143] The top-level proxy network is controlled to obtain the server status of the server, determine the resource allocation scheme based on the server status and different current processing states, and send the resource allocation scheme to each sub-proxy network and server;

[0144] After receiving the resource allocation plan from the top-level proxy network, resources are allocated to each functional module according to the resource allocation plan.

[0145] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0146] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0147] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0148] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0149] The implementation methods and beneficial effects of the various components of the electronic device in the above embodiments for solving the problem can be found in [reference needed]. Figure 2 The steps in the illustrated embodiments are used to implement the electronic device. Therefore, the specific working process and beneficial effects of the electronic device provided in this application will not be repeated here.

[0150] In another embodiment provided in this application, a computer-readable storage medium is also provided, which stores instructions that, when executed on a computer, cause the computer to perform any of the video analysis methods described in the above embodiments.

[0151] In another embodiment provided in this application, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the video analysis methods described in the above embodiments.

[0152] Those skilled in the art will understand that the embodiments in this application can be provided as methods, systems, or computer program products. Therefore, the embodiments in this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the embodiments in this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0153] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0154] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0155] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0156] Although preferred embodiments have been described in this application, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of this application.

[0157] Obviously, those skilled in the art can make various modifications and variations to the embodiments of this application without departing from the spirit and scope of the embodiments of this application. Therefore, if these modifications and variations to the embodiments of this application fall within the scope of this application and its equivalents, then these modifications and variations are also intended to be included in the embodiments of this application.

Claims

1. A video analytics system, characterized by, The video analysis system comprises a server and a plurality of edge nodes; the server comprises a top-level agent network and function modules of different video processing functions, each function module being configured with a sub-agent network; Each edge node is configured to acquire a real-time video stream and send the real-time video stream to a corresponding function module; Each function module is configured to receive the real-time video stream and process the real-time video stream based on processing parameters in the corresponding sub-agent network; Each sub-agent network is configured to monitor a current processing state of the corresponding function module when processing the real-time video stream in real time and send the current processing state to the top-level agent network; The top-level agent network is configured to acquire a server state of the server, integrate different current processing states sent by the sub-agent networks into a global state vector, input the server state and the global state vector into a pre-trained global resource allocation model to obtain a resource allocation scheme, and send the resource allocation scheme to the sub-agent networks and the server, so that the server allocates resources for the function modules according to the resource allocation scheme.

2. The system of claim 1, wherein, The resource allocation scheme comprises to-be-allocated resource amounts of different function modules; Each sub-agent network is further configured to: receive the resource allocation scheme issued by the top-level agent network; input the configured baseline processing parameters, the current processing state of the corresponding function module and the to-be-allocated resource amount into a corresponding parameter decision model based on a residual attention mechanism to determine processing parameters of the corresponding processing module; issue the processing parameters to the corresponding processing module, so that the corresponding processing module processes the real-time video stream based on the processing parameters.

3. The system of claim 2, wherein, Each sub-agent network is further configured to: calculate a first reward value of the corresponding function module at a current time according to a current processing state of the corresponding function module at the current time and a historical processing state of the corresponding function module at a previous time adjacent to the current time; send the first reward value to the top-level agent network and update the parameter decision model based on the residual attention mechanism using the first reward value.

4. The system of claim 3, wherein, The top-level agent network is further configured to: determine a second reward value of the corresponding function module at the current time according to the first reward value sent by each sub-agent network, a server state of the server at the current time and a historical server state of the server at the previous time adjacent to the current time; update the global resource allocation model using the second reward value.

5. The system of claim 4, wherein, Each sub-agent network is further configured to: predict a predicted processing state at a future preset time according to processing states of the corresponding function module at a plurality of continuous times, and send the predicted processing state to the top-level agent network; The top-level agent network is further configured to: integrate different predicted processing states sent by the sub-agent networks into a global predicted state vector; input the server state, the global state vector and the global predicted state vector into the pre-trained global resource allocation model to obtain a resource allocation scheme at the future preset time.

6. The system of claim 1, wherein, The function modules comprise a stream reading module, an inference module, a tracking module, a behavior analysis module, a rendering and image storage module and a message sending module; The reading stream module is configured to receive the real-time video stream, decode the real-time video stream based on reading stream parameters in a corresponding reading stream sub-agent network, and obtain a decoded video stream; and send the decoded video stream to the inference module. The inference module is configured to receive the decoded video stream sent by the reading stream module, perform target detection and classification on the decoded video stream based on inference parameters in a corresponding inference sub-agent network, and obtain a target detection result; and send the target detection result bound with the corresponding decoded video stream to the tracking module. The tracking module is configured to receive the target detection result and the decoded video stream sent by the inference module; perform cross-frame target association on the target detection result based on tracking parameters in a corresponding tracking sub-agent network, assign a unique ID to each target detected in the target detection result and track a trajectory of the target, and obtain trajectory data of each target; and send the trajectory data to the behavior analysis module. The behavior analysis module is configured to receive the trajectory data sent by the tracking module, judge whether the trajectory data of each target conforms to a configured event rule based on analysis parameters in a corresponding behavior analysis sub-agent network, and obtain an event detection result; send the event detection result to the message sending module, and send video frame data conforming to the event rule to the rendering and storage module after labeling the video frame data. The rendering and storage module is configured to receive the video frame data sent by the behavior analysis module, perform rendering on the video frame data based on rendering parameters in a corresponding rendering sub-agent network, and store the video frame data after rendering. The message sending module is configured to receive the event detection result sent by the behavior analysis module, generate alarm information according to the event detection result based on alarm parameters in a corresponding message sub-agent network, and send the alarm information.

7. A method of video analytics, characterized by, The method applied to a server of the video analysis system of any one of claims 1-6, the method comprising: controlling each functional module to receive real-time video streams sent by each edge node, and processing the real-time video streams based on processing parameters in a corresponding sub-agent network; controlling each sub-agent network to monitor a current processing state of a corresponding functional module when processing the real-time video streams in real time, and sending the current processing state to the top-level agent network; controlling the top-level agent network to obtain a server state of the server, integrating different current processing states sent by each sub-agent network into a global state vector, inputting the server state and the global state vector into a pre-trained global resource allocation model, obtaining a resource allocation scheme, and sending the resource allocation scheme to each sub-agent network and the server; after receiving the resource allocation scheme of the top-level agent network, allocating resources for each functional module according to the resource allocation scheme.

8. An electronic device, comprising: The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is configured to store a computer program. The processor is configured to execute the program stored on the memory to implement the method of claim 7.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the method in claim 7.

Images