Video analysis method, apparatus, medium, and device

Abstract

This disclosure provides a video analysis method, apparatus, medium, and device. It involves receiving an execution instruction from a target video analysis task, instructing the analysis of a video stream to be analyzed using a target video analysis model. When a video stream to be analyzed exists, a first decoded video frame corresponding to the stream is acquired. The first video frame to be analyzed is determined from the first decoded video frame. When the target video analysis model is in a running state, the first video frame to be analyzed is analyzed using the target video analysis model to obtain a first video analysis result. Therefore, when multiple video analysis tasks use the same video stream for analysis, only the same video stream is retrieved and decoded once. When multiple video analysis tasks use the same video analysis model to analyze multiple video streams, only one copy of the video analysis model is loaded and run, saving system resources.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a video analysis method, apparatus, medium, and device. Background Technology

[0002] Video analytics tasks involve using a video analytics model to analyze and process a video stream to achieve a specific objective. For example, a face detection model might be used to analyze a video stream for face detection. The processing of video analytics tasks typically follows a sequence of video stream retrieval, decoding, and model inference. In a multi-task vision management system, different video analytics models might be used to analyze the same video stream, while the same model might be used to analyze different video streams.

[0003] In related technologies, for each video analysis task, the video stream to be analyzed is retrieved and decoded, and then the video analysis model used by the task is loaded and run for inference to complete each task. When there are multiple video analysis tasks, the related technologies proceed as follows: Figure 1 As shown, each video analysis task is a pipeline, and each pipeline performs the above steps. Resources are used independently between multiple video analysis tasks, without interference. That is to say, in related technologies, when multiple video analysis tasks use the same video stream for analysis, the same video stream will be fetched and decoded multiple times; when multiple video analysis tasks use the same video analysis model to analyze multiple video streams, multiple copies of the same video analysis model will be loaded and run, resulting in a significant waste of system resources and low overall system resource utilization. Summary of the Invention

[0004] This disclosure provides a video analysis method, apparatus, medium, and device. When multiple video analysis tasks use the same video stream for analysis and processing, the same video stream will only be retrieved and decoded once. When multiple video analysis tasks use the same video analysis model to analyze and process multiple video streams, only one video analysis model will be loaded and run, which can save system resources and make the overall system resource utilization rate higher.

[0005] To address the aforementioned technical problems, the present disclosure provides the following technical solutions:

[0006] A video analysis method, comprising:

[0007] Receive the execution instruction for the target video analysis task, the execution instruction being used to instruct the target video analysis model to analyze and process the video stream to be analyzed;

[0008] When the video stream to be analyzed exists, obtain the first decoded video frame corresponding to the video stream to be analyzed;

[0009] The first video frame to be analyzed is determined from the first decoded video frame;

[0010] When the target video analysis model is in the running state, the first video frame to be analyzed is analyzed and processed by the target video analysis model to obtain the first video analysis result.

[0011] A video analysis device, comprising:

[0012] The receiving unit is used to receive the execution instruction of the target video analysis task, wherein the execution instruction is used to instruct the video stream to be analyzed and processed through the target video analysis model;

[0013] The acquisition unit is used to acquire the first decoded video frame corresponding to the video stream to be analyzed when the video stream to be analyzed exists.

[0014] The determining unit is used to determine the first video frame to be analyzed from the first decoded video frame;

[0015] The processing unit is configured to analyze and process the first video frame to be analyzed through the target video analysis model when the target video analysis model is in the running state, and obtain the first video analysis result.

[0016] In some embodiments, the video analysis device further includes a deployment unit and an operation unit. The deployment unit is configured to deploy the target video analysis model when the target video analysis model is in an unrunning state and the target video analysis model is in an undeployed state, so that the deployment state of the target video analysis model is in a deployed state.

[0017] The running unit is used to load and run the target video analysis model when the deployment status of the target video analysis model is "deployed", so that the running status of the target video analysis model is "running".

[0018] In some implementations, the determining unit is specifically used for:

[0019] According to the video analysis frame rate and the first video frame filtering strategy, the first video frame to be analyzed is determined from the first decoded video frame.

[0020] The processing unit is specifically used for:

[0021] According to the target data input conditions required by the target video analysis model, the first video frame to be analyzed is subjected to data conversion processing to obtain the first converted video frame.

[0022] The first converted video frame is input into the target video analysis model for analysis and processing to obtain the first video analysis result.

[0023] In some embodiments, the video analysis device further includes a fetching unit and a decoding unit, wherein the fetching unit is used to fetch the video stream to be analyzed when the video stream to be analyzed does not exist;

[0024] The decoding unit is used to decode the video stream to be analyzed to obtain a second decoded video frame;

[0025] The determining unit is used to determine the second video frame to be analyzed from the second decoded video frame;

[0026] The processing unit is used to analyze and process the second video frame to be analyzed through the target video analysis model when the target video analysis model is in the running state, so as to obtain the second video analysis result.

[0027] In some embodiments, the execution instruction further carries a target model identifier of the target video analysis model, and the determining unit is specifically used for:

[0028] For each second decoded video frame obtained, determine whether the target model identifier and the target data input conditions need to be added to the distribution list corresponding to the obtained second decoded video frame according to the video analysis frame rate and the second video frame filtering strategy.

[0029] If it is necessary to add the target model identifier and the target data input conditions to the distribution list corresponding to the obtained second decoded video frame, then add the target model identifier and the target data input conditions to the distribution list corresponding to the obtained second decoded video frame, and determine the second decoded video frame containing the target model identifier and the target data input conditions in the corresponding distribution list as the second video frame to be analyzed;

[0030] The processing unit is specifically used for:

[0031] According to the target data input conditions in the distribution list corresponding to the second video frame to be analyzed, the second video frame to be analyzed is subjected to data conversion processing to obtain the second converted video frame.

[0032] Based on the target model identifier in the distribution list corresponding to the second video frame to be analyzed, the second converted video frame is distributed to the target video analysis model for analysis and processing to obtain the second video analysis result.

[0033] In some embodiments, the video analysis apparatus further includes an update unit, the update unit being configured to:

[0034] Obtain a model data list, which is used to record the model data corresponding to the model identifiers of different video analysis models;

[0035] When the target model data corresponding to the target model identifier does not exist in the model data list, the deployment status of the target video analysis model is determined to be undeployed, the running status of the target video analysis model is determined to be unrunning, and the target reference count corresponding to the target model identifier is set to a preset number.

[0036] Based on the target model identifier, the undeployed state, the non-running state, the target reference count, and the target data input conditions, target model data corresponding to the target model identifier is generated, and the target model data corresponding to the target model identifier is added to the model data list;

[0037] When the deployment status of the target video analysis model is "deployed", the non-deployed status of the target model data corresponding to the target model identifier is updated to "deployed".

[0038] When the target video analysis model is in the running state, the non-running state included in the target model data corresponding to the target model identifier is updated to the running state.

[0039] In some embodiments, the updating unit is further configured to:

[0040] When the target model data corresponding to the target model identifier exists in the model data list, the target reference count included in the target model data corresponding to the target model identifier is increased by a preset number from the current count.

[0041] In some embodiments, the execution instruction further carries the target video stream identifier of the video stream to be analyzed and the video stream address of the video stream to be analyzed, and the update unit is further configured to:

[0042] Obtain a video stream data list, which is used to record the video stream data corresponding to the video stream identifiers of different video streams;

[0043] When the target video stream data corresponding to the target video stream identifier does not exist in the video stream data list, it is determined that the video stream to be analyzed does not exist, and a distribution strategy list corresponding to the target video stream identifier is created;

[0044] Based on the target model identifier, the target data input conditions, and the video analysis frame rate, target distribution strategy data is generated and added to the distribution strategy list;

[0045] Based on the target video stream identifier, the distribution strategy list, and the video stream address, target video stream data corresponding to the target video stream identifier is generated, and the target video stream data corresponding to the target video stream identifier is added to the video stream data list;

[0046] The pull unit is specifically used for:

[0047] The video stream to be analyzed is retrieved using the video stream address of the video stream to be analyzed;

[0048] The determining unit is specifically used for:

[0049] For each second decoded video frame obtained, based on the video analysis frame rate and the second video frame filtering strategy included in the target distribution strategy data, it is determined whether the target model identifier and target data input conditions included in the target distribution strategy data need to be added to the distribution list corresponding to the obtained second decoded video frame.

[0050] If it is necessary to add the target model identifier and target data input conditions included in the target distribution strategy data to the distribution list corresponding to the obtained second decoded video frame, then the target model identifier and target data input conditions included in the target distribution strategy data are added to the distribution list corresponding to the obtained second decoded video frame, and the second decoded video frame containing the target model identifier and target data input conditions included in the target distribution strategy data in the corresponding distribution list is determined as the second video frame to be analyzed.

[0051] In some embodiments, the updating unit is further configured to:

[0052] When the target video stream data corresponding to the target video stream identifier exists in the video stream data list, target distribution strategy data is generated based on the target model identifier, the target data input conditions and the video analysis frame rate, and the target distribution strategy data is added to the distribution strategy list included in the target video stream data corresponding to the target video stream identifier.

[0053] The determining unit is further configured to:

[0054] When the target video stream data corresponding to the target video stream identifier exists in the video stream data list, it is determined that the video stream to be analyzed exists.

[0055] In some embodiments, the updating unit is further configured to:

[0056] Based on the task identifier, the target video stream identifier, and the target model identifier, task data is generated;

[0057] Add the task data to the task data list.

[0058] In some embodiments, the video analysis device further includes a stop unit, the stop unit being configured to:

[0059] Receive a stop command for the target video analysis task, the stop command being used to instruct the execution of the target video analysis task to be stopped;

[0060] Stop executing the target video analysis task.

[0061] In some embodiments, the stopping unit is specifically used for:

[0062] Remove the task data from the task data list;

[0063] Remove the target distribution strategy data from the distribution strategy list;

[0064] The target reference count is reduced by a preset number from the current count.

[0065] In some embodiments, the updating unit is further configured to:

[0066] When the number of references to the target is less than the preset number, the target model data corresponding to the target model identifier is deleted from the model data list;

[0067] The stopping unit is specifically used to: stop the operation of the target video analysis model.

[0068] In some embodiments, the updating unit is further configured to:

[0069] When the distribution strategy list is empty, the target video stream data corresponding to the target video stream identifier is deleted from the video stream data list;

[0070] The stop unit is specifically used to: stop pulling the video stream to be analyzed.

[0071] A computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute the video analysis method described above.

[0072] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the video analysis method described above.

[0073] A computer program product or computer program includes computer instructions stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium and executes the computer instructions to implement the aforementioned video analysis method.

[0074] This embodiment of the disclosure receives an execution instruction from a target video analysis task, which instructs the target video analysis model to analyze and process the video stream to be analyzed. When the video stream to be analyzed exists, a first decoded video frame corresponding to the video stream to be analyzed is obtained. A first video frame to be analyzed is determined from the first decoded video frame. When the target video analysis model is in a running state, the first video frame to be analyzed is analyzed and processed by the target video analysis model to obtain a first video analysis result. Thus, when multiple video analysis tasks use the same video stream for analysis and processing, the same video stream will only be retrieved and decoded once. When multiple video analysis tasks use the same video analysis model to analyze and process multiple video streams, only one video analysis model will be loaded and run, which can save system resources and make the overall system resource utilization rate higher.

[0075] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0076] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0077] Figure 1 A schematic diagram illustrating the scenarios for solutions provided by related technologies.

[0078] Figure 2 This is a schematic diagram of a video analysis system provided in an embodiment of the present disclosure.

[0079] Figure 3 This is a flowchart illustrating the video analysis method provided in an embodiment of the present disclosure.

[0080] Figure 4 This is a schematic diagram of a video analysis method provided in an embodiment of the present disclosure.

[0081] Figure 5 This is a schematic diagram illustrating the processing procedure after receiving the execution instruction for the target video analysis task, as provided in an embodiment of this disclosure.

[0082] Figure 6 This is a schematic diagram illustrating the specific implementation process of step 302 provided in the embodiments of this disclosure.

[0083] Figure 7 A schematic diagram illustrating the specific implementation process of step 308 provided in this embodiment of the disclosure.

[0084] Figure 8 This is a schematic diagram illustrating the processing procedure after receiving a stop command for a target video analysis task, as provided in an embodiment of this disclosure.

[0085] Figure 9 This is a schematic diagram illustrating the specific implementation process of step 402 provided in the embodiments of this disclosure.

[0086] Figure 10 This is a schematic diagram of the structure of the video analysis device provided in an embodiment of this disclosure.

[0087] Figure 11 A schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0088] To enable those skilled in the art to better understand the solutions of this application, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0089] It should be noted that while some processes described in the specification, claims, and accompanying drawings contain multiple steps that appear in a specific order, it should be clearly understood that these steps may not be performed in the order they appear herein, or may be performed in parallel. The step numbers are merely used to distinguish different steps and do not represent any particular order of execution. Furthermore, descriptions such as "first," "second," or "objective" in this document are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0090] Please see Figure 2 , Figure 2This is a schematic diagram of a video analysis system provided in an embodiment of this disclosure. The video analysis system includes a camera device 110, a server 120, etc.

[0091] Camera equipment 110 includes, but is not limited to, network cameras (IP cameras, IPCs), analog cameras with video encoders, PTZ cameras, dome cameras, etc. Camera equipment 110 can communicate with server 120 and exchange data via wired or wireless means.

[0092] Server 120 can be a single high-performance computer, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines), etc. The server stores pre-defined timing logic. Server 120 can pull video streams from camera device 110 for analysis and processing to perform video analysis tasks.

[0093] The video analysis method of this disclosure can be implemented on server 120.

[0094] Server 120 receives the execution instruction of the target video analysis task. The execution instruction is used to instruct the analysis and processing of the video stream to be analyzed by camera device 110 through the target video analysis model. When there is a video stream to be analyzed, the first decoded video frame corresponding to the video stream to be analyzed is obtained. The first video frame to be analyzed is determined from the first decoded video frame. When the target video analysis model is in the running state, the first video frame to be analyzed is analyzed and processed through the target video analysis model to obtain the first video analysis result.

[0095] It should be noted that, Figure 2 The schematic diagram of the video analysis system shown is merely an example. The video analysis system and scenario described in this disclosure are intended to more clearly illustrate the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided in this disclosure. As those skilled in the art will know, with the evolution of video processing and the emergence of new business scenarios, the technical solutions provided in this disclosure are also applicable to similar technical problems.

[0096] In this embodiment, the video analysis device will be described from the perspective of a video analysis device. Specifically, the video analysis device can be integrated into a computer device that has a storage unit and a microprocessor installed, thus having computing power. The computer device can be a server or a terminal device. In this embodiment, the computer device will be described as a server.

[0097] Please see Figure 3 , Figure 3 This is a flowchart illustrating the video analysis method provided in this embodiment. The video analysis method includes:

[0098] In step 201, an execution instruction for the target video analysis task is received, which instructs the target video analysis model to perform analysis and processing on the video stream to be analyzed.

[0099] Video analysis tasks, such as target video analysis tasks or visual analysis tasks, involve using a video analysis model (e.g., a target video analysis model) to analyze and process a video stream (e.g., the video stream to be analyzed) to achieve specific objectives, such as face detection, pedestrian feature extraction, vehicle detection, and license plate recognition. For example, video analysis tasks can include face detection, pedestrian feature extraction, vehicle detection, and license plate recognition.

[0100] Video analytics models, such as target video analytics models, are models used to analyze and process video streams, such as face detection models, pedestrian feature extraction models, vehicle detection models, and license plate recognition models.

[0101] A video stream, such as the video stream to be analyzed, refers to a data stream consisting of consecutive video frames arranged in chronological order, transmitted over a network or stored locally. In this embodiment, the video stream is mainly used by the video analysis model for analysis and processing to perform video analysis tasks.

[0102] As described above, a video analysis task typically involves two main resources: the video stream used for analysis and the video analysis model used to analyze the video stream. In related technologies, video analysis tasks are usually processed sequentially in the following steps: video stream retrieval, decoding, model loading and execution, and model inference. After model inference, post-processing of the inference results may also be involved. Furthermore, since the data format of the decoded frames may not meet the data input conditions required by the model, data conversion processing of the decoded frames may also be involved before model inference. In a multi-task vision management system, different video analysis models may be used to analyze the same video stream, while the same video analysis model may be used to analyze different video streams.

[0103] For example, in a multi-task vision management system for a smart park, the video streams from multiple cameras connected to the park require two visual analysis tasks: face detection and pedestrian feature extraction, to identify unauthorized entry. Furthermore, for cameras at the entrances and exits of the park's parking garage, in addition to face detection, vehicle detection and license plate recognition are also required, among other visual analysis tasks. Therefore, the multi-task vision management system involves the analysis and processing of multiple models and multiple video streams.

[0104] In related technologies, when multiple video analysis tasks use the same video stream for analysis and processing, the same video stream will be fetched and decoded multiple times. For the same video stream, multiple decoders need to be started for decoding, which will consume hardware video image processing resources. Since image data is a large amount of data, copying image data to multiple decoders will cause large blocks of image data to be moved frequently between video memory and system memory, excessively consuming input / output (IO) read and write bandwidth. When multiple video analysis tasks use the same video analysis model to analyze and process multiple video streams, multiple copies of the same video analysis model will be loaded and run. The size of the model parameters will determine the amount of video memory occupied by the model during loading and running. For large models, such resource waste will be more obvious, resulting in a large waste of system resources, low overall system resource utilization, and low task processing efficiency.

[0105] Based on the above problems, this disclosure provides a video analysis method. In this video analysis method, when the same video stream is analyzed and processed using different video analysis models, the video stream is only fetched and decoded once. When multiple video streams are analyzed and processed using the same video analysis model, only one video analysis model is deployed on the system, and only one video analysis model is loaded and run to analyze and process multiple video streams. The specific implementation process is as follows.

[0106] For example, an external device can automatically send an execution instruction for the target video analysis task to the server according to a first preset rule, and the server will then receive the execution instruction for the target video analysis task.

[0107] For example, a smart park's multi-task visual management system is pre-programmed to "execute face detection tasks on the video stream of the camera equipment at the east gate of the park every day from 9:00 to 18:00". When this time period arrives, external devices automatically send execution instructions for the face detection task to the server to perform face detection tasks on the video stream of the camera equipment at the east gate of the park.

[0108] For example, managers can manually send execution instructions for target video analysis tasks to the server through external devices, such as operating computers, mobile phones, tablets, and other terminal devices in the monitoring center of a smart park, and then the server receives the execution instructions for the target video analysis tasks.

[0109] For example, if the monitoring personnel in a smart park discover abnormal activity in a certain area of ​​the park, they can send an execution command for a face detection task to the server via a computer, so as to perform a face detection task on the video stream of the camera equipment in that area.

[0110] For example, the server may include a task management unit and a receiving unit. The task management unit can automatically send the execution instruction of the target video analysis task to the receiving unit according to a first preset rule, or the administrator can manually send the execution instruction of the target video analysis task to the receiving unit through the task management unit, so that the receiving unit receives the execution instruction of the target video analysis task.

[0111] The first preset rule can be set in advance, and no specific restrictions are imposed here.

[0112] The execution instructions for the target video analysis task correspond to the target video analysis task and are used to instruct the video stream to be analyzed to be processed through the target video analysis model.

[0113] For example, the execution instruction for the target video analysis task could be instruction C1 for face detection task T1, instructing face detection processing of the video stream V1 to be analyzed using face detection model M1. The execution instruction for the target video analysis task could also be instruction C2 for face detection task T2, instructing face detection processing of the video stream V2 to be analyzed using face detection model M1. Furthermore, the execution instruction for the target video analysis task could be instruction C3 for pedestrian feature extraction task T3, instructing pedestrian feature extraction processing of the video stream V1 to be analyzed using pedestrian feature extraction model M2.

[0114] In step 202, when there is a video stream to be analyzed, the first decoded video frame corresponding to the video stream to be analyzed is obtained.

[0115] The first decoded video frame is the video frame obtained after the video stream to be analyzed is decoded by the decoder.

[0116] Considering that the video stream to be analyzed may have already been retrieved and decoded by the server after receiving execution instructions from other video analysis tasks, upon receiving the execution instruction from the target video analysis task, the system first checks whether the video stream to be analyzed exists on the server. If the video stream to be analyzed exists, it can be determined that the video stream to be analyzed has already been retrieved and decoded by the server after receiving execution instructions from other video analysis tasks. Therefore, it is not necessary to retrieve and decode the video stream to be analyzed again; the first decoded video frame corresponding to the video stream to be analyzed decoded by the server can be directly obtained.

[0117] In step 203, the first video frame to be analyzed is determined from the first decoded video frame.

[0118] Once the first decoded video frame is obtained, the first video frame to be analyzed can be determined from the first decoded video frame.

[0119] In some embodiments, the execution instructions of the target video analysis task carry the video analysis frame rate of the target video analysis task, and determine the first video frame to be analyzed from the first decoded video frame, including:

[0120] Based on the video analysis frame rate and the first video frame filtering strategy, the first video frame to be analyzed is determined from the first decoded video frame.

[0121] The first video frame to be analyzed is processed using a target video analysis model to obtain the first video analysis result, including:

[0122] According to the target data input conditions required by the target video analysis model, the first video frame to be analyzed is subjected to data conversion processing to obtain the first converted video frame.

[0123] The first converted video frame is input into the target video analysis model for analysis and processing to obtain the first video analysis result.

[0124] The video analysis frame rate refers to the number of frames processed from a video stream per unit of time, measured in frames per second (fps). It is carried by the execution instructions of the video analysis task and is a core quantitative indicator derived from decoded video frames. Conversely, the video analysis frame rate of the target video analysis task, carried by the execution instructions, specifically refers to the number of frames processed from the video stream to be analyzed per unit of time. It is a core quantitative indicator for selecting the first video frame to be analyzed from the first decoded video frames or the second video frame to be analyzed from the second decoded video frames.

[0125] The first video frame filtering strategy refers to the rules or methods for selecting the first video frame to be analyzed from the first decoded video frames. This can be set by administrators or by the server based on certain rules; no specific restrictions are imposed here. For example, the first video frame filtering strategy could be a uniform interval filtering strategy based on the video analysis frame rate. This means that the first video frame to be analyzed is selected from the first decoded video frames at fixed time intervals based on the video analysis frame rate of the target video analysis task, ensuring a uniform distribution of the first video frame to be analyzed, which is suitable for most common scenarios.

[0126] The first video frame to be analyzed refers to the first decoded video frame determined from the first decoded video frame according to the video analysis frame rate of the target video analysis task and the first video frame filtering strategy, and which is ultimately analyzed and processed by the target video analysis model.

[0127] For example, assuming the frame rate of the video stream to be analyzed is 30fps, the frame rate of the video analysis task is 3fps, and the first video frame filtering strategy is to extract one frame every 10 frames based on the video analysis frame rate, then the 10th, 20th, and 30th frames decoded within 1 second can be determined as the first video frames to be analyzed.

[0128] For example, assuming the frame rate of the video stream to be analyzed is 30fps, the frame rate of the video analysis task is 10fps, and the first video frame filtering strategy is to extract one frame every 3 frames based on the video analysis frame rate, then the first decoded video frames of the 3rd, 6th, 9th, 12th, 15th, 18th, 21st, 24th, 27th, and 30th frames decoded within 1 second can be determined as the first video frames to be analyzed.

[0129] In step 204, when the target video analysis model is in the running state, the first video frame to be analyzed is analyzed and processed by the target video analysis model to obtain the first video analysis result.

[0130] Among them, the target video analysis model is the video analysis model used to analyze and process the first video frame to be analyzed, as indicated by the execution instructions of the target video analysis task, such as a face detection model, a pedestrian feature extraction model, or a license plate recognition model.

[0131] Considering that the target video analysis model has already been run by the server after receiving execution instructions from other target video analysis tasks (i.e., the target video analysis model is in a running state), the running state of the target video analysis model can be checked after receiving the execution instructions for the target video analysis task. When the running state of the target video analysis model is "running," there is no need to load and run a new target video analysis model. Instead, the first video frame to be analyzed can be directly analyzed and processed using the already running target video analysis model to obtain the first video analysis result.

[0132] In some embodiments, when the target video analysis model is in a running state, before analyzing and processing the first video frame to be analyzed by the target video analysis model to obtain the first video analysis result, the method further includes:

[0133] When the target video analysis model is in an inactive state and its deployment state is not deployed, deploy the target video analysis model so that its deployment state is deployed.

[0134] When the target video analysis model is in the deployed state, load and run the target video analysis model, so that the target video analysis model is in the running state.

[0135] Understandably, even when the target video analytics model is in a non-running state, its deployment status can still be checked. If the target video analytics model is in a non-deployed state, it can be determined that the target video analytics model was not deployed by the server after receiving execution instructions from other video analytics tasks. Therefore, the target video analytics model can be deployed, changing its deployment status to deployed. Then, when the target video analytics model is deployed, it can be loaded and run, thus changing its running status to running.

[0136] When the target video analysis model is in the deployed state, it can be determined that the target video analysis model has been deployed by the server after receiving the execution instructions of other video analysis tasks. Then, we can wait for the target video analysis model to change to the running state before using the target video analysis model to analyze and process the first video frame to be analyzed.

[0137] As can be seen from the above, in this embodiment of the multi-task vision management system, the video analysis model is deployed only when a video analysis task uses a video analysis model for analysis and processing for the first time. Then, the video analysis model is loaded and run, and inference is performed. When subsequent video analysis tasks use the same video analysis model for analysis and processing, it is only necessary to directly use the same video analysis model to analyze and process the video stream targeted by the subsequent video analysis task. This can reduce the occupation of video memory resources, thereby saving system resources and making the system resource utilization rate higher.

[0138] In some embodiments, after obtaining the first video analysis result, the first video analysis result can be returned directly.

[0139] In other embodiments, after obtaining the first video analysis result, the first video analysis result can be post-processed to obtain a first post-processed result, and the first post-processed result can be returned.

[0140] For example, assuming the target video analysis task is a face detection task, the first video analysis result is the face detection result obtained by analyzing and processing the first video frame to be analyzed through a face detection model. After obtaining the face detection result, face detection results with a confidence level lower than the first preset confidence level can be filtered out, or post-processing such as data encapsulation in the appropriate format can be performed according to the needs of the face detection task to obtain the post-processed face detection result, and the post-processed face detection result can be returned to the external device or task management unit.

[0141] In some embodiments, the execution instruction of the target video analysis task also carries the target video stream identifier of the video stream to be analyzed and the task identifier of the target video analysis task. When returning the first video analysis result or the first post-processing result, the target video stream identifier of the video stream to be analyzed, the task identifier of the target video analysis task, and the timestamp information of the first video frame to be analyzed can also be returned simultaneously to ensure a globally unified spatiotemporal identifier.

[0142] The first pre-set reliability can be set in advance, and there are no specific restrictions here. For example, the first pre-set reliability can be 0.85.

[0143] As described above, this embodiment of the present disclosure receives an execution instruction from a target video analysis task, which instructs the target video analysis model to analyze and process the video stream to be analyzed. When a video stream to be analyzed exists, the first decoded video frame corresponding to the video stream to be analyzed is obtained. The first video frame to be analyzed is determined from the first decoded video frame. When the target video analysis model is in a running state, the first video frame to be analyzed is analyzed and processed by the target video analysis model to obtain the first video analysis result. Thus, when multiple video analysis tasks use the same video stream for analysis and processing, the same video stream will only be retrieved and decoded once. When multiple video analysis tasks use the same video analysis model to analyze and process multiple video streams, only one video analysis model will be loaded and run, which can save system resources and make the overall system resource utilization rate higher.

[0144] In some embodiments, after receiving the execution instruction for the target video analysis task, the method further includes:

[0145] If the video stream to be analyzed does not exist, fetch the video stream to be analyzed.

[0146] The video stream to be analyzed is decoded to obtain the second decoded video frame;

[0147] The second video frame to be analyzed is determined from the second decoded video frame;

[0148] When the target video analysis model is in the running state, the second video frame to be analyzed is analyzed and processed by the target video analysis model to obtain the second video analysis result.

[0149] It is understandable that when the video stream to be analyzed does not exist on the server, it can be determined that the video stream to be analyzed has not been retrieved and decoded by the server after receiving the execution instructions of other video analysis tasks. Therefore, the video stream to be analyzed can be retrieved, and then the decoder can be used to decode the video stream to be analyzed to obtain the second decoded video frame. The second video frame to be analyzed can then be determined from the second decoded video frame. When the target video analysis model is in the running state, the target video analysis model is used to analyze the second video frame to be analyzed to obtain the second video analysis result.

[0150] For example, assuming the video stream to be analyzed is the video stream of a camera device in a smart park, the video stream to be analyzed can be retrieved from the camera device, and the decoder can be started to decode the video stream to obtain the second decoded video frame. Then, subsequent processing can be performed based on the second decoded video frame.

[0151] In some embodiments, after obtaining the second video analysis result, the second video analysis result can be returned directly.

[0152] In some embodiments, after obtaining the second video analysis result, the second video analysis result can be post-processed to obtain a second post-processed result, and the second post-processed result can be returned.

[0153] For example, assuming the target video analysis task is license plate recognition, the second video analysis result is the license plate recognition result obtained by analyzing and processing the second video frame to be analyzed through the license plate recognition model. After obtaining the license plate recognition result, license plate recognition results with confidence levels lower than the second preset confidence level can be filtered out, or post-processing such as data encapsulation in the appropriate format can be performed according to the needs of the license plate recognition task to obtain the post-processed license plate recognition result, and the post-processed license plate recognition result can be returned to the external device or task management unit.

[0154] In some embodiments, the execution instruction of the target video analysis task also carries the target video stream identifier of the video stream to be analyzed and the task identifier of the target video analysis task. When returning the second video analysis result or the second post-processing result, the target video stream identifier of the video stream to be analyzed, the task identifier of the target video analysis task, and the timestamp information of the second video frame to be analyzed can also be returned simultaneously to ensure a globally unified spatiotemporal identifier.

[0155] The second pre-set reliability can be set in advance, and there are no specific restrictions here. For example, the second pre-set reliability can be 0.88.

[0156] As can be seen from the above, in this embodiment of the disclosure, when a video analysis task uses a video stream for analysis and processing for the first time in the multi-task vision management system, the decoder resources are scheduled to pull and decode the video stream. When subsequent video analysis tasks use the same video stream for analysis and processing, it is only necessary to filter the decoded video frames for analysis and processing by the video analysis model according to the video analysis frame rate of different video analysis tasks. Thus, when multiple video analysis tasks use the same video stream for analysis and processing, the video stream will only be pulled and decoded once, avoiding the high-frequency movement of large blocks of image data in video memory and memory, which consumes too much IO read and write bandwidth, thereby saving system resources and making the system resource utilization rate higher.

[0157] In some embodiments, the execution instruction carries the video analysis frame rate of the target video analysis task and the target model identifier of the target video analysis model, and determines the second video frame to be analyzed from the second decoded video frame, including:

[0158] For each second decoded video frame obtained, determine whether the target model identifier and target data input conditions need to be added to the distribution list corresponding to the obtained second decoded video frame according to the video analysis frame rate and the second video frame filtering strategy.

[0159] If it is necessary to add the target model identifier and target data input conditions to the distribution list corresponding to the obtained second decoded video frame, then add the target model identifier and target data input conditions to the distribution list corresponding to the obtained second decoded video frame, and determine the second decoded video frame containing the target model identifier and target data input conditions in the corresponding distribution list as the second video frame to be analyzed;

[0160] The second video frame to be analyzed is processed using the target video analysis model to obtain the second video analysis results, including:

[0161] According to the target data input conditions in the distribution list corresponding to the second video frame to be analyzed, the second video frame to be analyzed is processed by data conversion to obtain the second converted video frame.

[0162] Based on the target model identifier in the distribution list corresponding to the second video frame to be analyzed, the second converted video frame is distributed to the target video analysis model for analysis and processing to obtain the second video analysis result.

[0163] The target model identifier is used to uniquely identify the target video analysis model.

[0164] The target data input conditions describe the requirements for the image data input to the target video analysis model, such as the image data format and resolution requirements. For example, assuming the target data input conditions are that the image data required by the target video analysis model is in RGB format and the resolution is 256*256, then after obtaining the second video frame to be analyzed, it needs to be converted into a second converted video frame in RGB format with a resolution of 256*256 before being sent to the target video analysis model for analysis and processing.

[0165] For example, after obtaining each second decoded video frame, the system determines whether to distribute the obtained second decoded video frame according to the video analysis frame rate of the target video analysis task and the second video frame filtering strategy. If the obtained second decoded video frame needs to be distributed, the target model identifier of the target video analysis model and the target data input conditions required by the target video analysis model are added to the distribution list corresponding to the obtained second decoded video frame.

[0166] Next, it can be determined whether the distribution list corresponding to the obtained second decoded video frame is empty. If the distribution list is not empty, and the distribution list contains a target model identifier and target data input conditions, the obtained second decoded video frame can be identified as the second video frame to be analyzed. Then, according to the target data input conditions in the distribution list, the second video frame to be analyzed can be processed to obtain a second converted video frame. This converted video frame is then distributed to the video analysis model indicated by the target model identifier, i.e., the target video analysis model. The target video analysis model analyzes the second converted video frame to obtain the second video analysis result. If the distribution list corresponding to the obtained second decoded video frame is empty, it means that the second decoded video frame does not need to be distributed. Therefore, the second decoded video frame can be directly discarded until the target video analysis task is stopped.

[0167] The process of converting the second video frame into a second converted video frame according to the target data input conditions in the distribution list corresponding to the second video frame to be analyzed can be referred to in the following example.

[0168] Assuming the second video frame to be analyzed has a resolution of 1920*1080 and a data format of YUV, and the target data input conditions include: the image data format required by the target video analysis model is RGB format, and the image data resolution required by the target video analysis model is 256*256, then the second video frame to be analyzed can be converted and scaled to a second converted video frame with a resolution of 256*256 and a data format of RGB.

[0169] In some embodiments, the execution instruction for the target video analysis task carries the target model identifier of the target video analysis model, and after receiving the execution instruction for the target video analysis task, it further includes:

[0170] Obtain the model data list, which records the model data corresponding to the model identifiers of different video analysis models;

[0171] When there is no target model data corresponding to the target model identifier in the model data list, the deployment status of the target video analysis model is determined to be undeployed, the running status of the target video analysis model is determined to be unrunning, and the target reference count corresponding to the target model identifier is set to a preset number.

[0172] Based on the target model identifier, non-deployed status, non-running status, target reference count, and target data input conditions, generate the target model data corresponding to the target model identifier, and add the target model data corresponding to the target model identifier to the model data list;

[0173] When the deployment status of the target video analysis model is "deployed", the non-deployed status of the target model data corresponding to the target model identifier will be updated to "deployed".

[0174] When the target video analysis model is in the running state, the non-running state included in the target model data corresponding to the target model identifier is updated to the running state.

[0175] In this embodiment, the server can maintain a task management information database. Each time an execution instruction for a video analysis task is received, the database can be updated based on the task parameters carried by the instruction. These task parameters may include model information, video stream information, and task information. Therefore, after receiving an execution instruction for a video analysis task, the task parameters carried by the instruction can be parsed to obtain the model information, video stream information, and task information, and the task management information database can be updated based on these parameters.

[0176] The task management information database includes a model data list. This list records the model data corresponding to the model identifiers of different video analysis models. When a video analysis task on the server uses a video analysis model for analysis for the first time, model data corresponding to the model identifier of that video analysis model can be generated based on the model information included in the task parameters carried by the execution instructions of that video analysis task, and added to the model data list. The model identifier of a video analysis model is used to uniquely identify that video analysis model.

[0177] For example, the model data structure can include the model identifier of the video analysis model, the deployment status of the video analysis model, the running status of the video analysis model, the data input conditions of the video analysis model, and the number of references. Therefore, based on the target model information included in the task parameters carried by the execution instructions of the target video analysis task, the target model data corresponding to the target model identifier can be generated according to the above model data structure. The number of references corresponding to the target model identifier is positively correlated with the number of video analysis tasks using the target video analysis model; that is, the more video analysis tasks using the target video analysis model, the more references the target model identifier will have.

[0178] The target model information may include the target model identifier and the download address of the target video analysis model package. The existence of the target model data corresponding to the target model identifier in the model data list is determined by checking if the target model identifier exists in the list. If the target model identifier does not exist in the model data list, it is determined that the target model data corresponding to the target model identifier does not exist in the model data list. If the target model identifier exists in the model data list, it is determined that the target model data corresponding to the target model identifier exists in the model data list.

[0179] When the target model data corresponding to the target model identifier does not exist in the model data list, it can be determined that the target video analysis model is being used for the first time in the target video analysis task. Therefore, the deployment status of the target video analysis model can be determined to be "not deployed," the running status of the target analysis model can be determined to be "not running," and the target reference count can be set to a preset number. Thus, the target video analysis model package can be downloaded from the download address of the target video analysis model package. The target video analysis model package includes the target video analysis model, algorithm program, management information, and technical information metadata description file. Among them, the technical information metadata description file includes the target data input conditions required by the target video analysis model. Therefore, the target data input conditions required by the target video analysis model can be obtained from the technical information metadata description file. Then, based on the target model identifier, the "not deployed," the "not running," the target reference count, and the target data input conditions, the target model data corresponding to the target model identifier can be generated, and the target model data corresponding to the target model identifier can be added to the model data list to record the model identifier, deployment status, running status, reference count, and target data input conditions of the target video analysis model.

[0180] The preset number of attempts can be set in advance, and no specific limit is imposed here. For example, in order to explicitly record the number of video analysis tasks using the same video analysis model, the preset number of attempts can be set to 1.

[0181] In some embodiments, the model information may further include other descriptive information about the target video analysis model, such as its version, name, and main executable file. When generating target model data corresponding to the target model identifier based on the target model identifier, its non-deployment status, non-running status, target reference count, and target data input conditions, the target model data corresponding to the target model identifier can be generated according to the following model data structure based on the target model identifier, non-deployment status, non-running status, target reference count, target data input conditions, and the aforementioned other descriptive information. The model data structure may include fields such as target model identifier, deployment status, running status, reference count, data input conditions, and other descriptive information about the model.

[0182] It should be noted that the above is merely an example of a model data structure and is not intended to limit this application. In practical applications, model data can also be generated according to other model data structures, and no specific restrictions are imposed here.

[0183] Understandably, since the target model data corresponding to the target model identifier records the deployment and running status of the target video analysis model, the deployment and running status of the target video analysis model can be determined by examining the deployment and running statuses included in the target model data corresponding to the target model identifier. Specifically, when the target model data corresponding to the target model identifier includes a "not deployed" status, the target video analysis model is determined to be in a "not deployed" state. When the target model data corresponding to the target model identifier includes a "deployed" status, the target video analysis model is determined to be in a "deployed" state. When the target model data corresponding to the target model identifier includes a "not running" status, the target video analysis model is determined to be in a "not running" state. When the target model data corresponding to the target model identifier includes a "running" status, the target video analysis model is determined to be in a "running" state.

[0184] After determining that the target video analytics model is in an undeployed state, it can be deployed based on the target video analytics model package, thus changing its deployment status to deployed. When the target video analytics model is deployed, the undeployed status of the target model data corresponding to the target model identifier can be updated to deployed, thereby updating the target model data corresponding to the target model identifier. Furthermore, when the target video analytics model is deployed, it can be loaded and run based on the main executable file included in the target model information, thus changing its running status to running. And when the target video analytics model is running, the unrunning status of the target model data can be updated to running, thereby updating the target model data again.

[0185] In some embodiments, after obtaining the model data list, the method further includes:

[0186] When the target model data corresponding to the target model identifier exists in the model data list, the target reference count of the target model data corresponding to the target model identifier will be increased by a preset number from the current count.

[0187] It is understandable that when the target model data corresponding to the target model identifier exists in the model data list, it can be determined that the server has already generated the target model data corresponding to the target model identifier based on the model information included in the task parameters carried by the execution instructions of other video analysis tasks after receiving the execution instructions of other video analysis tasks. Therefore, it is no longer necessary to generate the target model data corresponding to the target model identifier based on the model information included in the task parameters carried by the execution instructions of the target video analysis task. Instead, the number of target references included in the target model data corresponding to the target model identifier is increased by a preset number from the current number to update the record of the number of video analysis tasks using the target video analysis model.

[0188] For example, assuming the current count is 3, it means that the number of video analysis tasks using the target video analysis model is 3. Then, the target citation count can be increased by 1 from 3, so that the updated target citation count is 4, which means that the number of video analysis tasks using the target video analysis model is 4.

[0189] It is understandable that even if the target model data corresponding to the target model identifier is generated by the server based on the model information included in the task parameters carried by the execution instructions of other video analysis tasks, the deployment status and running status of the target video analysis model are still recorded in the target model data corresponding to the target model identifier. Therefore, even if the target model data corresponding to the target model identifier is generated by the server based on the model information included in the task parameters carried by the execution instructions of other video analysis tasks, the deployment status and running status of the target video analysis model can still be determined by the deployment status and running status included in the target model data corresponding to the target model identifier.

[0190] In some embodiments, the execution instructions for the target video analysis task further carry the target video stream identifier and the video stream address of the video stream to be analyzed.

[0191] After receiving the execution instructions for the target video analysis task, it also includes:

[0192] Get the video stream data list. The video stream data list is used to record the video stream data corresponding to the video stream identifier of different video streams.

[0193] If the target video stream data corresponding to the target video stream identifier does not exist in the video stream data list, it is determined that there is no video stream to be analyzed, and a distribution strategy list corresponding to the target video stream identifier is created.

[0194] Based on the target model identifier, target data input conditions, and video analysis frame rate, target distribution strategy data is generated and added to the distribution strategy list;

[0195] Based on the target video stream identifier, distribution strategy list, and video stream address, generate target video stream data corresponding to the target video stream identifier, and add the target video stream data corresponding to the target video stream identifier to the video stream data list;

[0196] Pull the video stream to be analyzed, including:

[0197] The video stream to be analyzed is retrieved using the video stream address of the video stream to be analyzed.

[0198] For each second decoded video frame obtained, based on the video analysis frame rate and the second video frame filtering strategy, it is determined whether the target model identifier and target data input conditions need to be added to the distribution list corresponding to the obtained second decoded video frame, including:

[0199] For each second decoded video frame obtained, based on the video analysis frame rate and second video frame filtering strategy included in the target distribution strategy data, determine whether it is necessary to add the target model identifier and target data input conditions included in the target distribution strategy data to the distribution list corresponding to the obtained second decoded video frame.

[0200] If it is necessary to add the target model identifier and target data input conditions to the distribution list corresponding to the obtained second decoded video frame, then add the target model identifier and target data input conditions to the distribution list corresponding to the obtained second decoded video frame, and determine the second decoded video frame containing the target model identifier and target data input conditions in the corresponding distribution list as the second video frame to be analyzed, including:

[0201] If it is necessary to add the target model identifier and target data input conditions included in the target distribution strategy data to the distribution list corresponding to the obtained second decoded video frame, then the target model identifier and target data input conditions included in the target distribution strategy data are added to the distribution list corresponding to the obtained second decoded video frame, and the second decoded video frame containing the target model identifier and target data input conditions included in the target distribution strategy data in the corresponding distribution list is determined as the second video frame to be analyzed.

[0202] The task management information database includes a video stream data list. This list records the video stream data corresponding to the video stream identifiers of different video streams. When a video analysis task on the server uses a video stream for analysis for the first time, it can generate the video stream data corresponding to the video stream identifier based on the video stream information included in the task parameters carried by the execution instructions of that video analysis task, and add it to the video stream data list. The video stream identifier of a particular video stream is used to uniquely identify that video stream.

[0203] For example, the video stream data structure may include a video stream identifier, a distribution strategy list, a video analysis frame rate, and a video stream address. Therefore, based on the target video stream information and target task information included in the task parameters carried by the execution instruction of the target video analysis task, the target video stream data corresponding to the target video stream identifier can be generated according to the above video stream data structure.

[0204] The target video stream information may include the target video stream identifier and the video stream address of the video stream to be analyzed. The target task information may include the video analysis frame rate of the target video analysis task. The existence of target video stream data corresponding to the video stream to be analyzed in the video stream data list can be determined by detecting whether the target video stream identifier exists in the video stream data list. If the target video stream identifier exists in the video stream data list, it is determined that the target video stream data corresponding to the video stream to be analyzed exists in the video stream data list. If the target video stream identifier does not exist in the video stream data list, it is determined that the target video stream data corresponding to the video stream to be analyzed does not exist in the video stream data list. When the target video stream data corresponding to the video stream to be analyzed is not found in the video stream data list, it can be determined that the video stream to be analyzed is being used for the first time by the target video analysis task. Therefore, a distribution strategy list corresponding to the target video stream identifier can be created first. Then, based on the target model identifier, the target data input conditions required by the target video analysis model, and the video analysis frame rate of the target video analysis task, target distribution strategy data can be generated and added to the distribution strategy list. Then, based on the target video stream identifier, the distribution strategy list, and the video stream address, target video stream data corresponding to the target video stream identifier can be generated and added to the video stream data list.

[0205] In some embodiments, the target video stream information may further include other descriptive information of the video stream, such as stream type. When generating target video stream data corresponding to the target video stream identifier based on the target video stream identifier, the distribution strategy list, and the video stream address, the target video stream data corresponding to the target video stream identifier may be generated according to the following video stream data structure based on the target video stream identifier, the distribution strategy list, the video stream address, and the aforementioned other descriptive information of the video stream. The video stream data structure may include fields such as target model identifier, distribution strategy list, video stream address, and other descriptive information of the video stream. The distribution strategy list may include subfields such as model identifier, data input conditions, and video analysis frame rate.

[0206] It should be noted that the above is merely an example of a video stream data structure and is not intended to limit this application. In practical applications, video stream data can also be generated according to other video stream data structures, and no specific restrictions are imposed here.

[0207] When the target video stream data corresponding to the video stream to be analyzed does not exist in the video stream data list, it can be determined that the video stream to be analyzed is being used for the first time by the target video analysis task. Therefore, it can be determined that the video stream to be analyzed does not exist in the server. Thus, the video stream to be analyzed can be retrieved from the video stream address of the video stream to be analyzed, and the video stream to be analyzed can be decoded to obtain the second decoded video frame.

[0208] After obtaining each second decoded video frame, the distribution strategy list corresponding to the target video stream identifier is read. Based on the video analysis frame rate in the target distribution strategy data in the distribution strategy list, it is determined whether the second decoded video frame should be distributed according to the second video frame filtering strategy. If the second decoded video frame needs to be distributed, the target model identifier and target data input conditions in the target distribution strategy data, such as the image data format and resolution required by the target video analysis model, are added to the distribution list corresponding to the second decoded video frame. The second decoded video frame containing the target model identifier and target data input conditions in the corresponding distribution list is the second video frame to be analyzed.

[0209] It is understandable that during the decoding process of the video stream to be analyzed, the server may also generate other distribution strategy data after receiving execution instructions from other video analysis tasks, and add it to the distribution strategy list corresponding to the video stream to be analyzed. For details of the processing, please refer to the processing procedure for the target distribution strategy data in the embodiments of this disclosure, which will not be elaborated here. That is to say, the distribution strategy list corresponding to the target video stream identifier also contains other distribution strategy data corresponding to other video analysis tasks. This other distribution strategy data includes other model identifiers of other video analysis models used by other video analysis tasks, other data input conditions required by other video analysis models, and the video analysis frame rate of other video analysis tasks. Therefore, the other model identifiers and other data input conditions in the other distribution strategy data can also be added to the distribution list corresponding to the corresponding second decoded video frame according to the above process.

[0210] Next, it can be determined whether the distribution list corresponding to the second decoded video frame is empty. When the distribution list corresponding to the second decoded video frame is not empty, data conversion processing can be performed on the second decoded video frame according to the data input conditions in the distribution list corresponding to the second decoded video frame to obtain a converted video frame. Specifically, when the distribution list corresponding to the second decoded video frame contains the target data input conditions in the target distribution strategy data, data conversion processing can be performed on the second decoded video frame to obtain the second converted video frame. When the distribution list corresponding to the second decoded video frame contains other data input conditions in other distribution strategy data, data conversion processing can be performed on the second decoded video frame to obtain other converted video frames. Since the second converted video frame is obtained based on the target data input conditions corresponding to the target model identifier, after obtaining the second converted video frame, it can be distributed to the video analysis model indicated by the target model identifier, i.e., the target video analysis model, for analysis and processing to obtain the second video analysis result. Since other converted video frames are obtained based on other data input conditions corresponding to other model identifiers, when other converted video frames are obtained, they can be distributed to the video analysis models represented by other model identifiers, i.e., other video analysis models perform analysis and processing to obtain other video analysis results. When the distribution list corresponding to the second decoded video frame is empty, it means that the second decoded video frame does not need to be distributed. Therefore, the second decoded video frame can be directly discarded, and the process described above continues to determine whether the next second decoded video frame should be distributed, until the target video analysis task is stopped.

[0211] For example, please see Figure 4Assume there are two distribution strategy data in the distribution strategy list corresponding to the target video stream identifier: target distribution strategy data D1 corresponding to the target video analysis task T1 and other distribution strategy data D2 corresponding to other video analysis tasks T2. The video analysis frame rate F1 in target distribution strategy data D1 is 10fps, and the video analysis frame rate F2 in other distribution strategy data D2 is 3fps. The frame rate of the video stream to be analyzed is 30fps. The second video frame filtering strategy is to extract one frame every 3 frames based on the video analysis frame rate F1 and one frame every 10 frames based on the video analysis frame rate F2. Then, the 3rd, 6th, 9th, 12th, 15th, 18th, 21st, 24th, 27th, and 30th frames decoded in 1 second need to be sent to the target video analysis model M1 used by the target video analysis task T1. The target model identifier and target data input conditions of the target video analysis model M1, such as the image data format and resolution required by the target video analysis model M1, can be added to the distribution list of the second decoded video frames. Using the same strategy, frames 10, 20, and 30 need to be sent to other video analysis models M2 used by other video analysis tasks T2. Therefore, the other model identifiers and other data input conditions of these other video analysis models M2, such as the image data format and resolution required by M2, can be added to the distribution list of the second decoded video frame. Thus, the second decoded video frames extracted within one second are frames 3, 6, 9, 10, 12, 15, 18, 20, 21, 24, 27, and 30. Other second decoded video frames are discarded without further image data processing. The 30th second decoded video frame needs to be simultaneously distributed to the target video analysis model M1 and both video analysis models M2. When processing the 30th second decoded video frame, the distribution list will contain two data items: the target model identifier and target data input conditions corresponding to the target video analysis model M1, and the other model identifiers and other data input conditions corresponding to the other video analysis models M2. When processing other second-decoded video frames, the distribution list contains only one data item. Specifically, the distribution lists for frames 3, 6, 9, 12, 15, 18, 21, 24, and 27 contain the target model identifier and target data input conditions corresponding to the target video analysis model M1. The distribution lists for frames 10 and 20 contain the other model identifiers and other data input conditions corresponding to the other video analysis model M2.Assuming the second decoded video frame is 1920*1080 YUV image data, the target data input condition is that the image data format required by the target video analysis model M1 is RGB format with a resolution of 256*256, and the other data input condition is that the image data format required by other video analysis models M2 is YUV format with a resolution of 384*640. Then, the second decoded video frames 3, 6, 9, 12, 15, 18, 21, 24, and 27 are subjected to data conversion processing, such as image data format conversion and scaling to second converted video frames with a resolution of 256*256 and format of RGB, and are distributed to the video analysis model identified by the target model identifier, i.e., the target video analysis model M1. The second decoded video frames 10 and 20 are scaled to other converted video frames with a resolution of 384*640 and format of YUV, and are distributed to the video analysis models identified by other model identifiers, i.e., other video analysis models M2. The 30th frame, after being decoded for the second time, undergoes two image processing steps. First, it undergoes image data format conversion and scaling to a 256*256 resolution, RGB format second converted video frame, which is then distributed to the video analysis model identified by the target model identifier, i.e., target video analysis model M1. Then, it is scaled to a resolution of 384*640, YUV format, and distributed to other video analysis models identified by other model identifiers, i.e., other video analysis models M2. Description information corresponding to the second converted video frame can also be distributed to target video analysis model M1, and description information corresponding to other converted video frames can be distributed to other video analysis models M2. The description information corresponding to the second converted video frame carries information such as the target video stream identifier and the timestamp of the second converted video frame. The description information corresponding to other converted video frames carries information such as the target video stream identifier and the timestamp of the other converted video frame.

[0212] The target video analysis model M1 and other video analysis models M2 respectively receive the second converted video frame and its corresponding descriptive information, and other converted video frames and their corresponding descriptive information. The target video analysis model M1 analyzes and processes the second converted video frame to obtain the second video analysis result, and returns the second video analysis result and its corresponding descriptive information. The other video analysis models M2 analyze and process the other converted video frames to obtain other video analysis results, and return the other video analysis results and their corresponding descriptive information. After obtaining the second video analysis result, post-processing can be performed on it to obtain a second post-processed result, and this post-processed result and its corresponding descriptive information can be returned. Similarly, after obtaining other video analysis results, post-processing can be performed on them to obtain other post-processed results, and these post-processed results and their corresponding descriptive information can be returned.

[0213] In some embodiments, after obtaining the video stream data list, the method further includes:

[0214] When the target video stream data corresponding to the target video stream identifier exists in the video stream data list, target distribution strategy data is generated based on the target model identifier, target data input conditions and video analysis frame rate, and the target distribution strategy data is added to the distribution strategy list included in the target video stream data corresponding to the target video stream identifier.

[0215] When a video stream to be analyzed exists, before obtaining the first decoded video frame corresponding to the video stream to be analyzed, the process also includes:

[0216] If the target video stream data corresponding to the target video stream identifier exists in the video stream data list, it is determined that there is a video stream to be analyzed.

[0217] It is understandable that when the target video stream data corresponding to the video stream to be analyzed exists in the video stream data list, it can be determined that the video stream to be analyzed has been used by other video analysis tasks. That is, the server has already pulled and decoded the video stream to be analyzed based on other video analysis tasks, and generated target video stream data corresponding to the target video stream identifier based on the task parameters carried by the execution instructions of other video analysis tasks. The target video stream data includes a distribution strategy list corresponding to the target video stream identifier. The distribution strategy list includes other distribution strategy data corresponding to other video analysis tasks. The other distribution strategy data includes other model identifiers of other video analysis models used by other video analysis tasks to analyze the video stream to be analyzed, other required data input conditions, and video analysis frame rates of other video analysis tasks. Since the video analysis models used by other video analysis tasks to analyze the video stream to be analyzed are different from the target video analysis model, target distribution strategy data can be generated based on the target model identifier, target data input conditions, and video analysis frame rate of the target video analysis task, and the target distribution strategy data can be added to the distribution strategy list corresponding to the target video stream identifier. That is, at this time, the distribution strategy list corresponding to the target video stream identifier includes other distribution strategy data and target distribution strategy data.

[0218] Understandably, when the server has already retrieved and decoded the video stream to be analyzed based on other video analysis tasks, it can be determined that the video stream to be analyzed exists, along with the first decoded video frame obtained after decoding the video stream. Therefore, the first decoded video frame can be directly obtained. Once the first decoded video frame is obtained, the first video frame to be analyzed can be determined from the first decoded video frame according to the video analysis frame rate and the first video frame filtering strategy in the target distribution strategy data. After obtaining the first video frame to be analyzed, data conversion processing can be performed on the first video frame to be analyzed based on the target data input conditions in the target distribution strategy data to obtain the first converted video frame, thus converting the first video frame to be analyzed into a data format supported by the target video analysis model. After obtaining the first converted video frame, it can be distributed to the target video analysis model for analysis processing based on the target model identifier in the target distribution strategy data to obtain the first video analysis result.

[0219] In some embodiments, the execution instruction for the target video analysis task further carries a task identifier for the target video analysis task, and after receiving the execution instruction for the target video analysis task, it further includes:

[0220] Based on the task identifier, target video stream identifier, and target model identifier, generate task data;

[0221] Add the task data to the task data list.

[0222] The target task information includes a task identifier, which can be used to generate task data based on the task identifier, the target video stream identifier, and the target model identifier; and the task data can be added to the task data list.

[0223] In some embodiments, the task information may also include other descriptive information about the task, such as control information like whether to output features or results. Therefore, when generating task data based on the task identifier, target video stream identifier, and target model identifier, the task data can be generated according to the following task data structure, based on the task identifier, target video stream identifier, target model identifier, and other descriptive information about the task. The task data structure may include fields such as task identifier, video stream identifier, model identifier, and other descriptive information about the task.

[0224] It should be noted that the above is merely an example of a task data structure and is not intended to limit this application. In practical applications, task data can also be generated according to other task data structures, and no specific restrictions are imposed here.

[0225] When returning the first video analysis result, the second video analysis result, the first post-processed data, or the second post-processed data, the task startup status can also be reported, such as the reported task startup status being "started". Resource usage information can also be updated and reported, such as computing power usage and memory consumption.

[0226] After receiving the execution instruction for the target video analysis task, during the execution of subsequent steps, the task startup status can be reported upon successful execution of each step, such as reporting the target video analysis task as "Starting". If a step fails, the task startup status can also be reported, such as reporting the target video analysis task as "Not Started", and the reason for the failure can be included. External devices or the task management unit can then trigger a retry based on this reason, such as resending the execution instruction to the server or to another server to restart the target video analysis task.

[0227] It should be noted that in the embodiments of this disclosure, the specific implementation process of "determining the first video frame to be analyzed from the first decoded video frame; when the target video analysis model is in the running state, analyzing and processing the first video frame to be analyzed through the target video analysis model to obtain the first video analysis result" can be the same as the specific implementation process of "determining the second video frame to be analyzed from the second decoded video frame; when the target video analysis model is in the running state, analyzing and processing the second video frame to be analyzed through the target video analysis model to obtain the second video analysis result". However, the embodiments of this disclosure emphasize different aspects of the above two specific implementation processes. Therefore, any part not described in one specific implementation process can be referred to the other specific implementation process, and will not be repeated in the embodiments of this disclosure.

[0228] In some embodiments, such as Figure 5 As shown, the processing steps after receiving an execution instruction for a video analysis task, such as a target video analysis task, may include:

[0229] Step 301: Parse the task parameters carried by the execution instructions of the target video analysis task.

[0230] Step 302: Update the task management information database based on the parsed task parameters. The task management information database includes a task data list, a model data list, and a video stream data list.

[0231] like Figure 6 As shown, step 302 includes:

[0232] Step 30201: Extract target task information, target video stream information, and target model information through task parameters.

[0233] Step 30202: Generate task data based on the target task information and task data structure, and add it to the task data list.

[0234] Step 30203: Query the model data list based on the target model identifier.

[0235] Step 30204: Determine if the target model identifier already exists in the model data list. If the target model identifier does not exist in the model data list, proceed to step 30205. If the target model identifier exists in the model data list, proceed to step 30206.

[0236] Step 30205: Based on the target model information and model data structure, generate the target model data corresponding to the target model identifier and add it to the model data list. The target model data includes the target model identifier.

[0237] Step 30206: Update the reference count in the model data corresponding to the target model identifier in the model data list, and increment the reference count by 1.

[0238] Step 30207: Query the video stream data list based on the target video stream identifier.

[0239] Step 30208: Determine whether the target video stream identifier already exists in the video stream data list. If the target video stream identifier does not exist in the video stream data list, proceed to step 30209. If the target video stream identifier exists in the video stream data list, proceed to step 30310.

[0240] Step 30209: Based on the target video stream information and video stream data structure, generate the target video stream data corresponding to the target video stream identifier and add it to the video stream data list. The target video stream data includes the target video stream identifier.

[0241] Step 30210: Update the distribution strategy list of target video stream data corresponding to the target video stream identifier in the video stream data list. That is, generate target distribution strategy data based on target model information and target task information, and add it to the distribution strategy list.

[0242] Step 303: When updating the task management information database in step 302, if the target model identifier is newly added to the model data list, it means that no video analysis task is currently using the target video analysis model for analysis and processing, then proceed to step 304. If the target model identifier already exists in the model data list, it means that other video analysis tasks are currently using the target video analysis model for analysis and processing, and the target video analysis model has been deployed and loaded for operation, then proceed to step 309.

[0243] Step 304: Download the target video analysis model package from the download address and deploy the target video analysis model. After successful deployment, execute step 302 to update the deployment status of the target model identifier in the target model data list in the task management information database.

[0244] Step 305: Based on the main executable file and other information, load and run the target video analysis model. After the target video analysis model runs successfully, execute step 302 to update the running status of the target model identifier in the target model data list in the task management information database.

[0245] Step 306: When updating the task management information database in step 302, if the target video stream identifier is newly added to the video stream data list, it means that no video analysis task is currently using this video stream for analysis, and step 307 is executed. If the target video stream identifier already exists in the video stream data list, it means that other video analysis tasks are currently using this video stream for analysis, and the video stream is already being retrieved and decoded, then step 307 is skipped, and step 308 is executed according to the updated distribution strategy list.

[0246] Step 307: Retrieve the video stream to be analyzed from the video stream address of the video stream to be analyzed, initialize the decoder, and decode the video stream to be analyzed.

[0247] Step 308: For each decoded video frame, process and distribute the decoded video frame according to the distribution strategy list. That is, determine whether the frame needs to be distributed according to the video analysis frame rate in the distribution strategy list. If it does not need to be distributed, discard it directly. If it needs to be distributed, add the corresponding model identifier and data input conditions to the distribution list of the decoded video frame, perform data conversion processing according to the corresponding data input conditions in the distribution list, and distribute the converted video frame to the video analysis model indicated by the corresponding model identifier.

[0248] like Figure 7 As shown, step 308 includes:

[0249] Step 3081: Decode a decoded video frame and read the distribution strategy list in the target video stream data corresponding to the target video stream identifier.

[0250] Step 3082: Based on the video analysis frame rate in each distribution strategy data in the distribution strategy list, determine whether the decoded video frame should be distributed according to the video frame filtering strategy.

[0251] Step 3083: If the decoded video frame needs to be distributed, the model identifier and data input conditions in the corresponding distribution strategy data, such as the target model identifier and target data input conditions in the target distribution strategy data, are added to the distribution list corresponding to the decoded video frame.

[0252] Step 3084: Determine if the distribution list corresponding to the decoded video frame is empty. If the distribution list corresponding to the decoded video frame is empty, proceed to step 3085. If the distribution list corresponding to the decoded video frame is not empty, proceed to step 3086.

[0253] Step 3085: Discard the decoded video frame without further processing or distribution.

[0254] Step 3086: Based on the data input conditions in the distribution list, such as the target data input conditions required by the target video analysis model, perform data conversion processing on the decoded video frame and distribute it to the corresponding video analysis model, such as distributing it to the target video analysis model referred to by the target model identifier corresponding to the target data input conditions.

[0255] Step 309: The video analysis model receives the converted video frames and analyzes and processes the received video frames to obtain the video analysis results.

[0256] Step 310: Post-process the video analysis results to obtain the post-processed results.

[0257] Step 311: Report the task start status.

[0258] Step 312: Update and report resource usage information.

[0259] In some embodiments, the video analysis method further includes:

[0260] Receive a stop command for the target video analysis task. The stop command is used to indicate that the execution of the target video analysis task should be stopped.

[0261] Stop executing the target video analysis task.

[0262] For example, an external device can automatically send a stop command for the target video analysis task to the server according to a second preset rule, and the server will then receive the stop command. Upon receiving the stop command, the server can stop executing the target video analysis task.

[0263] For example, a multi-task visual management system in a smart park is preset to "stop the face detection task for the video stream of the camera equipment at the east gate of the park after 18:00 every day". When 18:00 arrives, the external device automatically sends a stop command to the server for the target video analysis task, so as to stop the face detection task for the video stream of the camera equipment at the east gate of the park.

[0264] For example, administrators can manually send a stop command to the server for the target video analysis task via external devices, such as operating computers, mobile phones, and tablets in the monitoring center of a smart park. Upon receiving the stop command, the server can then cease executing the target video analysis task.

[0265] For example, the server may include a task management unit and a receiving unit. The task management unit can automatically send a stop command for the target video analysis task to the receiving unit according to a second preset rule, or the administrator can manually send a stop command for the target video analysis task to the receiving unit through the task management unit, thereby receiving the stop command for the target video analysis task.

[0266] The second preset rule can be set in advance, and no specific restrictions are imposed here.

[0267] In some embodiments, since the target video analysis task analyzes and processes the video stream to be analyzed through a target video analysis model, upon receiving a stop instruction for the target video analysis task, the execution of steps related to the target video analysis task can be directly stopped to halt the execution of the target video analysis task. For example, if the current step is "determining the first video frame to be analyzed from the first decoded video frame" or "determining the second video frame to be analyzed from the second decoded video frame," then the execution of that step can be directly stopped. Similarly, if the current step is "when the target video analysis model is in a running state, analyzing and processing the first video frame to be analyzed through the target video analysis model to obtain the first video analysis result" or "when the target video analysis model is in a running state, analyzing and processing the second video frame to be analyzed through the target video analysis model to obtain the second video analysis result," then the execution of that step can be directly stopped to halt the execution of the target video analysis task.

[0268] In some embodiments, stopping the execution of the target video analysis task includes:

[0269] Remove the task data from the task data list;

[0270] Remove the target distribution strategy data from the distribution strategy list;

[0271] Reduce the target reference count from the current count by a preset number.

[0272] Understandably, when it's necessary to stop executing the target video analysis task, the task data list no longer needs to record the task data for the target video analysis task. Therefore, the task data for the target video analysis task can be deleted from the task data list. Similarly, when it's necessary to stop executing the target video analysis task, the distribution strategy list included in the target video stream data no longer needs to record the target distribution strategy data. Therefore, the target distribution strategy data can be deleted from the distribution strategy list corresponding to the target video stream identifier, thereby stopping the distribution of the first or second converted video frames to the target video analysis model. This is how the execution of the target video analysis task is stopped.

[0273] Since the target distribution strategy data has been removed from the distribution strategy list corresponding to the target video stream identifier, the first video frame to be analyzed will no longer be selected from the first decoded video frame or the second video frame to be analyzed will no longer be selected from the second decoded video frame according to the video analysis frame rate in the target distribution strategy data. Therefore, the first video frame to be analyzed will no longer be converted into the first converted video frame or the second video frame to be analyzed will no longer be converted into the second converted video frame. Consequently, the first converted video frame or the second converted video frame will no longer be distributed to the target video analysis model. In other words, the target video analysis model will no longer analyze and process the video stream to be analyzed, and thus the target video analysis task will be stopped.

[0274] When it is necessary to stop executing the target video analysis task, it indicates that the number of video analysis tasks using the target video analysis model has decreased. In this case, the target reference count corresponding to the target model identifier can be reduced by a preset number from the current number to update the target reference count corresponding to the target model identifier, and the number of video analysis tasks using the target video analysis model can be updated and recorded.

[0275] For example, assuming the current count is 4, it means that the number of video analysis tasks using the target video analysis model is 4. Then, the target reference count can be reduced by 1 from 4, so that the updated target reference count is 3, which means that the number of video analysis tasks using the target video analysis model is 3.

[0276] Specifically, the stop command for the target video analysis task carries task parameters, which include task information, video stream information, and algorithm information.

[0277] The task information includes a task identifier. After parsing the task parameters, the task data can be retrieved from the task data list using the task identifier, and then the task data can be deleted from the task data list.

[0278] The video stream information includes a target video stream identifier, and the model information includes a target model identifier. The target video stream data can be retrieved from the video stream data list using the target video stream identifier. Then, the target distribution strategy data can be retrieved from the distribution strategy list in the target video stream data using the target model identifier, and the target distribution strategy data can be deleted from the distribution strategy list.

[0279] You can also query the target model data from the model data list using the target model identifier, and reduce the target reference count in the target model data from the current count to a preset count.

[0280] In some embodiments, after reducing the target reference count from the current count by a preset number, the method further includes:

[0281] When the number of times a target is referenced is less than the preset number, the target model data corresponding to the target model identifier will be deleted from the model data list, and the target video analysis model will be stopped from running.

[0282] It is understandable that when the target reference count corresponding to the target model identifier is the preset number, it means that only one video analysis task is currently using the target video analysis model. When the target reference count corresponding to the target model identifier is less than the preset number, it means that no video analysis task is currently using the target video analysis model. In this case, the target model data does not need to be recorded in the model data list. Therefore, the target model data can be deleted from the model data list, and the target video analysis model can be stopped to release the corresponding resources.

[0283] In some embodiments, after removing the target distribution policy data from the distribution policy list, the method further includes:

[0284] When the distribution strategy list is empty, the target video stream data corresponding to the target video stream identifier is deleted from the video stream data list, and the fetching of the video stream to be analyzed is stopped.

[0285] When the distribution strategy list is empty, meaning there is no distribution strategy data in the distribution strategy list, it means that no video analysis task is currently using the video stream to be analyzed. Therefore, the target video stream data does not need to be recorded in the video stream data list. Thus, the target video stream data can be deleted from the video stream data list, and the fetching of the video stream to be analyzed can be stopped to release the corresponding resources.

[0286] In some embodiments, after stopping the execution of the target video analysis task, the task stop status can also be reported, such as reporting that the target video analysis task is in a stopped state. Resource usage information can also be updated and reported, such as computing power usage and memory consumption.

[0287] In some embodiments, such as Figure 8 As shown, the processing procedure after receiving a stop command from a video analysis task, such as the target video analysis task, may include:

[0288] Step 401: Parse the task parameters carried by the stop command of the video analysis task.

[0289] Step 402: Update the task management information database based on the parsed task parameters.

[0290] Among them, such as Figure 9 As shown, step 402 includes:

[0291] Step 40201: Extract task information, video stream information, and model information through task parameters.

[0292] The task information includes a task identifier, the video stream information includes a target video stream identifier, and the model information includes a target model identifier.

[0293] Step 40202: Delete the task data containing the task identifier in the task data list.

[0294] Step 40203: Query the model data list based on the target model identifier.

[0295] Step 40204: Update the reference count in the target model data corresponding to the target model identifier in the model data list, and decrement the reference count by 1.

[0296] Step 40205: Determine whether the number of references in the target model data corresponding to the target model identifier is 0. If it is 0, proceed to step 40206.

[0297] Step 40206: Delete the target model data corresponding to the target model identifier in the model data list.

[0298] Step 40207: Query the video stream data list based on the target video stream identifier.

[0299] Step 40208: Update the distribution strategy list in the target video stream data corresponding to the target video stream identifier in the video stream data list, and delete the target distribution strategy data corresponding to the target model identifier from the distribution strategy list.

[0300] Step 40209: Determine whether the distribution strategy list in the target video stream data corresponding to the target video stream identifier is empty. If the distribution strategy list is empty, proceed to step 40210.

[0301] Step 40210: Delete the target video stream data corresponding to the target video stream identifier in the video stream data list.

[0302] Step 403: When updating the task management information database in step 402, if the reference count in the target model data corresponding to the target model identifier is 0, it means that no video analysis task is currently using the target video analysis model for analysis and processing, then proceed to step 404. If the reference count is not 0, it means that there are still other video analysis tasks using the target video analysis model for analysis and processing, and the target video analysis model still needs to be run, then proceed to step 408.

[0303] Step 404: Stop running the target video analysis model and release the corresponding resources.

[0304] Step 405: When updating the task management information database in step 402, if the distribution strategy list in the target video stream data corresponding to the target video stream identifier is empty, it means that no video analysis task is currently using the video stream to be analyzed for analysis, then proceed to step 406. If the distribution strategy list is not empty, it means that other video analysis tasks are currently using the video stream to be analyzed for analysis, and the video stream to be analyzed still needs to be pulled and decoded before being distributed to other video analysis models for analysis and processing, then proceed to step 207.

[0305] Step 406: Stop pulling and decoding the video stream to be analyzed, and release the corresponding resources such as the decoder.

[0306] Step 407: Based on the updated distribution strategy list, stop distributing decoded frames to the target video analysis model. Other video analysis tasks currently using this video stream for analysis are unaffected and continue to process and distribute them according to the distribution strategy list.

[0307] Step 408: Report the task stop status.

[0308] Step 409: Update and report resource usage information.

[0309] As described above, when multiple tasks use the same video stream for analysis, the video stream is only fetched and decoded once, and image data is processed and distributed on demand according to the distribution strategy. When multiple tasks use the same model to analyze multiple video streams, the model is also only deployed and loaded into the system once. Sharing resources and allocating and scheduling resources on demand minimizes resource consumption, meets the resource needs of multiple tasks, and ensures the independence of each task.

[0310] As can be seen from the above, the video analysis method provided in this disclosure addresses the problem of significant waste of system resources caused by traditional task management methods in multi-task visual management systems with multiple video streams and multiple algorithms. It proposes a method for task management based on resource dimensions. By maintaining a multi-dimensional mapping relationship between tasks, video streams, and models, the method manages tasks, allocates and schedules video streams and models as two major resources, and manages their lifecycles. The key innovations include at least the following aspects.

[0311] (1) By parsing task parameters, task information, video stream information, and model information are extracted, and task management information in the system is established, updated, and maintained. The task management information maintains the multi-dimensional mapping relationship between tasks, video streams, and models. Among them, the task data stores relevant information such as the core task identifier (ID) corresponding to the video stream identifier (ID) and model identifier (ID). The model data stores relevant information such as the core model ID, deployment status, running status, data format and resolution required for model input, and the number of times the task references the model method. The video stream data stores the core video stream ID and the video stream distribution strategy list. The distribution strategy list contains the model ID, the image data format and resolution required for model input, and the video analysis frame rate, etc. The task management information is dynamically maintained according to the start and stop of each task in the system.

[0312] (2) The lifecycle of video streams used by each task in the system is managed and allocated according to the task management information. When a video stream is used for the first time in the system, resources are scheduled to pull the video stream and decode it. Subsequent tasks using the same video stream will not pull the video stream and decode it again. When different tasks in the system use different models to analyze the video stream, the image data is processed and distributed as needed according to the video stream distribution strategy list in the task management information. When a task using the video stream stops, it is only necessary to update the video stream distribution strategy list and stop distributing image data to the model used by that task. The pulling and decoding of the video stream will stop and the corresponding resources will be released when the last task using the video stream in the system stops. The lifecycle of the video stream is dynamically managed and scheduled according to the lifecycle of the multiple tasks in the system to ensure that a video stream is pulled and decoded only once. The decoded image data is processed and distributed according to the distribution strategy to ensure that the image data is allocated as needed and to meet the needs of different tasks.

[0313] (3) Generate and maintain the distribution strategy for each video stream in real time based on the task management information, and process and distribute the image data according to the distribution strategy. The video stream analyzes the frame rate of each video in the distribution strategy, selects a video frame filtering strategy to determine whether the decoded frame needs to be distributed, and adds the model information that needs to be distributed to the distribution list of the frame. If the frame needs to be distributed, the decoded frame is processed by image data format conversion and scaling according to the required image data format and resolution of each model input in the distribution list, and then moved from the video memory to the memory and distributed to the corresponding algorithm. If no distribution is required, the frame is discarded directly and the next video frame is decoded.

[0314] (4) The lifecycle of the models used by each task in the system is managed and scheduled according to the task management information. When a model is used for the first time in the system, the model is downloaded and deployed, then loaded and run, receiving image data from different video streams for inference, and performing post-processing of the result data. The result data includes the video stream ID and timestamp information carried in the processed image data, so as to ensure a globally unified spatiotemporal ID when reporting algorithm results later. The model is stopped and the corresponding resources are released only when the last task using the model in the system stops. The lifecycle of the model is managed and scheduled according to the lifecycle of the multiple tasks in the system, ensuring that only one model is deployed in the system, only one model is loaded and run, processing image data and algorithm results from multiple video streams, returning algorithm results for different tasks, and meeting the needs of different tasks at the same time.

[0315] (5) Task management is carried out in terms of resources. Based on the task management information, the life cycle of multiple video streams and multiple models in the system is managed and scheduled. Resources are shared among multiple tasks and resources are allocated and scheduled as needed to minimize resource consumption. At the same time, the independence of multiple tasks is guaranteed and they do not interfere with each other.

[0316] As can be seen from the above, for a multi-task visual management system with multiple video streams and multiple models, the video analysis method provided in this disclosure establishes a multi-dimensional mapping relationship between tasks, video streams, and models to maintain the two core resources in video analysis tasks, namely video streams and models. This mapping relationship is dynamically maintained based on the start and stop of multiple tasks, thereby enabling the management and scheduling of tasks and resources. By maintaining information on tasks, video streams, and models in the task management information, task management can be performed. This allows for on-demand allocation and scheduling of the two major resources, video streams and models, according to task requirements. Furthermore, it manages the lifecycle of these resources based on the needs of multiple tasks, while ensuring maximum resource sharing among multiple tasks without mutual interference.

[0317] The system only triggers resource allocation for video stream fetching and decoding when a task first uses a video stream for processing. Subsequent tasks using the same video stream only require image data processing and distribution as needed, based on the video analysis frame rate and model input for each task. Video stream fetching and decoding only cease after the last task using that video stream has finished processing.

[0318] Similarly, the model is triggered only when a task uses a model for processing for the first time, at which point the corresponding unit downloads and deploys the model, loads and runs it, and performs inference. Subsequent tasks using the same model only need to receive the decoded image data from the video streams of different tasks, perform inference and post-processing of the results, and then return the algorithm results. The model is stopped only after the last task using it stops.

[0319] When multiple tasks use the same video stream for processing, the system only fetches and decodes the video stream once. Then, based on the distribution strategy list, it performs image data preprocessing for different models and distributes the video analysis frame rates according to demand. Compared to the same video stream requiring multiple decodings for different tasks, and the increased I / O bandwidth consumption from moving video frames from memory to the video memory of multiple decoders, performing decoding only once significantly saves on wasted hardware decoding resources and the I / O bandwidth consumption from large-scale data movement. This increases the system's processing capacity for video streams and improves overall system performance. Furthermore, after filtering decoded frames based on the video analysis frame rate, it's unnecessary to distribute every decoded frame to the model, saving the I / O bandwidth required to move each decoded frame from video memory to memory for algorithm distribution. Only after decoding, and based on the required model input, are the decoded frames processed for image data such as video format conversion and scaling before being distributed to the algorithm. This simultaneously saves video image processing resources and the I / O bandwidth consumed by moving data between memory and video memory, effectively utilizing the hardware's acceleration capabilities for video image processing and improving the speed of video data processing and task execution. After format conversion and scaling, the image data will be smaller than the decoded original size image data. Copying it from the video memory to the user memory and distributing it to the model at this time will greatly reduce the IO bandwidth consumption caused by moving large blocks of data between the video memory and the user memory.

[0320] When using the same model for multiple tasks, the system downloads and deploys only one copy of the model, loads and runs only one copy, and then processes image and result data from different video streams simultaneously. The size of the model's parameters directly affects the consumption of system resources such as video memory and RAM during loading and runtime. Loading and running the model only once, especially for models with larger parameters, significantly reduces the consumption of video memory and RAM, improving the overall inference and computation performance of the system.

[0321] Compared to related technologies where each task independently pulls and decodes streams and deploys independently, the video analysis method provided in this disclosure greatly saves system video image processing resources, computing power resources, and resources such as video memory, RAM, and I / O bandwidth. It can increase the number of video channels that the system can access and process, load and run more algorithms to meet the needs of different tasks, and improve the overall system performance and capabilities.

[0322] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of a video analysis device provided in an embodiment of the present disclosure. The video analysis device is applied to a server and may include a receiving unit 501, an acquisition unit 502, a determining unit 503, and a processing unit 504, etc.

[0323] The receiving unit 501 is used to receive the execution instruction of the target video analysis task, wherein the execution instruction is used to instruct the video stream to be analyzed and processed by the target video analysis model;

[0324] The acquisition unit 502 is used to acquire the first decoded video frame corresponding to the video stream to be analyzed when the video stream to be analyzed exists.

[0325] The determining unit 503 is used to determine the first video frame to be analyzed from the first decoded video frame;

[0326] The processing unit 504 is used to analyze and process the first video frame to be analyzed through the target video analysis model when the target video analysis model is in the running state, and obtain the first video analysis result.

[0327] In some embodiments, the video analysis device further includes a deployment unit and an operation unit. The deployment unit is configured to deploy the target video analysis model when the target video analysis model is in an unrunning state and the target video analysis model is in an undeployed state, so that the deployment state of the target video analysis model is in a deployed state.

[0328] The running unit is used to load and run the target video analysis model when the deployment status of the target video analysis model is "deployed", so that the running status of the target video analysis model is "running".

[0329] In some embodiments, the determining unit 503 is specifically used for:

[0330] According to the video analysis frame rate and the first video frame filtering strategy, the first video frame to be analyzed is determined from the first decoded video frame.

[0331] The processing unit 504 is specifically used for:

[0332] According to the target data input conditions required by the target video analysis model, the first video frame to be analyzed is subjected to data conversion processing to obtain the first converted video frame.

[0333] The first converted video frame is input into the target video analysis model for analysis and processing to obtain the first video analysis result.

[0334] In some embodiments, the video analysis device further includes a fetching unit and a decoding unit, wherein the fetching unit is used to fetch the video stream to be analyzed when the video stream to be analyzed does not exist;

[0335] The decoding unit is used to decode the video stream to be analyzed to obtain a second decoded video frame;

[0336] The determining unit 503 is used to determine the second video frame to be analyzed from the second decoded video frame;

[0337] The processing unit 504 is used to analyze and process the second video frame to be analyzed through the target video analysis model when the target video analysis model is in the running state, so as to obtain the second video analysis result.

[0338] In some embodiments, the execution instruction further carries a target model identifier of the target video analysis model, and the determining unit 503 is specifically used for:

[0339] For each second decoded video frame obtained, determine whether the target model identifier and the target data input conditions need to be added to the distribution list corresponding to the obtained second decoded video frame according to the video analysis frame rate and the second video frame filtering strategy.

[0340] If it is necessary to add the target model identifier and the target data input conditions to the distribution list corresponding to the obtained second decoded video frame, then add the target model identifier and the target data input conditions to the distribution list corresponding to the obtained second decoded video frame, and determine the second decoded video frame containing the target model identifier and the target data input conditions in the corresponding distribution list as the second video frame to be analyzed;

[0341] The processing unit 504 is specifically used for:

[0342] According to the target data input conditions in the distribution list corresponding to the second video frame to be analyzed, the second video frame to be analyzed is subjected to data conversion processing to obtain the second converted video frame.

[0343] Based on the target model identifier in the distribution list corresponding to the second video frame to be analyzed, the second converted video frame is distributed to the target video analysis model for analysis and processing to obtain the second video analysis result.

[0344] In some embodiments, the video analysis apparatus further includes an update unit, the update unit being configured to:

[0345] Obtain a model data list, which is used to record the model data corresponding to the model identifiers of different video analysis models;

[0346] When the target model data corresponding to the target model identifier does not exist in the model data list, the deployment status of the target video analysis model is determined to be undeployed, the running status of the target video analysis model is determined to be unrunning, and the target reference count corresponding to the target model identifier is set to a preset number.

[0347] Based on the target model identifier, the undeployed state, the non-running state, the target reference count, and the target data input conditions, target model data corresponding to the target model identifier is generated, and the target model data corresponding to the target model identifier is added to the model data list;

[0348] When the deployment status of the target video analysis model is "deployed", the non-deployed status of the target model data corresponding to the target model identifier is updated to "deployed".

[0349] When the target video analysis model is in the running state, the non-running state included in the target model data corresponding to the target model identifier is updated to the running state.

[0350] In some embodiments, the updating unit is further configured to:

[0351] When the target model data corresponding to the target model identifier exists in the model data list, the target reference count included in the target model data corresponding to the target model identifier is increased by a preset number from the current count.

[0352] In some embodiments, the execution instruction further carries the target video stream identifier of the video stream to be analyzed and the video stream address of the video stream to be analyzed, and the update unit is further configured to:

[0353] Obtain a video stream data list, which is used to record the video stream data corresponding to the video stream identifiers of different video streams;

[0354] When the target video stream data corresponding to the target video stream identifier does not exist in the video stream data list, it is determined that the video stream to be analyzed does not exist, and a distribution strategy list corresponding to the target video stream identifier is created;

[0355] Based on the target model identifier, the target data input conditions, and the video analysis frame rate, target distribution strategy data is generated and added to the distribution strategy list;

[0356] Based on the target video stream identifier, the distribution strategy list, and the video stream address, target video stream data corresponding to the target video stream identifier is generated, and the target video stream data corresponding to the target video stream identifier is added to the video stream data list;

[0357] The pull unit is specifically used for:

[0358] The video stream to be analyzed is retrieved using the video stream address of the video stream to be analyzed;

[0359] The determining unit 503 is specifically used for:

[0360] For each second decoded video frame obtained, based on the video analysis frame rate and the second video frame filtering strategy included in the target distribution strategy data, it is determined whether the target model identifier and target data input conditions included in the target distribution strategy data need to be added to the distribution list corresponding to the obtained second decoded video frame.

[0361] If it is necessary to add the target model identifier and target data input conditions included in the target distribution strategy data to the distribution list corresponding to the obtained second decoded video frame, then the target model identifier and target data input conditions included in the target distribution strategy data are added to the distribution list corresponding to the obtained second decoded video frame, and the second decoded video frame containing the target model identifier and target data input conditions included in the target distribution strategy data in the corresponding distribution list is determined as the second video frame to be analyzed.

[0362] In some embodiments, the updating unit is further configured to:

[0363] When the target video stream data corresponding to the target video stream identifier exists in the video stream data list, target distribution strategy data is generated based on the target model identifier, the target data input conditions and the video analysis frame rate, and the target distribution strategy data is added to the distribution strategy list included in the target video stream data corresponding to the target video stream identifier.

[0364] The determining unit 503 is further configured to:

[0365] When the target video stream data corresponding to the target video stream identifier exists in the video stream data list, it is determined that the video stream to be analyzed exists.

[0366] In some embodiments, the updating unit is further configured to:

[0367] Based on the task identifier, the target video stream identifier, and the target model identifier, task data is generated;

[0368] Add the task data to the task data list.

[0369] In some embodiments, the video analysis device further includes a stop unit, the stop unit being configured to:

[0370] Receive a stop command for the target video analysis task, the stop command being used to instruct the execution of the target video analysis task to be stopped;

[0371] Stop executing the target video analysis task.

[0372] In some embodiments, the stopping unit is specifically used for:

[0373] Remove the task data from the task data list;

[0374] Remove the target distribution strategy data from the distribution strategy list;

[0375] The target reference count is reduced by a preset number from the current count.

[0376] In some embodiments, the updating unit is further configured to:

[0377] When the number of references to the target is less than the preset number, the target model data corresponding to the target model identifier is deleted from the model data list;

[0378] The stopping unit is specifically used to: stop the operation of the target video analysis model.

[0379] In some embodiments, the updating unit is further configured to:

[0380] When the distribution strategy list is empty, the target video stream data corresponding to the target video stream identifier is deleted from the video stream data list;

[0381] The stop unit is specifically used to: stop pulling the video stream to be analyzed.

[0382] The specific implementation of each of the above units can be found in the previous embodiments, and will not be repeated here.

[0383] As described above, in this embodiment, the receiving unit 501 receives the execution instruction of the target video analysis task, which instructs the target video analysis model to analyze and process the video stream to be analyzed; the acquisition unit 502 acquires the first decoded video frame corresponding to the video stream to be analyzed when the video stream to be analyzed exists; the determination unit 503 determines the first video frame to be analyzed from the first decoded video frame; and the processing unit 504 analyzes and processes the first video frame to be analyzed through the target video analysis model when the target video analysis model is in the running state, and obtains the first video analysis result. Thus, when multiple video analysis tasks use the same video stream for analysis and processing, the same video stream will only be retrieved and decoded once. When multiple video analysis tasks use the same video analysis model to analyze and process multiple video streams, only one video analysis model will be loaded and run, which can save system resources and make the overall system resource utilization rate higher.

[0384] Figure 11 This is a partial structural block diagram of a computer device 1000 implementing an embodiment of the present disclosure. The computer device 1000 can vary significantly due to different configurations or performance characteristics, and may include one or more central processing units (CPUs) 422 (e.g., one or more processors) and a memory 432, and one or more storage media 430 (e.g., one or more mass storage devices) storing application programs 442 or data 444. The memory 432 and storage media 430 may be temporary or persistent storage. The program stored in the storage media 430 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server 500. Furthermore, the CPU 422 may be configured to communicate with the storage media 430 and execute the series of instruction operations in the storage media 430 on the server 500.

[0385] Computer device 1000 may also include one or more power supplies 425, one or more wired or wireless network interfaces 450, one or more input / output interfaces 458, and / or one or more operating systems 441, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0386] The central processing unit 422 in the computer device 1000 can be used to execute the video analysis method of the present disclosure embodiment: receiving an execution instruction for a target video analysis task, the execution instruction being used to instruct the analysis and processing of a video stream to be analyzed through a target video analysis model; when the video stream to be analyzed exists, acquiring a first decoded video frame corresponding to the video stream to be analyzed; determining a first video frame to be analyzed from the first decoded video frame; when the target video analysis model is in a running state, analyzing and processing the first video frame to be analyzed through the target video analysis model to obtain a first video analysis result.

[0387] This disclosure also provides a computer-readable storage medium for storing program code for executing the video analysis methods of the foregoing embodiments: receiving an execution instruction for a target video analysis task, the execution instruction instructing the analysis and processing of a video stream to be analyzed using a target video analysis model; when the video stream to be analyzed exists, acquiring a first decoded video frame corresponding to the video stream to be analyzed; determining a first video frame to be analyzed from the first decoded video frame; and when the target video analysis model is in a running state, analyzing and processing the first video frame to be analyzed using the target video analysis model to obtain a first video analysis result.

[0388] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the video analysis method described above.

[0389] Furthermore, the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or apparatus.

[0390] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0391] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0392] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0393] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0394] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0395] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0396] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0397] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0398] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A video analysis method, characterized in that, include: Receive the execution instruction for the target video analysis task, the execution instruction being used to instruct the target video analysis model to analyze and process the video stream to be analyzed; When the video stream to be analyzed exists, obtain the first decoded video frame corresponding to the video stream to be analyzed; The first video frame to be analyzed is determined from the first decoded video frame; When the target video analysis model is in the running state, the first video frame to be analyzed is analyzed and processed by the target video analysis model to obtain the first video analysis result; The running state of the target video analysis model is triggered by the execution instructions of the target video analysis task or other video analysis tasks, and multiple video analysis tasks share the running target video analysis model in parallel.

2. The video analysis method according to claim 1, characterized in that, Before analyzing the first video frame to be analyzed and processing by the target video analysis model to obtain the first video analysis result when the target video analysis model is in a running state, the method further includes: When the target video analysis model is in an inactive state and its deployment state is not deployed, the target video analysis model is deployed so that its deployment state is deployed. When the target video analysis model is in the deployed state, the target video analysis model is loaded and run, so that the running state of the target video analysis model is in the running state.

3. The video analysis method according to claim 2, characterized in that, The execution instruction carries the video analysis frame rate of the target video analysis task, and the step of determining the first video frame to be analyzed from the first decoded video frame includes: According to the video analysis frame rate and the first video frame filtering strategy, the first video frame to be analyzed is determined from the first decoded video frame. The step of analyzing and processing the first video frame to be analyzed using the target video analysis model to obtain the first video analysis result includes: According to the target data input conditions required by the target video analysis model, the first video frame to be analyzed is subjected to data conversion processing to obtain the first converted video frame. The first converted video frame is input into the target video analysis model for analysis and processing to obtain the first video analysis result.

4. The video analysis method according to claim 3, characterized in that, After receiving the execution instruction for the target video analysis task, the method further includes: If the video stream to be analyzed does not exist, retrieve the video stream to be analyzed. The video stream to be analyzed is decoded to obtain a second decoded video frame; The second video frame to be analyzed is determined from the second decoded video frame; When the target video analysis model is in the running state, the second video frame to be analyzed is analyzed and processed by the target video analysis model to obtain the second video analysis result.

5. The video analysis method according to claim 4, characterized in that, The execution instruction also carries the target model identifier of the target video analysis model, and the step of determining the second video frame to be analyzed from the second decoded video frame includes: For each second decoded video frame obtained, determine whether the target model identifier and the target data input conditions need to be added to the distribution list corresponding to the obtained second decoded video frame according to the video analysis frame rate and the second video frame filtering strategy. If it is necessary to add the target model identifier and the target data input conditions to the distribution list corresponding to the obtained second decoded video frame, then add the target model identifier and the target data input conditions to the distribution list corresponding to the obtained second decoded video frame, and determine the second decoded video frame containing the target model identifier and the target data input conditions in the corresponding distribution list as the second video frame to be analyzed; The step of analyzing and processing the second video frame to be analyzed using the target video analysis model to obtain the second video analysis result includes: According to the target data input conditions in the distribution list corresponding to the second video frame to be analyzed, the second video frame to be analyzed is subjected to data conversion processing to obtain the second converted video frame. Based on the target model identifier in the distribution list corresponding to the second video frame to be analyzed, the second converted video frame is distributed to the target video analysis model for analysis and processing to obtain the second video analysis result.

6. The video analysis method according to claim 5, characterized in that, After receiving the execution instruction for the target video analysis task, the method further includes: Obtain a model data list, which is used to record the model data corresponding to the model identifiers of different video analysis models; When the target model data corresponding to the target model identifier does not exist in the model data list, the deployment status of the target video analysis model is determined to be undeployed, the running status of the target video analysis model is determined to be unrunning, and the target reference count corresponding to the target model identifier is set to a preset number. Based on the target model identifier, the undeployed state, the non-running state, the target reference count, and the target data input conditions, target model data corresponding to the target model identifier is generated, and the target model data corresponding to the target model identifier is added to the model data list; When the deployment status of the target video analysis model is "deployed", the non-deployed status of the target model data corresponding to the target model identifier is updated to "deployed". When the target video analysis model is in the running state, the non-running state included in the target model data corresponding to the target model identifier is updated to the running state.

7. The video analysis method according to claim 6, characterized in that, After obtaining the model data list, the process also includes: When the target model data corresponding to the target model identifier exists in the model data list, the target reference count included in the target model data corresponding to the target model identifier is increased by a preset number from the current count.

8. The video analysis method according to claim 7, characterized in that, The execution instruction also carries the target video stream identifier and the video stream address of the video stream to be analyzed. After receiving the execution instruction for the target video analysis task, the method further includes: Obtain a video stream data list, which is used to record the video stream data corresponding to the video stream identifiers of different video streams; When the target video stream data corresponding to the target video stream identifier does not exist in the video stream data list, it is determined that the video stream to be analyzed does not exist, and a distribution strategy list corresponding to the target video stream identifier is created; Based on the target model identifier, the target data input conditions, and the video analysis frame rate, target distribution strategy data is generated and added to the distribution strategy list; Based on the target video stream identifier, the distribution strategy list, and the video stream address, target video stream data corresponding to the target video stream identifier is generated, and the target video stream data corresponding to the target video stream identifier is added to the video stream data list; The process of retrieving the video stream to be analyzed includes: The video stream to be analyzed is retrieved using the video stream address of the video stream to be analyzed; For each second decoded video frame obtained, according to the video analysis frame rate and the second video frame filtering strategy, it is determined whether the target model identifier and the target data input conditions need to be added to the distribution list corresponding to the obtained second decoded video frame, including: For each second decoded video frame obtained, based on the video analysis frame rate and the second video frame filtering strategy included in the target distribution strategy data, it is determined whether the target model identifier and target data input conditions included in the target distribution strategy data need to be added to the distribution list corresponding to the obtained second decoded video frame. If it is necessary to add the target model identifier and the target data input conditions to the distribution list corresponding to the obtained second decoded video frame, then the target model identifier and the target data input conditions are added to the distribution list corresponding to the obtained second decoded video frame, and the second decoded video frame containing the target model identifier and the target data input conditions in the corresponding distribution list is determined as the second video frame to be analyzed, including: If it is necessary to add the target model identifier and target data input conditions included in the target distribution strategy data to the distribution list corresponding to the obtained second decoded video frame, then the target model identifier and target data input conditions included in the target distribution strategy data are added to the distribution list corresponding to the obtained second decoded video frame, and the second decoded video frame containing the target model identifier and target data input conditions included in the target distribution strategy data in the corresponding distribution list is determined as the second video frame to be analyzed.

9. The video analysis method according to claim 8, characterized in that, After obtaining the video stream data list, the process also includes: When the target video stream data corresponding to the target video stream identifier exists in the video stream data list, target distribution strategy data is generated based on the target model identifier, the target data input conditions and the video analysis frame rate, and the target distribution strategy data is added to the distribution strategy list included in the target video stream data corresponding to the target video stream identifier. Before obtaining the first decoded video frame corresponding to the video stream to be analyzed when the video stream to be analyzed exists, the method further includes: When the target video stream data corresponding to the target video stream identifier exists in the video stream data list, it is determined that the video stream to be analyzed exists.

10. The video analysis method according to claim 8 or 9, characterized in that, The execution instruction also carries the task identifier of the target video analysis task, and after receiving the execution instruction of the target video analysis task, it further includes: Based on the task identifier, the target video stream identifier, and the target model identifier, task data is generated; Add the task data to the task data list.

11. The video analysis method according to claim 10, characterized in that, The method further includes: Receive a stop command for the target video analysis task, the stop command being used to instruct the execution of the target video analysis task to be stopped; Stop executing the target video analysis task.

12. The video analysis method according to claim 11, characterized in that, The step of stopping the execution of the target video analysis task includes: Remove the task data from the task data list; Remove the target distribution strategy data from the distribution strategy list; The target reference count is reduced by a preset number from the current count.

13. The video analysis method according to claim 12, characterized in that, After reducing the target reference count from the current count by a preset number, the method further includes: When the number of times the target is referenced is less than the preset number, the target model data corresponding to the target model identifier is deleted from the model data list, and the target video analysis model is stopped from running.

14. The video analysis method according to claim 12, characterized in that, After deleting the target distribution strategy data from the distribution strategy list, the method further includes: When the distribution strategy list is empty, the target video stream data corresponding to the target video stream identifier is deleted from the video stream data list, and the fetching of the video stream to be analyzed is stopped.

15. A video analysis device, characterized in that, include: The receiving unit is used to receive the execution instruction of the target video analysis task, wherein the execution instruction is used to instruct the video stream to be analyzed and processed through the target video analysis model; The acquisition unit is used to acquire the first decoded video frame corresponding to the video stream to be analyzed when the video stream to be analyzed exists. The determining unit is used to determine the first video frame to be analyzed from the first decoded video frame; The processing unit is configured to analyze and process the first video frame to be analyzed through the target video analysis model to obtain a first video analysis result when the target video analysis model is in a running state. The running state of the target video analysis model is triggered by the execution instructions of the target video analysis task or other video analysis tasks, and multiple video analysis tasks share the running target video analysis model in parallel.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a plurality of instructions adapted for loading by a processor to perform the video analysis method according to any one of claims 1 to 14.

17. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the video analysis method according to any one of claims 1 to 14.

Images