System and method for large-scale video access and AI reasoning enhancement

Abstract

The invention discloses a large-scale video access and AI reasoning enhancement system and method, belongs to the technical field of video processing, and aims to solve the technical problem of how to realize large-scale video stream efficient access, resource elastic scheduling and abnormity self-healing in a complex environment. Comprising a resource dynamic scheduling module, a decoding and reasoning control module, a resolution dynamic processing module, a multi-stage shared memory transmission module, an exception self-healing and fault-tolerant module and a task parallel execution module, and resource allocation is optimized and calculated by dynamically binding a CPU core and GPU hardware unit isolation; the resolution is dynamically adjusted based on the scene algorithm precision requirement, and invalid calculation is reduced; a stream pushing and frame rate decoding strategy is controlled by using a Redis mark, and memory occupation is reduced by combining long and short queues; transmission coding and decoding are reduced by adopting a shared memory, and the cross-process interaction efficiency is improved; and task self-healing is realized through dual anomaly detection and process-level heartbeat monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and specifically to a system and method for large-scale video access and AI inference enhancement. Background Art

[0002] With the rapid development of artificial intelligence and video surveillance technologies, the demand for real-time access and intelligent analysis of large-scale video streams is increasing day by day. However, the existing technologies face the following significant defects when processing high-concurrency video streams: low resource utilization rate, limited number of video channels supported by a single server; poor transmission efficiency, cross-process data transfer usually goes through encoding and decoding, increasing latency and CPU occupancy; weak fault tolerance, video stream interruption or process crash requires manual intervention, and the system reliability is insufficient; high computational redundancy, full-frame rate decoding and fixed-resolution processing result in ineffective computations.

[0003] How to achieve efficient access to large-scale video streams, elastic resource scheduling, and anomaly self-healing in a complex environment is a technical problem that needs to be solved. Summary of the Invention

[0004] The technical task of the present invention is to address the above deficiencies and provide a system and method for large-scale video access and AI inference enhancement to solve the technical problem of how to achieve efficient access to large-scale video streams, elastic resource scheduling, and anomaly self-healing in a complex environment.

[0005] In a first aspect, a system for large-scale video access and AI inference enhancement according to the present invention includes:

[0006] A resource dynamic scheduling module: configured to bind video task-related processes to the CPU core with the lowest utilization rate through CPU scheduling, and isolate the hardware units of the GPU through GPU scheduling, where the video task-related processes include a video process, an inference process, and a post-processing process. The video process is used to execute decoding tasks, the inference process is used to call an AI large model to perform inference tasks on the decoded image, and the post-processing process is used to execute post-processing operation logic and return an alarm message to the video process. Correspondingly, the video process is used to perform bounding box marking, encoding, and streaming tasks based on the alarm message;

[0007] A decoding and inference control module: configured to dynamically adjust the streaming flag of a video task and store the streaming flag in Redis. The inference flag is used to represent the streaming state and decoding frame rate of a video related to the video task. When the inference state is to start the streaming process, the corresponding decoding frame rate is full-frame rate decoding. When the streaming state is to stop the streaming process, the corresponding decoding frame rate is partial-frame rate decoding;

[0008] A resolution dynamic processing module: configured to dynamically adjust the resolution of the decoded image, and store the decoded image after resolution adjustment in a cache queue;

[0009] Multi-level shared memory transfer module: used to implement data interaction between different processes through a multi-level shared memory queue and dynamically adjust the capacity of the shared memory queue;

[0010] Exception self-healing and fault tolerance module: used to perform dual exception detection based on the decoded image and the inference results returned by the AI model, detect exceptions for processes related to video tasks, and perform unified scheduling and self-healing based on predefined rules;

[0011] Task parallel execution module: used to support the parallel execution of processes related to video tasks and support the parallel execution of multiple threads in a process.

[0012] Preferably, the resource dynamic scheduling module is used to perform the following:

[0013] CPU scheduling: Calculate the utilization rate of each CPU core in real time through psutil.cpu_percen. For each video task, bind the processes related to the video task to the CPU core with the lowest utilization rate, and isolate and allocate the logical cores of the same physical core to the processes of different video tasks to avoid resource competition;

[0014] GPU scheduling: Bind the decoding task and encoding task corresponding to the video task to the NVENC / NVDEC hardware unit of the GPU card, bind the inference task to the CUDA Core of the GPU card, and collect the decoding utilization rate, encoding utilization rate, and inference utilization rate of each GPU card in real time, and allocate the decoding task, encoding task, and inference task to the GPU card with the lowest utilization rate.

[0015] Preferably, for the high-precision algorithm scenario, the resolution dynamic processing module is used to maintain the original resolution of the decoded image, and for the low-precision algorithm scenario, the resolution dynamic processing module is used to perform scaling on the decoded image through the FFmpeg GPU filter chain. The mapping relationship between the scaling ratio and the algorithm type during scaling is stored in Redis.

[0016] Preferably, there are two cache queues, namely the long cache queue and the short cache queue. For video tasks with a streaming flag to start the streaming process, the decoded image after resolution adjustment is stored in the long cache queue, and for video tasks with a streaming flag to stop the streaming process, the decoded image after resolution adjustment is stored in the short cache queue.

[0017] Preferably, the multi-level shared memory transfer module is used to store data between different processes through a two-level shared memory queue;

[0018] Among them, the first-level shared memory queue is used to implement data interaction between the video process and the inference process. The data for interaction includes the decoded images obtained through the video process and information related to image indexes. The decoded images are directly stored in the shared memory queue in the format of a NUMPY array, and the information related to image indexes is stored in an ordinary information queue.

[0019] In the inference process, a producer thread is started. The producer thread respectively obtains image index information and image arrays from the ordinary information queue and the shared memory queue, and calls the AI model to perform an inference task to obtain image inference information.

[0020] The second-level shared memory queue is used to implement data interaction between the inference process and the post-processing process. The data for interaction includes image arrays and image inference information. After receiving the image arrays and image queue information, the post-processing process performs post-processing logic operations, structured assembly of data, and storage of defective images based on the requirements of the application scenario, and returns image coordinate exception information to the video process. The video process is used to perform frame marking processing based on the image coordinate exception information, where the image coordinate exception information is used as an alarm message.

[0021] Preferably, the shared memory queue is an implemented first-in, first-out queue based on multiprocessing and file locks.

[0022] Preferably, the exception self-healing and fault tolerance module is used to perform the following operations:

[0023] Dual exception detection: Continuously monitor the status of each decoded image obtained by decoding the video stream. If N consecutive decoded images are empty frames or the acquisition of decoded images fails, automatically trigger a video task stop instruction, release relevant resources, synchronously set the inference result timeout threshold, and continuously monitor the push stream result returned by the AI model. If the time to receive the inference result exceeds the inference result timeout threshold, terminate the current inference task and roll back the processing queue.

[0024] Process-level heartbeat monitoring: In the inference process, a monitoring thread is started. There is an uninterrupted heartbeat flag maintained in the monitoring thread, and the heartbeat flag is stored in Redis. Determine whether the current monitoring thread is performing an inference task through the heartbeat flag.

[0025] Parent-child process monitoring: The video process is the parent process of the inference process, the inference process is the parent process of the post-processing process, and the post-processing process is the parent process of the video process. For the video process, the inference process, and the post-processing process, regularly monitor whether their parent processes exist to prevent the inability to recycle this process due to the hanging of their parent processes.

[0026] Unified scheduling and self-healing: When it is detected that the inference process has stopped, it is judged whether the current video stream is online. If it is online, the inference task is started. If it is offline, after the configured interval, it continues to judge whether to start the inference task. Among them, the online / offline judgment of the video stream and the failure to start the task both adopt a multiple-attempt failure alarm mechanism to prevent infinite resource calls.

[0027] Preferably, a coroutine and multi-thread cooperation mechanism is adopted inside the process, and key operations are executed non-blockingly through an event loop.

[0028] In a second aspect, a method for large-scale video access and AI inference enhancement of the present invention is used to implement video access and AI inference through a system for large-scale video access and AI inference enhancement as described in any item of the first aspect. For the accessed video stream, the following operations are performed:

[0029] Resource dynamic scheduling: By executing CPU scheduling, the processes related to video tasks are bound to the CPU core with the lowest utilization rate. By executing GPU scheduling, the hardware units of the GPU are isolated. Among them, the processes related to video tasks include video processes, inference processes, and post-processing processes. The video process is used to execute decoding tasks. The inference process is used to call an AI large model to perform inference tasks on the decoded images. The post-processing process is used to execute post-processing operation logic and return alarm information to the video process. Correspondingly, the video process is used to perform frame marking, encoding, and streaming tasks based on the alarm information.

[0030] Decoding and inference control: Dynamically adjust the streaming flag of the video task and store the streaming flag in Redis. The inference flag is used to represent the streaming state and decoding frame rate of the video related to the video task. Among them, when the inference state is to start the streaming process, the corresponding decoding frame rate is full-frame decoding. When the streaming state is to stop the streaming process, the corresponding decoding frame rate is partial-frame decoding.

[0031] Resolution dynamic processing: Dynamically adjust the resolution of the decoded image, and the decoded image after resolution adjustment is stored in the cache queue.

[0032] Multi-level shared memory transmission: Realize data interaction between different processes through a multi-level shared memory queue, and dynamically adjust the capacity of the shared memory queue.

[0033] Exception self-healing and fault tolerance: Perform dual exception detection based on the decoded image and the inference result returned by the AI model, perform exception detection on the processes related to the video task, and perform unified scheduling and self-healing based on predefined rules.

[0034] Task parallel execution: The processes related to the video task are executed in parallel, and multiple threads in the process are executed in parallel.

[0035] The system and method for large-scale video access and enhanced AI inference of the present invention have the following advantages:

[0036] 1. The resource utilization rate is significantly improved, and the server CPU utilization rate is greatly reduced, especially for multi-core CPUs with low main frequencies that support hyper-threading;

[0037] 2. The transmission efficiency and computing energy efficiency are significantly improved. The encoding and decoding of pictures are removed by transmitting through shared memory between processes, and the computing amount is reduced by adopting dynamic resolution;

[0038] 3. Resource allocation on demand, non-full-frame rate decoding plus inference elastic control, reducing the hardware cost;

[0039] 4. The system reliability is greatly enhanced. The abnormal self-healing significantly improves the task survival rate, and the process-level fault tolerance automatically reclaims sub-processes to avoid the existence of zombie processes all the time;

[0040] 5. Break through the performance bottleneck of the traditional scheme and increase the number of video inference channels of a single server. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0042] The present invention will be further described below with reference to the drawings.

[0043] Figure 1 It is a flowchart of the working process of a system for large-scale video access and enhanced AI inference in Embodiment 1;

[0044] Figure 2 It is a timing diagram of dual-end detection abnormal recovery in a system for large-scale video access and enhanced AI inference in Embodiment 1. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] The following will further illustrate the present invention with reference to the drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it, but the embodiments cited are not intended to limit the present invention. Without conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0046] The embodiments of the present invention provide a system and method for large-scale video access and enhanced AI inference, which are used to solve the technical problem of how to achieve efficient access of large-scale video streams, elastic resource scheduling and abnormal self-healing in a complex environment.

[0047] Embodiment 1:

[0048] A system for enhancing large-scale video access and AI inference, including a resource dynamic scheduling module, a decoding and inference control module, a resolution dynamic processing module, a multi-level shared memory transmission module, an exception self-healing and fault tolerance module, and a task parallel execution module.

[0049] Resource dynamic scheduling module: Used to bind video task-related processes to the CPU core with the lowest utilization rate through CPU scheduling, and isolate the hardware units of the GPU through GPU scheduling. Among them, video task-related processes include video processes, inference processes, and post-processing processes. The video process is used to execute decoding tasks, the inference process is used to call the AI large model to perform inference tasks on the decoded image, and the post-processing process is used to execute the post-processing operation logic and return alarm information to the video process. Correspondingly, the video process is used to perform frame marking, encoding, and streaming tasks based on the alarm information.

[0050] As a specific implementation of the resource dynamic scheduling module, this module is used to perform the following operations:

[0051] (1) CPU scheduling: Calculate the utilization rate of each CPU core in real time through psutil.cpu_percen. For each video task, bind the processes related to the video task to the CPU core with the lowest utilization rate, and isolate and allocate the logical cores of the same physical core to the processes of different video tasks to avoid resource competition;

[0052] (2) GPU scheduling: Bind the decoding task and encoding task corresponding to the video task to the NVENC / NVDEC hardware unit of the GPU card, bind the inference task to the CUDA Core of the GPU card, and collect the decoding utilization rate, encoding utilization rate, and inference utilization rate of each GPU card in real time, and allocate the decoding task, encoding task, and inference task to the GPU card with the lowest utilization rate.

[0053] In this embodiment, the CPU core is dynamically specified through CPU scheduling. Each time, the core with the lowest CPU utilization rate is obtained as the allocation object, and the relevant processes (the three processes of the video process, the inference process, and the post-processing process) are allocated to this core, reducing frequent context switching, improving resource sharing, and reducing cpu resource competition. Especially when the CPU main frequency is low and hyper-threading is supported, the effect is particularly obvious. In the traditional way of allocating the CPU by the operating system, when the number of video channels and the number of processes increase, the utilization rate of a single-core CPU will soar due to resource contention, and the overall efficiency is extremely low.

[0054] In this embodiment, GPU scheduling is used to achieve hardware unit isolation within the GPU card. The decoding / encoding tasks are bound to the NVENC / NVDEC hardware units of the GPU, and the inference tasks exclusively occupy the CUDA Cores to avoid hardware resource conflicts. Moreover, this embodiment provides a multi-GPU load balancing strategy, which collects the decoding utilization rate, encoding utilization rate, and inference utilization rate of each GPU card in real time, and distributes the decoding, encoding, and inference tasks to the GPU card with the lowest utilization rate.

[0055] Decoding and Inference Control Module: It is used to dynamically adjust the streaming flag of the video task and store the streaming flag in Redis. The inference flag is used to represent the streaming status of the video related to the video task and the decoding frame rate. Among them, when the inference state is to start the streaming process, the corresponding decoding frame rate is full-frame decoding; when the streaming state is to stop the streaming process, the corresponding decoding frame rate is partial-frame decoding.

[0056] In this embodiment, the decoding and inference control module dynamically starts and stops the streaming process according to the large-screen display requirements. For the video streams to be displayed on the large screen, the corresponding streaming processes are started. Using Redis storage, the string prefix + "video task ID" is used as the streaming flag. The background periodically monitors the streaming flag. When the defined value is 1, streaming is performed, and the streaming process is started at this time. When the value is 0, no streaming is performed, and at this time, it is judged whether the streaming process exists. If it exists, the streaming process is killed to achieve dynamic switching of whether to stream.

[0057] During video decoding, it is determined whether to perform full-frame decoding according to whether to stream. When the streaming flag of the video task is to start streaming, in order to ensure the smooth stability of the output video stream, full-frame decoding is required at this time. If there is no streaming, partial decoding is performed, which is only used for inference. For a video with a frame rate of 25, it is default to take 5 frames per second. This is 80% less than full-frame decoding.

[0058] Resolution Dynamic Processing Module: It is used to dynamically adjust the resolution of the decoded image, and the decoded image after resolution adjustment is stored in the cache queue.

[0059] Among them, there are two cache queues, namely the long cache queue and the short cache queue. For the video task with the streaming flag to start the streaming process, the corresponding decoded image after resolution adjustment is stored in the long cache queue. For the video task with the streaming flag to stop the streaming process, the corresponding decoded image after resolution adjustment is stored in the short cache queue.

[0060] In this embodiment, for the high-precision algorithm scenario, the resolution dynamic processing module is used to maintain the original resolution of the decoded image. For the low-precision algorithm scenario, the resolution dynamic processing module is used to perform scaling on the decoded image through the FFmpeg GPU filter chain. The mapping relationship between the scaling ratio and the algorithm type during scaling is stored in Redis.

[0061] The resolution dynamic processing module determines the resolution of the decoded image according to the requirements of the algorithm scenario. For high-precision requirements, it maintains the original image resolution, and for low-precision requirements, it scales the decoded image proportionally, reducing the subsequent calculation of the entire process image.

[0062] In this embodiment, two image buffer queues are defined, a long queue and a short queue. When push streaming is enabled, the long queue is used, and when push streaming is not enabled, the short queue is used. Using the long queue can increase the image buffer time to ensure the annotation of the inference result. For most video tasks without push streaming, the short queue is used, and this mechanism further reduces the memory usage.

[0063] Multi-level shared memory transfer module: Used to achieve data interaction between different processes through a multi-level shared memory queue and dynamically adjust the capacity of the shared memory queue.

[0064] In this embodiment, the multi-level shared memory transfer module is used to store data between different processes through a two-level shared memory queue. Among them, the first-level shared memory queue is used to achieve data interaction between the video process and the inference process. The interactive data includes the decoded image obtained through the video process and the image index-related information. The decoded image is directly stored in the shared memory queue in the form of a NUMPY array, and the image index-related information is stored in the information ordinary queue; in the inference process, a producer thread is started. The producer thread obtains the image index information and the image array from the information ordinary queue and the shared memory queue respectively, and calls the AI model to perform inference tasks to obtain image inference information; the second-level shared memory queue is used to achieve data interaction between the inference process and the post-processing process. The interactive data includes the image array and the image inference information. After receiving the image array and the image queue information, the post-processing process performs post-processing logic operations, structured assembly of data, and landing of defective images based on the application scenario requirements, and returns the image coordinate exception information to the video process. The video process is used to perform frame marking processing based on the image coordinate exception information, where the image coordinate exception information is used as an alarm message.

[0065] In this embodiment, the shared memory queue implements a first-in, first-out queue based on multiprocessing and file locks.

[0066] Data interaction between processes is transmitted through the shared memory queue. The shared memory queue is a first-in, first-out queue implemented based on multiprocessing plus file locks. For the interaction between processes, the original NUMPY data format after image decoding is put into the shared memory queue, removing the encoding and decoding in the image transmission process, which greatly reduces the CPU utilization rate. The size of the shared memory queue will be automatically adjusted according to the current resolution of the decoded image, reducing the memory usage.

[0067] Abnormal Self-healing and Fault Tolerance Module: It is used to perform dual abnormal detection based on the decoded image and the inference result returned by the AI model, detect abnormalities in the processes related to video tasks, and perform unified scheduling and self-healing based on predefined rules.

[0068] As a specific implementation of the abnormal self-healing and fault tolerance module, this module is used to perform the following operations:

[0069] (1) Dual Abnormal Detection: Continuously monitor the status of each decoded image obtained by decoding the video stream. If the decoded images of N consecutive frames are empty frames or the acquisition of the decoded image fails, automatically trigger a video task stop instruction, release relevant resources, synchronously set the inference result timeout threshold, and continuously monitor the inference result returned by the AI model. If the time to receive the inference result exceeds the inference result timeout threshold, terminate the current inference task and roll back the processing queue;

[0070] (2) Process-level Heartbeat Monitoring: Start a monitoring thread in the inference process. There is an uninterrupted heartbeat flag maintained in the monitoring thread, and the heartbeat flag is stored in Redis. Determine whether the current monitoring thread is performing the inference task through the heartbeat flag;

[0071] (3) Parent-Child Process Monitoring: The video process is the parent process of the inference process, the inference process is the parent process of the post-processing process, and the post-processing process is the parent process of the video process. Regularly monitor whether the parent process of the video process, inference process, and post-processing process exists to prevent the inability to recycle this process due to the hanging of its parent process;

[0072] (4) Unified Scheduling and Self-healing: After detecting that the inference process has stopped, determine whether the current video stream is online. If it is online, start the inference task. If it is offline, continue to determine whether to start the inference task after the configured interval. Among them, the online / offline judgment of the video stream and the failure to start the task both adopt a mechanism of multiple attempt failures and alarms to prevent infinite resource calls.

[0073] Task Parallel Execution Module: It is used to support the parallel execution of processes related to video tasks and support the parallel execution of multiple threads in the process.

[0074] In this embodiment, the task parallel execution realizes multi-level asynchronous processing.

[0075] (1) Process-level Parallelism: The video decoding process, AI inference process, and post-processing process run independently and interact data through a shared memory queue;

[0076] (2) In-process asynchronous scheduling: Inside each process, a coroutine and multi-thread cooperation mechanism is adopted, and key operations are executed non-blockingly through an event loop; Video decoding process: Asynchronously execute streaming media pulling, GPU decoding, image framing, and pushing; AI inference process: Multithreaded (producer, consumer) parallel processing for model inference and result feedback; Post-processing process: Coroutine scheduling for alarm generation and abnormal image landing tasks.

[0077] The system of this embodiment, as Figure 1 shown, is divided into a control layer, an execution layer, a transmission layer, and a scheduling layer in terms of architecture. In the control layer, control is carried out through a resource dynamic scheduling, decoding and pushing notification module, and an exception self-healing and fault tolerance module. In the execution layer, a video process, an inference process, and a post-processing process are executed. In the transmission layer, interactive transmission between different processes is achieved through a shared memory transmission module. In the scheduling layer, resource dynamic scheduling is used to schedule resources for the CPU and GPU.

[0078] The system of this embodiment provides a pushing - frame rate - resolution - queue linkage mechanism, and the specific operations are as follows:

[0079] (1) Judgment of the triggering condition of the pushing flag: When "task ID + pushing flag" in Redis = 1, full frame rate decoding (≥25fps), original resolution processing, and long queue caching (capacity ≥ 60 frames) are executed. When the pushing flag = 0, partial frame rate decoding (≤5fps), resolution equal ratio compression (scaling ratio ≤ 50%), and short queue caching (capacity ≤ 10 frames) are executed;

[0080] (2) Resource linkage control: When the pushing flag changes, the frame rate, resolution, and queue capacity parameters are synchronously updated, and the parameter update event is broadcast through the Redis Pub / Sub channel to dynamically start and stop the pushing process. When the flag = 1, the pushing is started, and when the flag = 0, the pushing is terminated;

[0081] Based on the above operations, a large amount of invalid data transmission and data calculation are reduced, solving the pain point of resource waste.

[0082] The system of this embodiment isolates the CPU core and GPU hardware unit through dynamic binding, optimizing the allocation of computing resources; Dynamically adjusts the resolution based on the accuracy requirements of the scenario algorithm, reducing invalid calculations; Controls the pushing and frame rate decoding strategies using Redis flags, and combines long and short queues to reduce memory occupancy; Adopts shared memory NUMPY data direct storage, reducing transmission encoding and decoding, and improving the cross-process interaction efficiency; Designs dual exception detection and process-level heartbeat monitoring to achieve task self-healing; Builds an asynchronous parallel architecture based on Python Async to maximize the server throughput.

[0083] Embodiment 2:

[0084] A method for enhancing large-scale video access and AI inference. This method realizes video access and AI inference through the system provided in Embodiment 1. For the accessed video stream, six steps are executed: resource dynamic scheduling, decoding and inference control, resolution dynamic processing, multi-level shared memory transmission, exception self-healing and fault tolerance, and task parallel execution.

[0085] Step S100: Resource dynamic scheduling. Bind the processes related to video tasks to the CPU core with the lowest utilization rate through CPU scheduling, and isolate the hardware units of the GPU through GPU scheduling. Among them, the processes related to video tasks include video processes, inference processes, and post-processing processes. The video process is used to execute decoding tasks, the inference process is used to call the AI large model to perform inference tasks on the decoded image, and the post-processing process is used to execute the post-processing operation logic and return warning information to the video process. Correspondingly, the video process is used to perform bounding box marking, encoding, and streaming tasks based on the warning information.

[0086] In this embodiment, the resource dynamic scheduling includes the following two operations:

[0087] (1) CPU scheduling: Calculate the utilization rate of each CPU core in real time through psutil.cpu_percent. For each video task, bind the processes related to the video task to the CPU core with the lowest utilization rate, and isolate and allocate the logical cores of the same physical core to the processes of different video tasks to avoid resource competition.

[0088] (2) GPU scheduling: Bind the decoding task and encoding task corresponding to the video task to the NVENC / NVDEC hardware unit of the GPU card, bind the inference task to the CUDA Core of the GPU card, and collect the decoding utilization rate, encoding utilization rate, and inference utilization rate of each GPU card in real time, and allocate the decoding task, encoding task, and inference task to the GPU card with the lowest utilization rate.

[0089] In this embodiment, the CPU core is dynamically specified through CPU scheduling. Each time, the core with the lowest CPU utilization rate is obtained as the allocation object, and the relevant processes (the three processes of video process, inference process, and post-processing process) are allocated to this core, reducing frequent context switching, improving resource sharing, and reducing CPU resource competition. Especially when the CPU main frequency is low and hyper-threading is supported, the effect is particularly obvious. In the traditional way of the operating system allocating the CPU, when the number of video channels increases and the number of processes increases, the utilization rate of a single-core CPU will soar due to resource contention, and the overall efficiency is extremely low.

[0090] In this embodiment, hardware unit isolation within the GPU card is achieved through GPU scheduling. The decoding / encoding tasks are bound to the NVENC / NVDEC hardware units of the GPU, and the inference tasks exclusively occupy the CUDA Cores to avoid hardware resource conflicts. Moreover, this embodiment provides a multi-GPU load balancing strategy, which collects the decoding utilization rate, encoding utilization rate, and inference utilization rate of each GPU card in real time, and distributes the decoding, encoding, and inference tasks to the GPU card with the lowest utilization rate.

[0091] Step S200 Decoding and Inference Control: Dynamically adjust the streaming flag of the video task and store the streaming flag in Redis. The inference flag is used to represent the streaming status and decoding frame rate of the video related to the video task. Among them, when the inference status is to start the streaming process, the corresponding decoding frame rate is full-frame decoding; when the streaming status is to stop the streaming process, the corresponding decoding frame rate is partial-frame decoding.

[0092] In this embodiment, the streaming process is dynamically started and stopped according to the large-screen display requirements. For the video streams to be displayed on the large screen, the corresponding streaming processes are started. Using Redis storage, the string prefix + "video task ID" is used as the streaming flag. The background periodically monitors the streaming flag. When the defined value is 1, streaming is started, and at this time, the streaming process is started for streaming; when the value is 0, streaming is not performed, and at this time, it is judged whether the streaming process exists. If it exists, the streaming process is killed to achieve dynamic switching of whether to stream.

[0093] During video decoding, it is determined whether to perform full-frame decoding according to whether to stream. When the streaming flag of the video task is to start streaming, in order to ensure the smoothness and stability of the output video stream, full-frame decoding is required at this time. If not streaming, partial decoding is performed, which is only used for inference. For a video with a frame rate of 25, it is default to take 5 frames per second. This is 80% less than full-frame decoding.

[0094] Step S300 Dynamic Resolution Processing: Dynamically adjust the resolution of the decoded image, and the decoded image after resolution adjustment is stored in the cache queue.

[0095] Among them, there are two cache queues, namely the long cache queue and the short cache queue. For the video task with the streaming flag to start the streaming process, the decoded image after the corresponding resolution adjustment is stored in the long cache queue; for the video task with the streaming flag to stop the streaming process, the decoded image after the corresponding resolution adjustment is stored in the short cache queue.

[0096] In this embodiment, for the high-precision algorithm scenario, the original resolution of the decoded image is maintained. For the low-precision algorithm scenario, the decoded image is scaled through the FFmpeg GPU filter chain. The mapping relationship between the scaling ratio and the algorithm type during scaling is stored in Redis. The resolution of the decoded image is determined according to the algorithm scenario requirements. For high-precision requirements, the original image resolution is maintained, and for low-precision requirements, the decoded image is scaled proportionally, reducing the subsequent calculation of the entire process image.

[0097] In this embodiment, two picture cache queues are defined, a long queue and a short queue. When push streaming is enabled, the long queue is used, and when push streaming is not enabled, the short queue is used. Using the long queue can increase the picture cache time to ensure the annotation of the inference result. For most video tasks without push streaming, the short queue is used, and this mechanism further reduces the memory usage.

[0098] Step S400 multi-level shared memory transfer: Data interaction between different processes is achieved through a multi-level shared memory queue, and the capacity of the shared memory queue is dynamically adjusted.

[0099] In this embodiment, data between different processes is stored through a two-level shared memory queue. Among them, the first-level shared memory queue is used to achieve data interaction between the video process and the inference process. The data for interaction includes the decoded image obtained through the video process and image index-related information. The decoded image is directly stored in the shared memory queue in the format of a NUMPY array, and the image index-related information is stored in an ordinary information queue. In the inference process, a producer thread is started. The producer thread obtains the image index information and the image array from the ordinary information queue and the shared memory queue respectively, and calls the AI model to perform inference tasks to obtain image inference information. The second-level shared memory queue is used to achieve data interaction between the inference process and the post-processing process. The data for interaction includes the image array and the image inference information. After receiving the image array and the image queue information, the post-processing process performs post-processing logic operations, structured assembly of data, and landing of defective images based on the application scenario requirements, and returns image coordinate exception information to the video process. The video process is used to perform frame marking processing based on the image coordinate exception information, where the image coordinate exception information is used as an alarm message.

[0100] In this embodiment, the shared memory queue implements a first-in, first-out queue based on multiprocessing and file locks.

[0101] Data interaction between processes is transmitted through a shared memory queue, which is a first-in-first-out queue implemented based on multiprocessing plus file locks. For the interaction between processes, the original NUMPY data format after image decoding is put into the shared memory queue, eliminating encoding and decoding during image transmission and greatly reducing CPU utilization. The size of the shared memory queue is automatically adjusted according to the current decoded image resolution size, reducing memory usage.

[0102] Step S500 Exception Self-Healing and Fault Tolerance: Double exception detection is performed based on the decoded image and the inference result returned by the AI model. Exception detection is carried out on processes related to video tasks, and unified scheduling and self-healing are performed based on predefined rules.

[0103] As a specific implementation of exception self-healing and fault tolerance, it includes the following operations:

[0104] (1) Double exception detection: Continuously monitor the status of each decoded image obtained by decoding the video stream. If the decoded image is an empty frame for N consecutive frames or the acquisition of the decoded image fails, automatically trigger a video task stop instruction, release relevant resources, synchronously set the inference result timeout threshold, and continuously monitor the inference result returned by the AI model. If the time to receive the inference result exceeds the inference result timeout threshold, terminate the current inference task and roll back the processing queue;

[0105] (2) Process-level heartbeat monitoring: Start a monitoring thread in the inference process. There is an uninterrupted heartbeat flag maintained in the monitoring thread, and the heartbeat flag is stored in Redis. Determine whether the current monitoring thread is executing an inference task based on the heartbeat flag;

[0106] (3) Parent-child process monitoring: The video process is the parent process of the inference process, the inference process is the parent process of the post-processing process, and the post-processing process is the parent process of the video process. For the video process, inference process, and post-processing process, regularly monitor whether their parent processes exist to prevent the inability to recycle this process due to the hanging of their parent processes;

[0107] (4) Unified scheduling and self-healing: After detecting that the inference process has stopped, determine whether the current video stream is online. If it is online, start the inference task. If it is offline, continue to determine whether to start the inference task after a configured interval. For the online / offline determination of the video stream and the failure to start the task, a mechanism of multiple attempts and failure alarms is adopted to prevent infinite resource calls.

[0108] Step S600 Task Parallel Execution: Processes related to video tasks are executed in parallel, and multiple threads in the process are executed in parallel.

[0109] In this embodiment, task parallel execution realizes multi-level asynchronous processing.

[0110] (1) Process-level parallelism: The video decoding process, AI inference process, and post-processing process run independently and interact with each other through a shared memory queue to exchange data;

[0111] (2) Intra-process asynchronous scheduling: Each process adopts a cooperative mechanism of coroutines and multi-threading internally, and non-blockingly executes key operations through an event loop; Video decoding process: Asynchronously execute streaming media pulling, GPU decoding, image framing, and streaming; AI inference process: Multithreading (producer, consumer) parallel processing model inference and result feedback; Post-processing process: Coroutine scheduling for alarm generation and abnormal image landing tasks.

[0112] The above has introduced in detail the system and method for large-scale video access and enhanced AI inference provided by the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A system for enhancing large-scale video access and AI inference, characterized in that, Including: Resource dynamic scheduling module: used to bind video task-related processes to the CPU core with the lowest utilization rate through CPU scheduling, and isolate the hardware units of the GPU through GPU scheduling. Among them, video task-related processes include video processes, inference processes, and post-processing processes. The video process is used to execute decoding tasks, the inference process is used to call the AI large model to perform inference tasks on the decoded image, and the post-processing process is used to execute post-processing operation logic and return alarm information to the video process. Correspondingly, the video process is used to perform frame marking, encoding, and streaming tasks based on the alarm information; Decoding and inference control module: used to dynamically adjust the streaming flag of the video task and store the streaming flag in Redis. The inference flag is used to represent the streaming status and decoding frame rate of the video related to the video task. When the inference status is to start the streaming process, the corresponding decoding frame rate is full-frame decoding. When the streaming status is to stop the streaming process, the corresponding decoding frame rate is partial-frame decoding; Resolution dynamic processing module: used to dynamically adjust the resolution of the decoded image, and the decoded image after resolution adjustment is stored in the cache queue; Multi-level shared memory transfer module: used to achieve data interaction between different processes through a multi-level shared memory queue, and dynamically adjust the capacity of the shared memory queue; Exception self-healing and fault tolerance module: used to perform double exception detection based on the decoded image and the inference result returned by the AI model, detect exceptions for processes related to video tasks, and perform unified scheduling and self-healing based on predefined rules; Task parallel execution module: used to support the parallel execution of processes related to video tasks, and support the parallel execution of multiple threads in the process.

2. The system for enhancing large-scale video access and AI inference according to claim 1, wherein The resource dynamic scheduling module is used to perform the following: CPU scheduling: Calculate the utilization rate of each CPU core in real time through psutil.cpu_percen. For each video task, bind the processes related to the video task to the CPU core with the lowest utilization rate, and isolate and allocate the logical cores of the same physical core to the processes of different video tasks to avoid resource competition; GPU scheduling: Bind the decoding task and encoding task corresponding to the video task to the NVENC / NVDEC hardware unit of the GPU card, bind the inference task to the CUDA Core of the GPU card, and collect the decoding utilization rate, encoding utilization rate, and inference utilization rate of each GPU card in real time, and allocate the decoding task, encoding task, and inference task to the GPU card with the lowest utilization rate.

3. The system for enhancing large-scale video access and AI inference according to claim 1, wherein For high-precision algorithm scenarios, the resolution dynamic processing module is used to maintain the original resolution of the decoded image. For low-precision algorithm scenarios, the resolution dynamic processing module is used to perform scaling on the decoded image through the FFmpeg GPU filter chain. The mapping relationship between the scaling ratio and the algorithm type during scaling is stored in Redis.

4. The system for enhancing large-scale video access and AI inference according to claim 1, wherein There are two cache queues in total, namely the long cache queue and the short cache queue. For video tasks with the push stream flag indicating the start of the push stream process, the decoded images after resolution adjustment are stored in the long cache queue. For video tasks with the push stream flag indicating the stop of the push stream process, the decoded images after resolution adjustment are stored in the short cache queue.

5. The system for large-scale video access and AI inference enhancement according to claim 1, wherein The multi-level shared memory transfer module is used to store data between different processes through two-level shared memory queues; Among them, the first-level shared memory queue is used to realize data interaction between the video process and the inference process. The interactive data includes the decoded images obtained through the video process and image index-related information. The decoded images are directly stored in the shared memory queue in the format of a NUMPY array, and the image index-related information is stored in the information normal queue; A producer thread is started in the inference process. The producer thread obtains the image index information and the image array from the information normal queue and the shared memory queue respectively, and calls the AI model to perform the inference task to obtain the image inference information; The second-level shared memory queue is used to realize data interaction between the inference process and the post-processing process. The interactive data includes the image array and the image inference information. After receiving the image array and the image queue information, the post-processing process performs post-processing logic operations, data structured assembly, and landing of defective images based on the application scenario requirements, and returns the image coordinate exception information to the video process. The video process is used to perform frame marking processing based on the image coordinate exception information, where the image coordinate exception information is used as an alarm message.

6. The system for large-scale video access and enhanced AI inference according to claim 5, characterized in that, The shared memory queue is implemented as a first-in-first-out queue based on multiprocessing and file locks.

7. The system for large-scale video access and AI inference enhancement according to claim 1, wherein The exception self-healing and fault tolerance module is used to perform the following operations: Dual exception detection: Real-time monitor the state of each decoded image obtained by decoding the video stream. If N consecutive decoded images are empty frames or the acquisition of the decoded image fails, automatically trigger a video task stop instruction, release relevant resources, synchronously set the inference result timeout threshold, and real-time monitor the push stream result returned by the AI model. If the time to receive the inference result exceeds the inference result timeout threshold, terminate the current inference task and roll back the processing queue; Process-level heartbeat monitoring: A monitoring thread is started in the inference process. There is an uninterrupted heartbeat flag maintained in the monitoring thread. The heartbeat flag is stored in Redis, and it is used to judge whether the current monitoring thread is performing the inference task through the heartbeat flag; Parent-child process monitoring: The video process is the parent process of the inference process, the inference process is the parent process of the post-processing process, and the post-processing process is the parent process of the video process. For the video process, the inference process, and the post-processing process, regularly monitor whether their parent processes exist to prevent the inability to recycle this process due to the hanging of their parent processes; Unified scheduling self-healing: When it is detected that the inference process has stopped, determine whether the current video stream is online. If it is online, start the inference task. If it is offline, continue to determine whether to start the inference task after the configured interval. Among them, the online / offline judgment of the video stream and the failure to start the task both adopt a mechanism of multiple attempt failure alarms to prevent infinite resource calls.

8. The system for enhancing large-scale video access and AI inference according to claim 1, wherein Inside the process, a coroutine and multi-thread cooperation mechanism is adopted, and key operations are executed non-blockingly through an event loop.

9. A method for enhancing large-scale video access and AI inference, characterized in that, Used to implement video access and AI inference through a system for large-scale video access and AI inference enhancement as described in any one of claims 1-8. For the accessed video stream, the following operations are performed: Resource dynamic scheduling: Bind the video task-related processes to the CPU core with the lowest utilization rate by executing CPU scheduling, and isolate the hardware units of the GPU by executing GPU scheduling. Among them, the video task-related processes include a video process, an inference process, and a post-processing process. The video process is used to execute the decoding task, the inference process is used to call the AI large model to perform inference tasks on the decoded image, and the post-processing process is used to execute the post-processing operation logic and return warning information to the video process. Correspondingly, the video process is used to perform frame marking, encoding, and streaming tasks based on the warning information; Decoding and inference control: Dynamically adjust the streaming flag of the video task and store the streaming flag in Redis. The inference flag is used to represent the streaming state and decoding frame rate of the video related to the video task. Among them, when the inference state is to start the streaming process, the corresponding decoding frame rate is full-frame decoding. When the streaming state is to stop the streaming process, the corresponding decoding frame rate is partial-frame decoding; Dynamic resolution processing: Dynamically adjust the resolution of the decoded image, and the decoded image after resolution adjustment is stored in the cache queue; Multi-level shared memory transmission: Realize data interaction between different processes through a multi-level shared memory queue, and dynamically adjust the capacity of the shared memory queue; Exception self-healing and fault tolerance: Perform double exception detection based on the decoded image and the inference result returned by the AI model, perform exception detection on the processes related to the video task, and perform unified scheduling self-healing based on predefined rules; Task parallel execution: The processes related to the video task are executed in parallel, and multiple threads in the process are executed in parallel.

Images