Multi-end deployment based ai video analysis inference acceleration system

Abstract

The application provides an AI video analysis inference acceleration system based on multi-terminal deployment, and relates to the technical field of video analysis. The system comprises a virtual device pool, an AI computing power center, a hybrid programming acceleration module and an edge device agent rendering module. The hybrid programming acceleration module generates video data calling information according to the received analysis inference task information and sends it to the AI computing power center. The AI computing power center generates video data calling instructions according to the video data calling information, and the virtual device pool obtains physical video data and performs preprocessing according to the instructions to obtain standardized preliminary video data. The hybrid programming acceleration module performs hybrid programming analysis inference on the standardized preliminary video data to obtain analysis inference results. Finally, the edge device agent rendering module renders the standardized preliminary video data according to the analysis inference results to obtain target video data. Through the above technical scheme, the system can improve the compatibility and concurrency of video analysis.

Description

Technical Field

[0001] This application relates to the field of video analytics technology, specifically to an AI video analytics inference acceleration system based on multi-terminal deployment. Background Technology

[0002] Traditional AI video analytics systems typically rely on vendor-provided interfaces, such as the standard nine-grid layout. While this layout is simple and easy to use, it has limitations in practical operation. For example, in terms of monitoring range, the nine-grid layout only provides a limited field of view, lacking a comprehensive grasp of macroscopic information such as building complexes and road networks. Furthermore, traditional monitoring interfaces lack the ability to automatically focus on and independently detect critical events. Therefore, in emergency situations, operators may have to manually examine each frame to find the source of the problem, which obviously prolongs response time.

[0003] To address the limitations of traditional AI video analytics systems, particularly in terms of monitoring range and event response speed, advanced video analytics algorithms are widely employed in current technologies. These algorithms, utilizing deep learning and computer vision, can automatically analyze key information in surveillance videos and perform real-time event detection and early warning. However, while these algorithms have improved the intelligence level of surveillance systems to some extent, several problems remain in practical applications. First, differences in compatibility and standardization among devices from different manufacturers pose challenges to algorithm deployment and integration. Second, with the increasing demands for monitoring, concurrent processing capability has become a critical issue. Existing video analytics systems often struggle to process large volumes of surveillance video streams simultaneously, potentially leading to processing delays or the omission of important information at critical moments. Summary of the Invention

[0004] This application provides an AI video analysis inference acceleration system based on multi-terminal deployment, which is used to improve the compatibility and concurrency of video analysis systems.

[0005] Firstly, this application provides an AI video analysis inference acceleration system based on multi-terminal deployment, the method of which includes:

[0006] The system includes a virtual device pool, an AI computing power center, a hybrid programming acceleration module, and an edge device proxy rendering module;

[0007] The hybrid programming acceleration module is used to generate video data retrieval information based on the received analysis and reasoning task information, and send the video data retrieval information to the AI ​​computing power center;

[0008] The AI ​​computing center is used to receive video data request information and generate video data request instructions based on the video data request information.

[0009] The virtual device pool is used to obtain physical video data according to video data call instructions, and to preprocess the physical video data to obtain standardized preliminary video data;

[0010] The hybrid programming acceleration module is used to perform hybrid programming analysis and inference on standardized preliminary video data to obtain analysis and inference results;

[0011] The edge device proxy rendering module is used to render standardized preliminary video data based on the analysis and inference results to obtain target video data.

[0012] By employing the above technical solution, the collaborative work between the hybrid programming acceleration module, the virtual device pool, the AI ​​computing power center, and the edge device proxy rendering module enables real-time video analysis and inference. Specifically, the hybrid programming acceleration module generates video data retrieval information based on the received analysis and inference task information and sends it to the AI ​​computing power center. The AI ​​computing power center generates video data retrieval instructions based on the video data retrieval information to retrieve physical video data from the virtual device pool. The virtual device pool retrieves the physical video data based on the video data retrieval instructions, performs preprocessing, and outputs standardized preliminary video data. After obtaining the standardized preliminary video data, the hybrid programming acceleration module performs hybrid programming analysis and inference to generate analysis and inference results. Finally, the edge device proxy rendering module renders the standardized preliminary video data based on the analysis and inference results to obtain the final target video data.

[0013] Through this collaborative, streamlined workflow, complex video analysis and inference tasks can be broken down into smaller, more manageable modules. This approach fully leverages the strengths of each module while enabling dynamic matching of task modules, significantly reducing the overall analysis and inference time and allowing the system to output results in real-time. Therefore, the coordinated operation of the hybrid programming acceleration module, virtual device pool, AI computing center, and edge device proxy rendering module is crucial for achieving real-time video analysis and inference.

[0014] Optionally, the virtual device pool can also be used to abstract physical video devices into virtual devices and register the virtual devices into the virtual device pool.

[0015] By adopting the above technical solution, the virtual device pool is also used to abstract physical video devices into virtual devices and register these virtual devices into the virtual device pool.

[0016] Through this abstract registration mechanism of the virtual device pool, standardized and unified management of a massive number of physical video devices from different manufacturers, models, and interfaces can be achieved. When retrieving video data, the system only needs to issue a unified call command to the virtual device pool, without needing to concern itself with the specific physical device details, greatly simplifying the call logic. At the same time, the virtual device pool abstracts the complex and diverse physical video devices into standardized virtual devices, providing a unified data input format for subsequent video data processing.

[0017] In this technical solution, the system can obtain standardized video data input by calling a virtual device pool, avoiding the coupling and adaptation issues of processing video data from different sources. This creates favorable conditions for fast and efficient video analysis and inference. It can be said that the abstract registration mechanism of the virtual device pool greatly facilitates interoperability between different physical video devices and is an important foundation for realizing real-time video analysis and inference. Integrating all video sources into a unified virtual device pool makes the processing flow smoother and more efficient, effectively shortening the analysis and inference time and enhancing the system's real-time response capabilities.

[0018] Optionally, the hybrid programming acceleration module includes a C++ unit, a data-compatible unit, and a Python unit;

[0019] The C++ unit is used to preprocess the initial video data to obtain the first video data;

[0020] A data compatibility unit is used to perform data structure conversion on the first video data to obtain the second video data;

[0021] The Python unit is used to analyze and reason about the second video data to obtain the analysis and reasoning results.

[0022] By adopting the above technical solution, the hybrid programming acceleration module consists of a C++ unit, a data compatibility unit, and a Python unit. The C++ unit preprocesses the initial video data and outputs the first video data; the data compatibility unit performs structural transformation on the first video data and outputs the second video data; the Python unit then performs inference analysis on the second video data to finally obtain the analysis and inference results.

[0023] This hybrid programming approach leverages the strengths of different programming languages, enabling rapid processing of video analysis tasks. The C++ unit utilizes the efficiency of C++ to preprocess video data, improving processing speed. The data compatibility unit achieves data flow between C++ and Python through structure conversion, overcoming the difficulties of data incompatibility between languages. The Python unit, on the other hand, leverages Python's advantages in machine learning for rapid analysis and reasoning.

[0024] As can be seen, the combined use of C++, data-compatible, and Python units in the hybrid programming acceleration module improves the system's parallel processing capabilities, enabling efficient pipelined processing of video analysis tasks. The close collaboration between these three units leverages both the speed of C++ and the analytical strengths of Python, perfectly combining computational performance and analytical capabilities. Therefore, this hybrid programming approach is crucial for ensuring real-time video analysis inference; it permeates and accelerates the entire video processing flow, enabling the rapid completion of complex video analysis tasks.

[0025] Optional, C++ units include: data decomposer, multi-way classifier, decoder, renderer, painter, encoder, wrapper, and integrator;

[0026] A data decomposer is used to decompose preliminary video data into video data packets and put the video data packets into a queue to obtain a video data packet queue.

[0027] A multi-channel classifier is used to extract video data packets from the video data packet queue and perform multi-channel separation of the video data packets according to the preset data type to obtain a video frame queue and an audio data queue.

[0028] The decoder is used to decode the video frame data in the video frame queue to obtain decoded video frame data, and convert the video frame data into pixel data that can be processed by image processing.

[0029] A renderer is used to process pixel data to obtain rendered video frames.

[0030] The renderer is used to finally draw the rendered video frames to obtain the completed video frames;

[0031] An encoder is used to re-encode the drawn video frames to obtain encoded video data packets;

[0032] A wrapper is used to encapsulate encoded video data packets to obtain a wrapped video data stream.

[0033] A wrapper is used to encapsulate audio data packets in an audio data queue to obtain a wrapped audio data stream.

[0034] An integrator is used to combine encapsulated audio data streams and encapsulated video data streams to obtain the first video data.

[0035] By adopting the above technical solution, the C++ unit includes a data decomposer, a multi-channel classifier, a decoder, a renderer, a painter, an encoder, a wrapper, and an integrator. It can preprocess video data and output the first video data. Specifically, the data decomposer breaks down the initial video data into video data packets and queues them; the multi-channel classifier extracts data packets from the queue, performs multi-channel separation, and obtains a video frame queue and an audio data queue; the decoder decodes and converts the video frame queue; the renderer and painter render and draw the video frames; the encoder re-encodes the drawn video frames; the wrapper encapsulates the video frames and audio data respectively; and finally, the integrator synthesizes the encapsulated audio and video streams to form the first video data.

[0036] As can be seen, the various components in the C++ unit perform refined decomposition and specialized processing of video data, achieving rapid decoding, rendering, encoding, and reassembly of the video data. It fully leverages the execution efficiency advantages of C++, significantly shortening video preprocessing time through pipelined parallel processing. Therefore, the collaborative work of the components in the C++ unit is key to the efficient video processing achieved by the hybrid programming acceleration module. Its fine-grained parallel processing of video data not only improves preprocessing speed but also lays the foundation for subsequent analysis. The design of the C++ unit is a crucial component of the system's real-time video analysis and inference capabilities.

[0037] Optional Python units include: preprocessing units and analysis units;

[0038] The preprocessing unit is used to preprocess the second video data to obtain preprocessed second video data;

[0039] The analysis and processing unit is used to input the preprocessed second video data into a preset analysis and reasoning model to obtain the analysis and reasoning results.

[0040] By employing the aforementioned techniques, the Python unit comprises a preprocessing unit and an analysis processing unit. The preprocessing unit preprocesses the second video data and outputs preprocessed video data; the analysis processing unit inputs the preprocessed video data into a pre-defined analysis and inference model to ultimately obtain the analysis and inference results.

[0041] As can be seen, the Python unit leverages Python's strengths in machine learning and artificial intelligence, specifically for video data analysis and processing. The preprocessing unit optimizes and enhances the video data, laying the foundation for subsequent analysis. The analysis and processing unit, on the other hand, directly utilizes Python's powerful AI analysis capabilities, performing rapid video inference analysis through pre-defined inference models.

[0042] The Python unit's design fully leverages Python's unique advantages in data processing and artificial intelligence analysis, enabling efficient inference for video analysis tasks. The collaboration between the preprocessing and analysis units forms a specialized processing flow for video analysis. Therefore, the application of the Python unit is key to achieving fast and accurate video analysis inference in the hybrid programming module. It works efficiently with the C++ unit, perfectly combining the computational performance of C++ with the analytical capabilities of Python, and is an important component ensuring the system's real-time video analysis capabilities.

[0043] Optionally, the edge device proxy rendering module includes computing power devices, streaming media servers, and central servers;

[0044] The central server is used to determine configuration information based on the analysis and reasoning results;

[0045] The computing power equipment is used to render the initial video data according to the configuration information. The configuration information of the rendered initial video data includes the algorithm UUID, camera IP, and ROI.

[0046] A streaming media server is used to convert the format of the rendered initial video data to obtain the target video data.

[0047] By adopting the above technical solution, the edge device proxy rendering module consists of a central server, computing power devices, and a streaming media server. The central server determines configuration information based on analysis and reasoning results; the computing power devices use this configuration information to render preliminary video data and output the rendered video data; the streaming media server performs format conversion on the rendered video data to ultimately form the target video data.

[0048] As can be seen, the edge device proxy rendering module adopts a distributed edge computing architecture. The central server coordinates task allocation and directs edge computing devices to perform rendering processing based on dynamic analysis results. This approach achieves logical decoupling between analysis and rendering, allowing the central server to flexibly allocate computing resources as needed. Simultaneously, edge devices process data locally, reducing data transmission and improving rendering efficiency. The streaming media server, on the other hand, standardizes the output results, helping to obtain directly usable target videos.

[0049] In summary, the design of the edge device proxy rendering module fully leverages the advantages of edge computing to achieve distributed collaboration in video processing. The cooperation between the central server, computing devices, and streaming media servers not only improves resource utilization efficiency but also accelerates the rendering process. The application of this module is crucial for the system to achieve flexible and scalable real-time video analysis and processing. It provides an efficient and reliable output method and is an important component ensuring a smooth overall processing flow.

[0050] Optionally, the streaming media server is also used to transmit the target video data to the user terminal.

[0051] By adopting the above technical solution, the streaming media server can also transmit the final generated target video data to the user terminal.

[0052] The streaming media server's transmission capabilities enable rapid feedback and use of video analysis results. Once the edge device completes the video rendering process, the streaming media server can directly send the rendered target video to the user terminal to meet their needs.

[0053] As can be seen, the streaming media server's transmission function provides a timely output channel for video analysis tasks. Target video data can be quickly transmitted back through the streaming media server without waiting for the entire processing flow to complete, greatly shortening the time from analysis to application and meeting real-time requirements.

[0054] In summary, the streaming media server's transmission function enables extremely rapid feedback of key video analysis results, a crucial means of ensuring the real-time response of the video analysis system. It achieves seamless integration between the video processing workflow and user requirements, making it a key component in achieving robust real-time performance. The streaming media server's design fully considers practical application needs, endowing complex video analysis systems with simple, real-time, and efficient user interaction characteristics.

[0055] Optionally, the system may also include: a load distribution module;

[0056] The load distribution module is used to classify analysis and inference tasks, obtain classification results, and push analysis and inference tasks to the hybrid programming acceleration module of the corresponding device based on the computing power information of each device and the classification results.

[0057] By adopting the above technical solution, the system also includes a load distribution module, which can classify video analysis and reasoning tasks and push the tasks to the corresponding hybrid programming acceleration module according to the device computing power information and task classification results.

[0058] The intelligent scheduling of the load distribution module enables the rational allocation of video analytics tasks, ensuring the full and efficient utilization of limited computing power. It scientifically categorizes tasks based on type and priority, then matches them to devices equipped with corresponding hardware for execution, avoiding inefficient random allocation. Simultaneously, this module can comprehensively consider the computing power of each device, rationally scheduling tasks to prevent overload.

[0059] As can be seen, the load balancing module achieves dynamic matching of tasks and resources, ensuring the video analytics system remains in an optimal working state. It makes full use of limited computing resources, enabling high-speed and efficient processing of complex video analytics tasks. Therefore, the application of the load balancing module is crucial for achieving large-scale real-time video analytics; it enables refined management of tasks and resources and is a vital support for ensuring overall processing efficiency.

[0060] Optionally, the hybrid programming acceleration module also includes: the SJF algorithm module;

[0061] The SJF algorithm module is used to determine the estimated execution time of the analytical reasoning task based on the analytical reasoning task information, and select the analytical reasoning task with the shortest estimated execution time as the next analytical reasoning task to be executed.

[0062] By adopting the above technical solution, the hybrid programming acceleration module also includes an SJF algorithm module, which can determine the estimated execution time based on the information of the analysis and reasoning task, and select the task with the shortest time for priority execution.

[0063] This shortest job first (SJF) scheduling strategy allows critical and simple video analytics tasks to be completed as quickly as possible. For analytical inference tasks with predictable execution times, the SJF algorithm can scientifically determine their processing order, thereby shortening the waiting time for important tasks. It can be seen that the application of the SJF algorithm module further optimizes the real-time performance and critical task responsiveness of video analytics. Building upon the load balancing module, it performs fine-grained task sorting and scheduling within its own module. The use of the SJF algorithm ensures that high-priority, short jobs can be executed immediately, significantly reducing the latency of critical event analysis and improving real-time performance.

[0064] In summary, the SJF algorithm module enhances the real-time response capability of the video analytics system to critical events by finely managing the execution order of analysis tasks. Working in conjunction with the load balancing module, it achieves multi-level real-time optimization within the system, serving as a crucial means to ensure low latency end-to-end. The design of the SJF algorithm module improves the processing efficiency of important video analytics tasks.

[0065] Optionally, the data compatibility unit is obtained in the following way:

[0066] The data structure of the first video data is converted into the data type in Python units using the preset ctypes, resulting in a data-compatible unit.

[0067] By adopting the above technical solution, the data compatibility unit is obtained by converting the C++ data structure of the first video data into a Python data type using preset ctypes.

[0068] ctypes acts as a bridge for data structure conversion, enabling seamless conversion between the initial video data output from the C++ unit and the input from the Python unit. Through ctype conversion, video data processed by different programming languages ​​can interact and be transferred, enabling hybrid programming and leveraging the advantages of both languages. It is evident that the structure conversion implemented by ctype is key to the speed and efficiency of the hybrid programming module. It eliminates the differences in data format and structure between C++ and Python, allowing them to collaborate closely, improving the parallel processing speed of video analysis, shortening processing time, and thus enhancing the system's real-time responsiveness.

[0069] In summary, the data structure conversion implemented by the data compatibility unit using ctype is fundamental to ensuring efficient execution of hybrid programming and is one of the key reasons why the Analysis system achieves good real-time performance. It resolves the issue of data mismatch between languages, making the system flow smoothly and thus significantly improving the speed of video analysis and inference.

[0070] In summary, one or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0071] 1. This application achieves real-time video analysis and inference through the collaborative work of a hybrid programming acceleration module, a virtual device pool, an AI computing power center, and an edge device proxy rendering module. Specifically, the hybrid programming acceleration module generates video data retrieval information based on the received analysis and inference task information and sends it to the AI ​​computing power center. The AI ​​computing power center generates video data retrieval instructions based on the video data retrieval information to retrieve physical video data from the virtual device pool. The virtual device pool retrieves the physical video data based on the video data retrieval instructions, performs preprocessing, and outputs standardized preliminary video data. After obtaining the standardized preliminary video data, the hybrid programming acceleration module performs hybrid programming analysis and inference to generate analysis and inference results. Finally, the edge device proxy rendering module renders the standardized preliminary video data based on the analysis and inference results to obtain the final target video data.

[0072] Through this collaborative, streamlined workflow, complex video analysis and inference tasks can be broken down into smaller, more manageable modules. This approach fully leverages the strengths of each module while enabling dynamic matching of task modules, significantly reducing the overall analysis and inference time and allowing the system to output results in real-time. Therefore, the coordinated operation of the hybrid programming acceleration module, virtual device pool, AI computing center, and edge device proxy rendering module is crucial for achieving real-time video analysis and inference.

[0073] 2. This application achieves real-time video analysis and inference through the collaborative work of a hybrid programming acceleration module, a virtual device pool, an AI computing power center, and an edge device proxy rendering module. Specifically, the hybrid programming acceleration module generates video data retrieval information based on the received analysis and inference task information and sends it to the AI ​​computing power center. The AI ​​computing power center generates video data retrieval instructions based on the video data retrieval information to retrieve physical video data from the virtual device pool. The virtual device pool retrieves the physical video data based on the video data retrieval instructions, performs preprocessing, and outputs standardized preliminary video data. After obtaining the standardized preliminary video data, the hybrid programming acceleration module performs hybrid programming analysis and inference to generate analysis and inference results. Finally, the edge device proxy rendering module renders the standardized preliminary video data based on the analysis and inference results to obtain the final target video data.

[0074] Through this collaborative, streamlined workflow, complex video analysis and inference tasks can be broken down into smaller, more manageable modules. This approach fully leverages the strengths of each module while enabling dynamic matching of task modules, significantly reducing the overall analysis and inference time and allowing the system to output results in real-time. Therefore, the coordinated operation of the hybrid programming acceleration module, virtual device pool, AI computing center, and edge device proxy rendering module is crucial for achieving real-time video analysis and inference.

[0075] 3. The C++ unit of this application includes a data decomposer, a multi-channel classifier, a decoder, a renderer, a painter, an encoder, a wrapper, and an integrator, which can preprocess video data and output the first video data. Specifically, the data decomposer breaks down the initial video data into video data packets and queues them; the multi-channel classifier extracts data packets from the queue, performs multi-channel separation, and obtains a video frame queue and an audio data queue; the decoder decodes and converts the video frame queue; the renderer and painter render and draw the video frames; the encoder re-encodes the drawn video frames; the wrapper encapsulates the video frames and audio data separately; and finally, the integrator synthesizes the encapsulated audio and video streams to form the first video data. It can be seen that the various components in the C++ unit perform refined decomposition and specialized processing of video data, achieving rapid decoding, rendering, encoding, and reassembly of video data. It fully leverages the execution efficiency advantages of C++, greatly shortening the video preprocessing time through pipelined parallel processing. Therefore, the collaborative work of the components in the C++ unit is key to achieving efficient video processing in the hybrid programming acceleration module. It performs fine-grained parallel processing on video data, which not only improves preprocessing speed but also lays the foundation for subsequent analysis. The design of the C++ unit is an important component of the system's real-time video analysis and inference implementation. Attached Figure Description

[0076] Figure 1 A schematic diagram of the structure of an AI video analysis inference acceleration system based on multi-terminal deployment provided in an embodiment of this application;

[0077] Figure 2 This is a schematic diagram of the structure of a hybrid programming acceleration module provided in an embodiment of this application;

[0078] Figure 3 A schematic diagram of the structure of a Python unit provided in this application;

[0079] Figure 4 This is a schematic diagram of the structure of an edge device proxy rendering module provided in this application. Detailed Implementation

[0080] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0081] In the description of the embodiments of this application, the words "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design that is described as "for example" or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design options. Rather, the use of the words "for example" or "for instance" is intended to present the relevant concepts in a specific manner.

[0082] In the description of the embodiments of this application, the term "multiple" means two or more. For example, multiple systems means two or more systems, and multiple screen terminals means two or more screen terminals. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. The terms "comprising," "including," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0083] To facilitate understanding of the methods and systems provided in the embodiments of this application, the background of the embodiments of this application will be introduced before introducing the embodiments of this application.

[0084] Traditional AI video analytics systems typically rely on simple, standard interfaces, such as a grid layout, which are easy to use but have limited scope and field of view. These systems lack the ability to intelligently focus on and respond to critical events, requiring operators to manually review all footage to pinpoint the source of the problem in emergency situations, thus extending response time.

[0085] To address these issues, existing technologies employ advanced video analytics algorithms, utilizing deep learning and computer vision to achieve intelligent analysis of surveillance videos and real-time event detection and early warning. However, in practical applications, these algorithms still face challenges: firstly, compatibility and standardization issues between devices from different manufacturers increase the difficulty of algorithm deployment and integration; secondly, insufficient concurrent processing capabilities for massive video streams may cause processing delays or missed information at critical moments.

[0086] In summary, traditional video analytics systems suffer from limitations such as simplified interfaces but limited scope and field of view, and slow response to critical events. While existing intelligent video analytics algorithms have made some improvements in this area, poor device compatibility and weak large-scale concurrent processing capabilities still hinder practical applications. Further enhancing video capture range, collaborative computing capabilities, and response speed for critical tasks are urgent technological directions that need to be addressed.

[0087] After the background introduction above, those skilled in the art can understand the problems existing in the prior art. The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0088] Based on the aforementioned background technology, further please refer to... Figure 1 , Figure 1 This application provides a schematic diagram of the structure of an AI video analysis and inference acceleration system based on multi-terminal deployment. This system can be implemented using a computer program or run as an independent utility application. Specifically, in this application embodiment, the method can be applied to a server, but it can also be applied to electronic devices such as servers. An AI video analysis and inference acceleration system based on multi-terminal deployment includes the following steps:

[0089] The system includes a virtual device pool 1, an AI computing power center 2, a hybrid programming acceleration module 3, and an edge device proxy rendering module 4;

[0090] The hybrid programming acceleration module is used to generate video data retrieval information based on the received analysis and reasoning task information, and send the video data retrieval information to the AI ​​computing power center;

[0091] Specifically, the hybrid programming acceleration module can generate the necessary video data retrieval information based on the received video analysis and inference task information and send it to the AI ​​computing center. This operation is designed because video analysis and inference is a complex process that requires a large amount of relevant video data as input. To reduce the complexity of subsequent video data acquisition and improve efficiency, the hybrid programming acceleration module analyzes and generates the necessary video data retrieval information immediately upon receiving a specific inference task. This information can include parameters such as video source, time period, and resolution. After generating the video data retrieval information, the hybrid programming acceleration module sends this information to the AI ​​computing center. The computing center has video data storage and management capabilities, and can quickly locate and extract the required video data upon receiving the information. This avoids other modules needing to manage video data sources themselves, reducing system complexity. By generating accurate video data retrieval information in advance through the hybrid programming acceleration module, the entire video analysis and inference process can acquire only video data strongly related to the current task, rather than the entire dataset. This significantly reduces the amount of data, lowers the computational load, and improves processing efficiency.

[0092] This design primarily focuses on optimizing the inference process, enabling video analytics systems to become more accurate and efficient. The hybrid programming acceleration module can generate the necessary video data retrieval information based on the received video analytics inference task information and send it to the AI ​​computing center. This operation is designed because video analytics inference is a complex process that requires a large amount of relevant video data as input. To reduce the complexity of subsequent video data acquisition and improve efficiency, the hybrid programming acceleration module immediately analyzes and generates the necessary video data retrieval information upon receiving a specific inference task. This information can include parameters such as video source, time period, and resolution.

[0093] After generating video data retrieval information, the hybrid programming acceleration module sends this information to the AI ​​computing center. The computing center, possessing video data storage and management capabilities, can quickly locate and extract the required video data upon receiving the information. This avoids other modules needing to manage video data sources independently, reducing system complexity. By pre-generating precise video data retrieval information through the hybrid programming acceleration module, the entire video analysis inference process can acquire only video data strongly relevant to the current task, rather than the entire dataset. This significantly reduces the amount of data, lowers the computational load, and improves processing efficiency. In short, this design primarily focuses on optimizing the inference process, enabling the video analysis system to evolve towards precision and efficiency.

[0094] The AI ​​computing center is used to receive video data request information and generate video data request instructions based on the video data request information.

[0095] Specifically, after receiving video data request information from the hybrid programming acceleration module, the AI ​​computing center generates further video data request instructions based on this information. This process is designed to transform and refine the video data from initial information to actual data retrieval. The request information sent by the hybrid programming acceleration module may only include basic video attributes and features, while the computing center itself possesses a complete video data index and storage mapping. Therefore, the AI ​​computing center needs to transform and generate specific, operable video data request instructions based on the request information, such as index number and storage address. Upon receiving the video data request information, the AI ​​computing center initiates a parsing program to identify the information, extract key parameters as query conditions, and find matching video data in the center's database. Then, the system automatically generates precise request instructions pointing to this video data, including detailed information such as storage location and retrieval method.

[0096] Through this instruction conversion step, external modules only need to provide business-related video data features, without needing to concern themselves with underlying details such as data storage format and retrieval method. The computing center refines the instructions, transforming abstract external calls into concrete internal operations, greatly simplifying the complexity of interfaces and information exchange between systems. This further improves the collaborative efficiency of video analysis and inference.

[0097] The virtual device pool is used to acquire physical video data based on video data call commands and preprocess the physical video data to obtain standardized preliminary video data. Specifically, the virtual device pool can acquire the corresponding physical video data based on video data call commands output by the AI ​​computing center, and perform preprocessing to standardize the video, forming preliminary standardized video data. This process is designed because raw physical video data from different sources has varying formats, and directly using it would increase the complexity of subsequent analysis. The virtual device pool can directly acquire the required video source based on commands without needing to consider video format issues. Then, through a unified preprocessing workflow, various types of physical video data are decoded, converted, and encoded to output standardized preliminary video data format.

[0098] Specifically, after receiving a call command, the virtual device pool can directly locate the physical video device or storage of the target and retrieve the specified video data. Then, the system initiates an automated preprocessing procedure to perform format conversion, encoding standardization, and other operations on the video data. During processing, the data can be converted to a format optimized for video analysis, resulting in preliminary video data with standardized content and encoding. This preprocessing workflow allows for data standardization and optimization before video analysis, unifying long-sequenced, complex-formatted, and inconsistently encoded data into a standardized state, simplifying subsequent processing. It also avoids redundant operations such as repeatedly converting formats from different video sources, contributing to improved overall video analysis and processing efficiency.

[0099] Based on the above embodiments, as an optional embodiment, the virtual device pool is also used to abstract physical video devices into virtual devices and register the virtual devices into the virtual device pool.

[0100] Specifically, this design takes into account the differences between physical video devices from different manufacturers and models, as direct calls would increase system complexity. Therefore, setting up a unified virtual device abstraction layer enables standardized management of devices.

[0101] Specifically, for each type of physical camera, storage device, etc., the virtual device pool defines a set of virtual device standards aligned with its functionality. During registration, the virtual device pool collects information such as the parameters and interfaces of the physical device, uses an adapter to map and convert it into the attribute form of the standard virtual device, establishes a mapping relationship between the two, and registers it in the pool.

[0102] In this way, when accessing video data externally, only a request needs to be sent to the virtual device pool, without needing to know the specific physical device. The pool will then use the registered mapping to convert and direct the request to the corresponding physical device to complete the operation. This abstraction layer standardizes and unifies access to physical video devices, significantly reducing the complexity of calls and making the system easily scalable and compatible with different devices. External modules only need to interact with a small number of virtual devices, simplifying video data acquisition and management.

[0103] The hybrid programming acceleration module is used to perform hybrid programming analysis and inference on standardized preliminary video data to obtain analysis and inference results;

[0104] Specifically, the hybrid programming acceleration module can perform hybrid programming analysis and reasoning on standardized preliminary video data to ultimately obtain the analysis results of the video content. This step is designed because video analysis is a computationally intensive process requiring flexible AI analysis algorithms. Hybrid programming can fully leverage the advantages of different programming languages ​​to improve analysis efficiency. Specifically, after receiving the standardized preliminary video data, the hybrid programming acceleration module can preprocess it using programming languages ​​such as C++ to improve data processing speed. Then, with the help of a compatible unit, the data and structure are converted into AI programming languages ​​such as Python for recognition. On the Python side, its machine learning and deep learning algorithms are used to build a model for video content analysis and understanding, perform reasoning on complex scenes, and obtain accurate analysis results.

[0105] This hybrid programming approach leverages the computational performance advantages of languages ​​like C++ while utilizing the unique AI capabilities of languages ​​like Python. Compared to a single programming language, it enables highly efficient parallel and pipelined processing of complex video analysis tasks. It significantly reduces inference time, meeting the real-time and efficiency requirements of video analysis.

[0106] Based on the above embodiments, as an optional embodiment, the hybrid programming acceleration module further includes: an SJF algorithm module;

[0107] The SJF algorithm module is used to determine the estimated execution time of the analytical reasoning task based on the analytical reasoning task information, and select the analytical reasoning task with the shortest estimated execution time as the next analytical reasoning task to be executed.

[0108] Specifically, the hybrid programming acceleration module also includes a Shortest Job First (SJF) algorithm module. This module determines the estimated execution time of a task based on information from the analysis and inference task, and selects the task with the shortest estimated execution time as the next analysis and inference task to be executed. This step is designed to improve the real-time performance and average response time of task processing by employing the Shortest Job First (SJF) algorithm.

[0109] Specifically, the SJF algorithm module prioritizes shortest jobs by estimating task execution time, which improves module utilization, reduces average waiting time, and enhances real-time analysis capabilities. Compared to first-come, first-served scheduling, it manages and optimizes complex video analysis task flows more efficiently, enabling the system to continuously, efficiently, and stably handle a large number of concurrent analysis tasks. The introduction of the SJF scheduling strategy is a crucial step in optimizing module task flow control.

[0110] Based on the above embodiments, as an optional embodiment, please refer to... Figure 2 The hybrid programming acceleration module includes C++ unit 31, data compatibility unit 32, and Python unit 33;

[0111] The C++ unit is used to preprocess the initial video data to obtain the first video data;

[0112] Specifically, the hybrid programming acceleration module includes a C++ unit that preprocesses the initial video data and outputs the first video data. This design is based on the consideration that video data processing is a computationally intensive process, and leveraging the high efficiency of C++ can accelerate preprocessing. Specifically, the C++ unit performs preprocessing operations on the input initial video data, such as decoding, denoising, cropping, and bitrate adjustment. During these processes, the low-level calls and parallel computing advantages of the C++ language enable high-speed bit manipulation and conversion of video data. After C++ preprocessing, the initial video data is transformed into the optimized first video data format, laying the foundation for subsequent analysis and processing. The preprocessing stage, handled by the C++ unit, significantly reduces the time consumed in basic data processing, allowing for faster execution of complex subsequent analyses. Compared to directly using Python or similar languages ​​for preprocessing, the C++ unit significantly improves computational speed and parallel capabilities. This enables the overall system to achieve fast, pipelined processing, which is crucial for meeting the real-time requirements of video analysis.

[0113] Based on the above embodiments, as an optional embodiment, the C++ unit includes: a data decomposer, a multi-path classifier, a decoder, a renderer, a painter, an encoder, a wrapper, and an integrator;

[0114] A data decomposer is used to decompose preliminary video data into video data packets and put the video data packets into a queue to obtain a video data packet queue.

[0115] Specifically, the C++ unit includes a data decomposer that breaks down the initial video data into video data packets and places these packets into a queue, creating a video data packet queue. This step is designed because the initial video data volume is often very large, making direct processing inefficient. Setting up a data decomposer to break it down into smaller batches of video data packets enables streaming processing and improves parallel efficiency.

[0116] Specifically, the data decomposer divides the initial input video data into multiple small packets according to rules such as temporal order and spatial resolution. Each packet contains video content over a period of time or space. The system then distributes these packets into a queue and maintains an index relationship. In this way, the initial video data is decomposed into a queue of sequentially arranged video data packets. This decomposition allows for parallel and distributed processing of video data on a packet-by-packet basis, with different packets operating simultaneously on different processing units, fully utilizing computing resources. It also facilitates priority sorting based on packet criticality. Compared to the serial processing of the entire dataset, decomposition significantly improves the concurrency capabilities of C++ preprocessing and is an important means of achieving high-speed processing.

[0117] A multi-channel classifier is used to extract video data packets from the video data packet queue and perform multi-channel separation of the video data packets according to the preset data type to obtain a video frame queue and an audio data queue.

[0118] Specifically, the multi-path classifier can extract video data packets from the video data packet queue and perform multi-path separation on the video data packets according to preset data types, resulting in a video frame queue and an audio data queue. This step takes into account that video data packets typically contain both video frames and audio data. To further optimize the processing flow, these two types need to be separated and processed in parallel according to their respective types.

[0119] Specifically, the multi-path classifier sequentially extracts video data packets, detects the data format in each packet, and identifies different component types such as video frames and audio data. Then, the classifier organizes the video frame portion into a frame sequence and stores it in a video frame queue, and organizes the audio portion into audio data segments and stores them in an audio data queue. Through multi-path separation, video and audio data can be separated for targeted processing. Video frames can be quickly decoded to obtain image information, and audio can undergo optimizations such as noise reduction. Compared to processing mixed data as a whole, multi-path separation achieves divide-and-conquer, significantly improving the parallel efficiency of C++ preprocessing and preparing for subsequent algorithm processing.

[0120] The decoder is used to decode the video frame data in the video frame queue to obtain decoded video frame data, and convert the video frame data into pixel data that can be processed by image processing.

[0121] Specifically, the decoder decodes the video frame data in the video frame queue, obtaining the decoded video frame data, and converts it into pixel data suitable for image processing. This step is designed because video frames are stored in compressed encoding form, requiring decoding to obtain image information. Converting them into pixel data can then be optimized using image processing algorithms.

[0122] Specifically, the decoder plays a crucial role in the video processing workflow, acting as a connector and converter. It sequentially reads the video frame data output from the preprocessing stage, calls decoding algorithms to parse and decompress the compressed frame content, and restores the image and audio data to their original, algorithm-processable format. The decoded frame image is converted into a digital pixel matrix representing color and brightness, enabling subsequent modules to perform image analysis algorithms such as object detection and scene classification, using the image as the processing unit. The introduction of the decoder allows compressed video data to be easily converted into an algorithm-friendly format, providing a foundation for optimization processing and content analysis algorithms. Connecting the compression encoding stage and the algorithm processing stage, it plays a vital role in the video cloud processing workflow and is one of the core components for realizing video data utilization.

[0123] A renderer is used to process pixel data to obtain rendered video frames.

[0124] Specifically, the renderer processes the decoded pixel data and outputs rendered video frames. This step is designed to enhance and adjust the image, improving visual quality. Specifically, the renderer acquires the decoded pixel data and inputs it into a predefined image rendering model, performing image processing steps such as color adjustment, sharpening, and brightness optimization. This removes video noise, enhances contrast and clarity, and outputs a richly colored rendering effect. The rendered pixel data is then reorganized into aligned video frames.

[0125] Renderers effectively enhance the visual quality of video frames, improving overall image quality. High-quality frame input also helps subsequent algorithms extract more refined features, increasing the accuracy of analysis results. Renderers provide direct support for improving the quality of video content.

[0126] The renderer is used to finally draw the rendered video frames to obtain the completed video frames;

[0127] Specifically, the renderer can perform final rendering on the rendered video frames, resulting in a completed video frame. This step is designed to further render the video, embedding richer visual information into the frame and providing support for subsequent analysis. Specifically, the renderer acquires video frame data with improved rendering quality. Then, it performs graphic rendering operations such as border drawing, feature labeling, and time watermarking on the frame image. These renderings serve to annotate, explain, and supplement the image content. After rendering, a video frame that combines rendering effects with embedded auxiliary information is output. Compared to simply rendering, rendering can embed more features beneficial for analysis into the image. This provides additional support for the algorithm to understand the image content, improving the analysis results. The rendered content also serves an explanatory function when displaying the results. The application of the renderer can comprehensively enhance the information content and analyzability of video frames.

[0128] An encoder is used to re-encode the drawn video frames to obtain encoded video data packets;

[0129] Specifically, the encoder can re-encode the rendered video frames to obtain encoded video data packets. This step is designed because the rendered video frames exist in raw bitmap form, which is large in size and inconvenient to store and transmit. Re-encoding can compress the data volume.

[0130] Specifically, the encoder acquires the enhanced video frame bitmap data. Then, it uses video coding algorithms to perform processes such as directional prediction, DCT transform, and entropy coding to compress and package the frame information, ultimately outputting a reduced-size, transmittable encoded video data packet. This re-encoding process significantly reduces the amount of video frame data, lowering storage and transmission costs. It also facilitates subsequent storage and network transmission. Compared to the original bitmap, the encoded and compressed data packet is more suitable as the output of C++ preprocessing. The introduction of the encoder enables efficient compression of the enhanced video frames for output and is a key module for achieving data reduction of the processing results.

[0131] A wrapper is used to encapsulate encoded video data packets to obtain a wrapped video data stream.

[0132] Specifically, the encapsulator encapsulates the encoded video data packets to obtain a encapsulated video data stream. This step is designed because the encoded video data packets are not directly usable video stream formats and need further encapsulation into a standard format for easy reading and use. Specifically, the encapsulator obtains the video data packets output by the encoder. It then adds synchronization words, timestamps, packet headers, and other information, organizes the packet order, and encapsulates them into frame and packet structures conforming to video stream format standards. This outputs a standardized, encapsulated video data stream. This encapsulation process converts encoded video packets into a usable video stream format. The standardized stream format facilitates storage and subsequent distribution. Compared to the initial stream before decoding, the stream output by the encapsulator has both compression and format optimization capabilities. The introduction of the encapsulator enables the C++ preprocessing unit to output a standard-compliant, usable video stream.

[0133] A wrapper is used to encapsulate audio data packets in an audio data queue to obtain a wrapped audio data stream.

[0134] Specifically, the encapsulator can encapsulate audio data packets in the audio data queue to obtain a encapsulated audio data stream. This step is designed because the original audio data packets need further format encapsulation to form a standard audio data stream.

[0135] Specifically, the encapsulator extracts audio data packets from the audio data queue. For each packet, it adds header information, a timestamp, and organizes the order of the packets. Finally, it forms an encapsulation format that conforms to the audio stream format standard, containing ordered audio data packets within the stream. This results in a directly readable and usable encapsulated audio data stream. This encapsulation process converts raw audio packets into an audio stream, making it a format that the audio processing system can recognize and use. Standardized encapsulated audio streams facilitate system storage, processing, and transmission. The introduction of the encapsulator module also allows for standardized output of multi-channel separated audio data.

[0136] An integrator is used to combine encapsulated audio data streams and encapsulated video data streams to obtain the first video data.

[0137] Specifically, the integrator can merge the encapsulated audio data stream and the encapsulated video data stream to obtain the first video data. This step is designed because the audio and video streams are processed separately in the preceding multiplexing process, and a step is needed to reassemble them into a complete video data format.

[0138] Specifically, the integrator acquires the audio and video streams output by the encapsulator. It then matches and calibrates them based on their timelines and synchronization information, merging the two streams into a single synchronized composite stream. This results in the first video data, containing complete video frames and matching audio content. This integration process re-aggregates the separated audio and video streams, a crucial step in generating optimized first video data similar to the initial video data. The introduction of the integrator enables complex preprocessing workflows to ultimately output new video data that integrates the processing results, serving as input for subsequent algorithmic analysis.

[0139] A data compatibility unit is used to perform data structure conversion on the first video data to obtain the second video data;

[0140] Specifically, the data compatibility unit can transform the data structure of the first video data output by the C++ unit to obtain the second video data. This step is designed because languages ​​like C++ and Python have different data organization methods, and direct transmission would cause incompatibility. Setting up a data compatibility unit enables data conversion and transmission between the two languages, allowing them to be used together.

[0141] Specifically, the data compatibility unit uses pre-defined components to analyze the first video data in C++, extracting its core data structures and organizational patterns. Then, using an adapter, it converts the C++ structures and memory layouts into Python dictionaries and arrays. This enables seamless migration between the two language data models. Through the data compatibility unit's conversion, it ensures that the first video data can be output in a second video data format that Python can recognize. Subsequent Python units can then directly process this data without needing to address language-level incompatibility issues. This data structure compatibility greatly enhances the system's flexibility, enabling efficient execution of hybrid programming models.

[0142] Based on the above embodiments, as an optional embodiment, the data compatibility unit is obtained in the following way:

[0143] The data structure of the first video data is converted into the data type in Python units using the preset ctypes, resulting in a data-compatible unit.

[0144] Specifically, the data structure of the first video data is converted into a data type in Python using the pre-defined ctypes module, resulting in a data-compatible unit. This step is designed because the video data structure processed by C++ may be incompatible with Python, requiring encapsulation and conversion to enable data exchange between the two. Specifically, data types conforming to Python interface requirements are predefined, such as image frame classes. Then, the ctypes module is used to convert and package the C++ video data, such as the pixel matrix of image frames, into objects that Python can recognize and process, such as NumPy arrays. This eliminates the differences in data structures between the two, enabling the loading and computation of C++ output in Python. This results in a compatible data format that can be directly called by Python modules. Compared to simple I / O exchange, using ctypes for conversion achieves zero-copy, high-efficiency conversion, avoiding unnecessary data parsing and copying, and saving data exchange costs. Introducing data-compatible units is one of the key steps in achieving seamless integration between C++ and Python.

[0145] The Python unit is used to analyze and reason about the second video data to obtain the analysis and reasoning results.

[0146] Specifically, the Python unit can analyze and infer the second video data output by the data compatibility unit, ultimately obtaining the analysis results of the video content. This design is based on the fact that Python, developed by Anthropic, possesses a wealth of machine learning and deep learning algorithm tools, making it ideal for analyzing and inferring complex video data. Specifically, the Python unit builds deep learning models such as convolutional neural networks, inputs the compatible and transformed second video data, and performs video understanding tasks such as scene recognition, object detection, and motion tracking. During model inference, Python's efficient AI libraries are used to achieve accurate analysis of the video content. The final output includes detailed video analysis and inference results, including scene descriptions, bounding boxes, and classification results. Compared to a pure C++ implementation, the Python unit can leverage the power of AI algorithms to achieve more intelligent video analysis and obtain richer results. The combination of the two balances computational speed and analysis quality. The design of the Python unit provides video analysis and understanding capabilities, which is key to achieving intelligent video processing. The output analysis results can support various upper-layer applications.

[0147] Based on the above embodiments, as an optional embodiment, please refer to... Figure 3The Python unit includes: a preprocessing unit and an analysis unit;

[0148] The preprocessing unit is used to preprocess the second video data to obtain preprocessed second video data;

[0149] Specifically, the Python unit includes a preprocessing unit that preprocesses the second video data to obtain preprocessed second video data. This step is designed to improve the input quality and efficiency of subsequent video analysis through Python preprocessing. Specifically, the preprocessing unit utilizes Python's image processing libraries to perform optimization operations such as denoising, cropping, and color correction on the input second video data. This removes unwanted influences, improves the clarity of video frames, and crops out useless areas, reducing the processing scale. After Python preprocessing, the effective information in the second video data is extracted, improving its quality and feature representation capabilities, outputting preprocessed and optimized second video data. Compared to direct analysis, introducing preprocessing improves the quality of input materials and reduces interference from irrelevant content, enabling video analysis to be completed more efficiently and accurately. The design of the preprocessing unit enhances the usability of materials for Python analysis.

[0150] The analysis and processing unit is used to input the preprocessed second video data into a preset analysis and reasoning model to obtain the analysis and reasoning results.

[0151] Specifically, the analysis and processing unit can input the preprocessed second video data into a preset analysis and inference model to obtain the analysis and inference results of the video content. This step is designed to leverage Python's machine learning technology to achieve automated video analysis and understanding.

[0152] Specifically, the analysis and processing unit constructs video analysis models such as convolutional neural networks and recurrent neural networks. The improved second-generation video data is sequentially fed into these models for processing. The models utilize deep learning capabilities to analyze video content, performing tasks such as object detection, behavior recognition, and scene description. Ultimately, this results in specific video analysis and reasoning results, including object bounding boxes, classification labels, and scene information. Compared to simple filtering and compression, leveraging Python's AI technology enables more intelligent video understanding and processing. The introduction of the analysis and processing unit significantly expands the functional scope of the video processing system, elevating it from the data level to the semantic analysis level, enabling the output of human-understandable video content analysis results. This provides support for complex video processing business needs.

[0153] The edge device proxy rendering module is used to render standardized preliminary video data based on the analysis and inference results to obtain target video data.

[0154] Specifically, the edge device proxy rendering module can render the pre-standardized video data based on the results of video analysis inference, and output the target video data. This design takes into account that different video analysis tasks may require different result display methods, and a single standardized video data may not meet the requirements. Setting up an independent rendering module allows for flexible generation of result videos.

[0155] Specifically, the rendering module receives video analysis results containing information such as scene and target details. Then, guided by key information extracted from the results, it designs different video rendering templates, such as target bounding box highlighting and trajectory overlay. These customized rendering templates are applied to a standardized preliminary video for rendering synthesis, outputting target video data rich in analytical information. This result-guided rendering approach allows the video analysis system to output personalized result videos for different needs, expanding the flexibility of the analysis process and result presentation. Users do not need to design templates themselves; they can intuitively obtain videos with embedded analytical information through the rendering module. The introduction of the rendering module lowers the barrier to entry and improves the usability of the video analysis solution.

[0156] Based on the above embodiments, as an optional embodiment, please refer to... Figure 4 The edge device proxy rendering module includes computing power devices, streaming media servers, and a central server;

[0157] The central server is used to determine configuration information based on the analysis and reasoning results;

[0158] Specifically, the edge device proxy rendering module includes a central server that determines configuration information based on video analysis and inference results. This step is designed so that rendering needs to determine the most suitable rendering scheme based on the analysis results, and the central server can handle this decision-making process.

[0159] Specifically, the central server receives video analysis and inference results, including detection bounding boxes and classification labels. Based on key information extracted from these results, such as the detected target category, quantity, location distribution, and scene attributes, it determines the optimal rendering configuration. This includes determining the rendering algorithm, the location and extent of the target area, and the method for highlighting key elements. Ultimately, it generates configuration information containing customized rendering parameters for the algorithm and regions. Compared to manual determination, the central server, through comprehensive consideration of the analysis results, can automatically decide on a reasonable rendering scheme. Intelligent decision-making generates accurate configuration information, guiding subsequent rendering devices to optimize the final result. The introduction of the central server significantly improves the automation level of rendering configuration.

[0160] The computing power equipment is used to render the initial video data according to the configuration information. The configuration information of the rendered initial video data includes the algorithm UUID, camera IP, and ROI.

[0161] Specifically, the computing devices can render the initial video data based on the configuration information provided by the central server, resulting in rendered preliminary video data. The configuration information includes algorithm UUID, camera IP, ROI, etc. This step is designed to use a customized rendering scheme based on the central server's rendering decisions, thereby optimizing the display and analysis results.

[0162] Specifically, the computing power device acquires a configuration file containing information such as the algorithm, the source of the image to be processed, and key regions. Based on this, it loads the specified algorithm model and identifies the source camera of the image to be processed. Then, for the region of interest in the configuration, it performs highlight rendering of the target recognition results, such as target bounding box drawing and trajectory overlay. After customized rendering, a video with rendered effects incorporating enhanced analysis information can be output. Compared to a single rendering mode, the computing power device can perform intelligent customized rendering by parsing the configuration information. Different analysis results can all be presented with the most suitable visualization. The collaborative approach of computing power improves the refinement of rendering and helps optimize the expression of results.

[0163] A streaming media server is used to convert the format of the rendered initial video data to obtain the target video data.

[0164] Specifically, the streaming media server can perform format conversion on the rendered preliminary video data to obtain the target video data. This step is designed with the consideration that the format of the rendered video data may not be suitable for use in different systems, and needs to be converted and standardized.

[0165] Specifically, the streaming media server acquires the pre-rendered video data output from the computing device. It then processes this data through encoding, compression, and encapsulation, converting the video into a format suitable for network transmission and playback, such as MP4. This ensures that parameters such as resolutions and bitrates meet the requirements of different usage scenarios. The final output is directly applicable target video data. Compared to the original rendered result, further formatting yields video data that is compatible with multiple systems and can be used directly. The introduction of the streaming media server standardizes the rendered result into a format required by the application environment, completing the streaming architecture from processing to application.

[0166] Based on the above embodiments, as an optional embodiment, the streaming media server is also used to transmit the target video data to the user terminal.

[0167] Specifically, the streaming media server can also transmit the target video data to the user terminal. This step is designed to deliver the processed and rendered video to the user, achieving an end-to-end process from processing to application.

[0168] Specifically, the streaming media server acquires formatted, ready-to-use target video data. Then, using streaming media transmission protocols, it pushes the video stream data to subscribed user terminals in real time over the network. Users can then access and play the intelligently analyzed and optimized real-time video stream. Compared to the original video, the streaming media server achieves instant distribution of intelligent enhancement results. Users enjoy a video viewing experience with embedded analytics. Streaming media transmission is a crucial step in connecting video cloud processing capabilities with application scenarios, completing a closed-loop application from cloud to end.

[0169] Based on the above embodiments, as an optional embodiment, the system further includes: a load distribution module;

[0170] The load distribution module is used to classify analysis and inference tasks, obtain classification results, and push analysis and inference tasks to the hybrid programming acceleration module of the corresponding device based on the computing power information of each device and the classification results.

[0171] Specifically, the system also includes a load allocation module, which classifies video analysis and inference tasks, obtains classification results, and rationally allocates tasks to corresponding hybrid programming acceleration modules based on device computing power information and classification results. This design consideration is that different types and complexities of analysis tasks require matching processing modules with different computing power to ensure processing efficiency. Specifically, the load allocation module receives complex video analysis and inference tasks, such as object detection and scene parsing. It first classifies the tasks by type and difficulty, labeling tasks requiring GPU computation, ordinary tasks, etc. Then, based on the module's computing power, such as GPU and CPU configuration, it allocates GPU-intensive tasks to GPU-enabled modules and ordinary analysis tasks to basic modules. This achieves optimal matching of computing power and tasks, ensuring the system's parallel processing efficiency. Compared to simple equal distribution, refined task classification and allocation maximize the capabilities of different modules. The introduction of the load allocation module enables complex analysis tasks to be efficiently scheduled and processed, effectively improving the overall system's analysis capabilities.

[0172] In the various embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some service interface; the indirect coupling or communication connection between systems or units may be electrical or other forms.

[0173] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0174] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0175] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0176] The above are merely exemplary embodiments of this disclosure and should not be construed as limiting the scope of this disclosure. Any equivalent changes and modifications made in accordance with the teachings of this disclosure shall still fall within the scope of this disclosure. Other embodiments of this disclosure will readily conceive of those skilled in the art upon consideration of the specification and the disclosure of practical truths.

[0177] This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not described in this disclosure. The specification and embodiments are to be considered exemplary only, and the scope and spirit of this disclosure are defined by the claims.

Claims

1. An AI video analysis and inference acceleration system based on multi-terminal deployment, characterized in that, The system includes a virtual device pool, an AI computing power center, a hybrid programming acceleration module, and an edge device proxy rendering module; The hybrid programming acceleration module is used to generate video data call information based on the received analysis and reasoning task information, and send the video data call information to the AI ​​computing power center. The AI ​​computing center is used to receive the video data request information and generate video data request instructions based on the video data request information. The virtual device pool is used to obtain physical video data according to the video data call instruction, and to preprocess the physical video data to obtain standardized preliminary video data. The hybrid programming acceleration module is used to perform hybrid programming analysis and reasoning on the standardized preliminary video data to obtain analysis and reasoning results; The edge device proxy rendering module is used to render the standardized preliminary video data according to the analysis and reasoning results to obtain the target video data.

2. The system according to claim 1, characterized in that, The virtual device pool is also used to abstract physical video devices into virtual devices and register the virtual devices into the virtual device pool.

3. The system according to claim 1, wherein the hybrid programming acceleration module comprises a C++ unit, a data compatibility unit, and a Python unit; The C++ unit is used to preprocess the preliminary video data to obtain the first video data; The data compatibility unit is used to perform data structure conversion on the first video data to obtain the second video data. The Python unit is used to analyze and reason about the second video data to obtain the analysis and reasoning results.

4. The system according to claim 3, characterized in that, The C++ unit includes: a data decomposer, a multi-path classifier, a decoder, a renderer, a drawer, an encoder, a wrapper, and an integrator; The data decomposer is used to decompose the preliminary video data into video data packets and put the video data packets into a queue to obtain a video data packet queue. The multi-path classifier is used to extract the video data packets from the video data packet queue and perform multi-path separation on the video data packets according to a preset data type to obtain a video frame queue and an audio data queue. The decoder is used to decode the video frame data in the video frame queue to obtain decoded video frame data, and convert the video frame data into pixel data that can be processed by image processing. The renderer is used to render the pixel data to obtain rendered video frames. The renderer is used to perform final rendering of the rendered video frames to obtain the rendered video frames. The encoder is used to re-encode the drawn video frames to obtain encoded video data packets; The encapsulator is used to encapsulate the encoded video data packets to obtain an encapsulated video data stream. The encapsulator is used to encapsulate the audio data packets in the audio data queue to obtain an encapsulated audio data stream; The integrator is used to integrate the encapsulated audio data stream and the encapsulated video data stream to obtain the first video data.

5. The system according to claim 3, characterized in that, The Python unit includes: a preprocessing unit and an analysis processing unit; The preprocessing unit is used to preprocess the second video data to obtain preprocessed second video data; The analysis and processing unit is used to input the preprocessed second video data into a preset analysis and reasoning model to obtain the analysis and reasoning results.

6. The system according to claim 1, characterized in that, The edge device proxy rendering module includes computing power devices, streaming media servers, and a central server; The central server is used to determine configuration information based on the analysis and reasoning results; The computing device is used to render the preliminary video data according to the configuration information to obtain the rendered preliminary video data. The configuration information includes algorithm UUID, camera IP, and ROI. The streaming media server is used to convert the format of the rendered preliminary video data to obtain the target video data.

7. The system according to claim 6, characterized in that, The streaming media server is also used to transmit the target video data to the user terminal.

8. The system according to claim 6, characterized in that, The system also includes: a load distribution module; The load allocation module is used to classify the analysis and reasoning tasks, obtain classification results, and push the analysis and reasoning tasks to the hybrid programming acceleration module of the corresponding device according to the computing power information of each device and the classification results.

9. The system according to claim 8, characterized in that, The hybrid programming acceleration module also includes: an SJF algorithm module; The SJF algorithm module is used to determine the estimated execution time of the analysis and reasoning task based on the analysis and reasoning task information, and select the analysis and reasoning task with the shortest estimated execution time as the next analysis and reasoning task to be executed.

10. The system according to claim 3, characterized in that, The data compatibility unit is obtained in the following way: The data structure of the first video data is converted into the data type in the Python unit using the preset ctypes to obtain the data compatible unit.

Images