AI video analysis reasoning acceleration system based on multi-terminal deployment

Abstract

The invention provides an AI video analysis reasoning acceleration system based on multi-terminal deployment, and relates to the technical field of video analysis. The system comprises a virtual device pool, an AI computing power center, a hybrid programming acceleration module and an edge device proxy rendering module. And the hybrid programming acceleration module generates video data calling information according to the received analysis reasoning task information and sends the video data calling information to the AI computing power center. The AI computing power center generates a video data calling instruction according to the video data calling information, and the virtual device pool obtains physical video data according to the instruction and preprocesses the physical video data to obtain standardized preliminary video data; and the hybrid programming acceleration module performs hybrid programming analysis and reasoning on the standardized preliminary video data to obtain an analysis and reasoning result. And finally, the edge device proxy rendering module renders the standardized preliminary video data according to an analysis and reasoning result to obtain target video data. According to the technical scheme, the system can improve the compatibility and concurrency of video analysis.

Description

Technical Field

[0001] The present application relates to the field of video analysis technology, and specifically to an AI video analysis inference acceleration system based on multi-terminal deployment. Background Art

[0002] Traditional AI video analysis systems usually rely on interfaces provided by manufacturers, such as the standard nine-square layout. Although this layout is simple and easy to use, it has limitations in actual operation. For example, in terms of monitoring range, the nine-square layout only provides a limited field of view and lacks a comprehensive grasp of macro information such as building complexes and road networks. In addition, traditional monitoring interfaces lack automatic focus and independent perception capabilities for key events. Therefore, when an emergency occurs, operators may have to manually check each picture to find the source of the problem. This approach will obviously prolong the response time to the event.

[0003] In existing technologies, advanced video analysis algorithms are generally used to address the limitations of traditional AI video analysis systems, especially in terms of monitoring range and event response speed. These algorithms use deep learning and computer vision technology to automatically analyze key information in surveillance videos and perform event detection and early warning in real time. However, although these video analysis algorithms have improved the intelligence level of surveillance systems to a certain extent, there are still some problems in practical applications. First, there are differences in compatibility and standardization between devices from different manufacturers, which leads to certain challenges in the deployment and integration of algorithms. Secondly, with the continuous increase in monitoring needs, concurrent processing capabilities have become an urgent problem to be solved. Existing video analysis systems often find it difficult to process a large number of surveillance video streams at the same time, resulting in processing delays or omissions of important information at certain critical moments. Summary of the invention

[0004] The present application provides an AI video analysis inference acceleration system based on multi-terminal deployment to improve the compatibility and concurrency of the video analysis system.

[0005] In a first aspect, the present application provides an AI video analysis and reasoning acceleration system based on multi-terminal deployment, and the method includes: The system includes a virtual device pool, an AI computing center, a hybrid programming acceleration module, and an edge device proxy rendering module; A hybrid programming acceleration module is used to generate video data call information based on the received analysis and reasoning task information, and send the video data call information to the AI ​​computing center; The AI ​​computing center is used to receive video data call information and generate video data call instructions according to the video data call information; The virtual device pool is used to obtain physical video data according to the video data calling instruction, and pre-process the physical video data to obtain standardized preliminary video data; A hybrid programming acceleration module is used to perform hybrid programming analysis and reasoning on standardized preliminary video data to obtain analysis and reasoning results; The edge device proxy rendering module is used to render the standardized preliminary video data according to the analysis and reasoning results to obtain the target video data.

[0006] By adopting the above technical solution, the hybrid programming acceleration module, the virtual device pool, the AI ​​computing power center, and the edge device proxy rendering module work together to realize real-time video analysis and reasoning. Specifically, the hybrid programming acceleration module generates video data call information based on the received analysis and reasoning task information, and sends it to the AI ​​computing power center. The AI ​​computing power center generates a video data call instruction based on the video data call information to call the physical video data in the virtual device pool. The virtual device pool obtains and preprocesses the physical video data according to the video data call instruction, and outputs standardized preliminary video data. After obtaining the standardized preliminary video data, hybrid programming analysis and reasoning are performed through the hybrid programming acceleration module to generate analysis and reasoning results. Finally, the edge device proxy rendering module renders the standardized preliminary video data according to the analysis and reasoning results to obtain the final target video data.

[0007] Through this collaborative process-based working mechanism, different modules can split complex video analysis and reasoning tasks, fully utilizing the advantages of each module and realizing dynamic matching of TASK-MODULE, thereby greatly shortening the entire analysis and reasoning time, enabling the system to output results according to real-time requirements. Therefore, the coordination and collaboration of the hybrid programming acceleration module, virtual device pool, AI computing center, and edge device proxy rendering module in this solution is the key to achieving real-time video analysis and reasoning.

[0008] Optionally, the virtual device pool is also used to abstract the physical video device into a virtual device and register the virtual device into the virtual device pool.

[0009] By adopting the above technical solution, the virtual device pool is also used to abstract physical video devices into virtual devices, and register these virtual devices in the virtual device pool.

[0010] Through this abstract registration mechanism of the virtual device pool, the standardization and unified management of a large number of physical video devices of different manufacturers, models, interfaces, etc. can be achieved. When calling video data, the system only needs to issue a unified call instruction to the virtual device pool without worrying about the specific physical device situation, which greatly simplifies the call logic. At the same time, the virtual device pool abstracts the complex and diverse physical video devices into standardized virtual devices, and also provides a unified data input form for subsequent video data processing.

[0011] In this technical solution, the system can obtain standardized video data input by calling the virtual device pool, avoiding the coupling and adaptation problems of processing video data from different sources. This creates good conditions for fast and efficient video analysis and reasoning. It can be said that the abstract registration mechanism of the virtual device pool provides great convenience for the interoperability between different physical video devices, and is an important basis for realizing real-time video analysis and reasoning. Integrating all video sources into a unified virtual device pool makes the processing flow smoother and more efficient, effectively shortens the analysis and reasoning time, and enhances the real-time response capability of the system.

[0012] Optional, hybrid programming acceleration modules include C++ unit, data compatibility unit, and Python unit; A C++ unit, used for preprocessing the preliminary video data to obtain first video data; A data compatibility unit, used for converting the data structure of the first video data to obtain the second video data; The Python unit is used to analyze and infer the second video data to obtain analysis and inference results.

[0013] By adopting the above technical solution, the hybrid programming acceleration module is composed of a C++ unit, a data compatibility unit and a Python unit. Among them, the C++ unit pre-processes the preliminary video data and outputs the first video data; the data compatibility unit performs structural conversion on the first video data and outputs the second video data; the Python unit then performs reasoning analysis on the second video data and finally obtains the analysis and reasoning results.

[0014] This hybrid programming approach leverages the advantages of different programming languages ​​and enables fast processing of video analysis tasks. The C++ unit uses the efficiency of the C++ language to preprocess video data, improving data processing speed. The data compatibility unit achieves data flow between C++ and Python through structural conversion, overcoming the difficulty of data incompatibility between languages. The Python unit uses Python's advantages in machine learning to perform fast analysis and reasoning.

[0015] It can be seen that the coordinated use of C++ units, data compatibility units, and Python units in the hybrid programming acceleration module improves the parallel processing capability of the system and realizes the pipeline-style efficient processing of video analysis tasks. The three work closely together, taking advantage of the fast processing speed of C++ and the advantages of Python in analysis, perfectly combining computing performance and analysis capabilities. Therefore, this hybrid programming method is the key to ensuring real-time video analysis reasoning. It runs through and accelerates the entire video processing process, enabling complex video analysis tasks to be completed quickly.

[0016] Optional, C++ units, including: data decomposer, multi-way classifier, decoder, renderer, painter, encoder, wrapper and integrator; A data decomposer, used for decomposing preliminary video data into video data packets, and placing the video data packets into a queue to obtain a video data packet queue; A multi-channel classifier is used to extract video data packets from the video data packet queue and perform multi-channel demultiplexing on the video data packets according to preset data types to obtain a video frame queue and an audio data queue; A decoder, used to decode the video frame data in the video frame queue to obtain decoded video frame data, and convert the video frame data into pixel data that can be used for image processing; A renderer, used for rendering pixel data to obtain rendered video frames; A renderer, used for performing final drawing on the rendered video frame to obtain a completed video frame; The encoder is used to re-encode the drawn video frame to obtain an encoded video data packet; An encapsulator, used for encapsulating the encoded video data packet to obtain an encapsulated video data stream; The encapsulator is used to encapsulate the audio data packets in the audio data queue to obtain an encapsulated audio data stream; The integrator is used to integrate the packaged audio data stream and the packaged video data stream to obtain the first video data.

[0017] By adopting the above technical solution, the C++ unit includes a data decomposer, a multi-channel classifier, a decoder, a renderer, a drawer, an encoder, a packager and an integrator, which can pre-process the video data and output the first video data. Among them, the data decomposer decomposes the preliminary video data into video data packets and merges them into a queue; the multi-channel classifier extracts the data packets from the queue and performs multi-channel separation to obtain a video frame queue and an audio data queue; the decoder decodes and converts the video frame queue; the renderer and the drawer render and draw the video frame; the encoder re-encodes the drawn video frame; the packager packages the video frame and audio data respectively; and finally the integrator synthesizes the packaged audio and video streams to form the first video data.

[0018] It can be seen that the various components in the C++ unit perform fine disassembly and professional processing of the video data, realizing fast decoding, rendering, encoding and reorganization of the video data. It gives full play to the execution efficiency advantage of C++ and greatly shortens the video preprocessing time through pipeline parallel processing. Therefore, the collaborative work of the components in the C++ unit is the key to the hybrid programming acceleration module to achieve efficient video processing. It performs fine-grained parallel processing of video data, which not only improves the preprocessing speed, but also lays the foundation for subsequent analysis. The design of the C++ unit is an important part of the system to achieve real-time video analysis and reasoning.

[0019] Optional, Python units, including: preprocessing unit and analysis processing unit; A preprocessing unit, used for preprocessing the second video data to obtain preprocessed second video data; The analysis and processing unit is used to input the preprocessed second video data into a preset analysis and reasoning model to obtain an analysis and reasoning result.

[0020] By adopting the above technical means, the Python unit includes a preprocessing unit and an analysis processing unit. The preprocessing unit preprocesses the second video data and outputs the preprocessed video data; the analysis processing unit inputs the preprocessed video data into a preset analysis and reasoning model to finally obtain the analysis and reasoning results.

[0021] It can be seen that the Python unit takes advantage of Python in machine learning and artificial intelligence and is specifically used for video data analysis and processing. The preprocessing unit optimizes and improves the video data, laying the foundation for the next step of analysis. The analysis and processing unit directly uses Python's powerful artificial intelligence analysis capabilities to perform rapid video reasoning analysis through pre-set reasoning models.

[0022] The design of the Python unit fully utilizes Python's unique advantages in data processing and artificial intelligence analysis, and realizes efficient reasoning of video analysis tasks. The cooperation between the preprocessing unit and the analysis processing unit forms a set of specialized processing flows for video analysis. Therefore, the application of the Python unit is the key to the hybrid programming module to achieve fast and accurate video analysis reasoning. It forms an efficient collaboration with the C++ unit, perfectly combining the computing performance of C++ and the analysis capabilities of Python, and is an important part of ensuring the system's real-time video analysis capabilities.

[0023] Optionally, the edge device proxy rendering module includes a computing device, a streaming media server, and a central server; A central server is used to determine configuration information based on the analysis and reasoning results; The computing device is used to render the preliminary video data according to the configuration information, and obtain the configuration information of the rendered preliminary video data including the algorithm UUID, the camera IP, and the ROI; The streaming media server is used to convert the format of the rendered preliminary video data to obtain the target video data.

[0024] By adopting the above technical solution, the edge device proxy rendering module is composed of a central server, a computing device and a streaming media server. Among them, the central server determines the configuration information based on the analysis and reasoning results; the computing device uses the configuration information to render the preliminary video data and output the rendered video data; the streaming media server converts the format of the rendered video data and finally forms the target video data.

[0025] It can be seen that the edge device proxy rendering module adopts a distributed edge computing architecture. The central server coordinates task allocation and directs edge computing devices to perform rendering processing based on dynamic analysis results. This method achieves logical decoupling of analysis and rendering, and the central server can flexibly schedule computing resources as needed. At the same time, edge devices process nearby to reduce data transmission and improve rendering efficiency. The streaming media server realizes the standardization of output results, helping to obtain target videos that can be used directly.

[0026] In summary, the design of the edge device proxy rendering module fully utilizes the advantages of edge computing and realizes distributed collaboration in video processing. The cooperation of the central server, computing power equipment, and streaming media server not only improves resource utilization efficiency, but also accelerates the rendering process. The application of this module is the key to the system's flexible and scalable real-time video analysis and processing. It provides an efficient and reliable way to output results, and is an important part of ensuring the smooth overall processing flow.

[0027] Optionally, the streaming media server is also used to transmit the target video data to the user end.

[0028] By adopting the above technical solution, the streaming media server can also transmit the finally generated target video data to the user end.

[0029] Through the transmission function of the streaming media server, the video analysis results can be quickly transmitted and used. Once the edge device completes the rendering of the video, the streaming media server can directly send the rendered target video to the required user end to meet the user's usage needs.

[0030] It can be seen that the transmission function of the streaming media server provides a timely output channel for the video analysis task. The target video data can be quickly transmitted back through the streaming media server without waiting for the entire processing flow to be completely completed, which greatly shortens the time from analysis to application and meets the real-time requirements.

[0031] In summary, the transmission function of the streaming media server enables extremely fast feedback of key video analysis results, which is an important means to ensure the real-time response of the video analysis system. It realizes the seamless connection between the video processing flow and user needs, and is one of the key components for the system to achieve strong real-time performance. The design of the streaming media server fully considers the actual application needs, so that the complex video analysis system has simple, real-time and efficient user interaction characteristics.

[0032] Optionally, the system further comprises: a load distribution module; The load distribution module is used to classify the analysis and reasoning tasks, obtain the classification results, and push the analysis and reasoning tasks to the hybrid programming acceleration module of the corresponding device based on the computing power information and classification results of each device.

[0033] By adopting the above technical solution, the system also includes a load distribution module, which can classify video analysis and reasoning tasks, and push the tasks to the corresponding hybrid programming acceleration module based on the device computing power information and task classification results.

[0034] Through the intelligent scheduling of the load distribution module, the video analysis tasks can be reasonably allocated, so that the limited computing power can be fully and efficiently utilized. It scientifically classifies tasks according to the type and priority of tasks, and then matches them to the devices equipped with the corresponding hardware for execution, avoiding inefficient random allocation. At the same time, the module can also comprehensively consider the computing power of each device, reasonably formulate tasks, and avoid overloading.

[0035] It can be seen that the load distribution module realizes the dynamic matching of tasks and resources, which can keep the video analysis system in an optimal working state at all times. It makes full use of limited computing resources, so that complex video analysis tasks can be processed quickly and efficiently. Therefore, the application of the load distribution module is the key to realizing large-scale real-time video analysis. It realizes the refined management of tasks and resources and is an important support for ensuring overall processing efficiency.

[0036] Optional, hybrid programming acceleration module, also includes: SJF algorithm module; The SJF algorithm module is used to determine the estimated execution time of the analysis and reasoning tasks according to the analysis and reasoning task information, and select the analysis and reasoning task with the shortest estimated execution time as the next analysis and reasoning task to be executed.

[0037] By adopting the above technical solution, the hybrid programming acceleration module also includes an SJF algorithm module, which can determine the estimated execution time based on the information of the analysis and reasoning task, and select the task with the shortest time for priority execution.

[0038] Through this shortest job first scheduling strategy, critical and simple video analysis tasks can be completed as quickly as possible. For analytical reasoning tasks with predictable execution time, the SJF algorithm can scientifically determine their processing order, thereby shortening the waiting time of important tasks. It can be seen that the application of the SJF algorithm module further optimizes the real-time and critical task responsiveness of video analysis. On the basis of the load distribution module, it performs fine-grained task sorting and scheduling within the module. The use of the SJF algorithm ensures that high-priority short jobs can be executed immediately, greatly reducing the delay of important event analysis and improving real-time performance.

[0039] In summary, the SJF algorithm module strengthens the real-time response capability of the video analysis system to key events by finely managing the execution order of analysis tasks. It cooperates with the load distribution module to achieve multi-level real-time optimization within the system, which is one of the important means to ensure low latency end-to-end. The design of the SJF algorithm module improves the processing efficiency of important video analysis tasks.

[0040] Optionally, the data compatible unit is obtained by: The data structure of the first video data is converted into a data type in the Python unit by using the preset ctypes to obtain a data compatible unit.

[0041] By adopting the above technical solution, the data compatible unit is obtained by converting the C++ data structure of the first video data into a Python data type by using the preset ctypes.

[0042] As a bridge for data structure conversion, ctypes realizes the seamless conversion of the first video data output by the C++ unit to the input of the Python unit. Through ctype conversion, video data processed by different programming languages ​​can interact and transfer, so that the system can realize hybrid programming and give full play to the advantages of the two languages. It can be seen that the structural conversion realized by ctype is the key to the fast and effective hybrid programming module. It eliminates the differences in data format and structure between C++ and Python, enables the two to work closely together, improves the parallel computing speed of video analysis, shortens the processing time, and thus enhances the real-time response capability of the system.

[0043] In summary, the data structure conversion achieved by the data compatibility unit with the help of ctype is the basis for ensuring the efficient execution of hybrid programming, and is also one of the important reasons why the Analysis system achieves good real-time performance. It solves the problem of data mismatch between languages, makes the system process coherent and smooth, and thus greatly improves the speed of video analysis reasoning.

[0044] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. This application realizes real-time video analysis and reasoning through the collaborative work between the hybrid programming acceleration module, the virtual device pool, the AI ​​computing power center and the edge device proxy rendering module. Specifically, the hybrid programming acceleration module generates video data call information based on the received analysis and reasoning task information, and sends it to the AI ​​computing power center. The AI ​​computing power center generates a video data call instruction based on the video data call information to call the physical video data in the virtual device pool. The virtual device pool obtains and preprocesses the physical video data according to the video data call instruction, and outputs standardized preliminary video data. After obtaining the standardized preliminary video data, hybrid programming analysis and reasoning are performed through the hybrid programming acceleration module to generate analysis and reasoning results. Finally, the edge device proxy rendering module renders the standardized preliminary video data according to the analysis and reasoning results to obtain the final target video data.

[0045] Through this collaborative process-based working mechanism, different modules can split complex video analysis and reasoning tasks, fully utilizing the advantages of each module and realizing dynamic matching of TASK-MODULE, thereby greatly shortening the entire analysis and reasoning time, enabling the system to output results according to real-time requirements. Therefore, the coordination and collaboration of the hybrid programming acceleration module, virtual device pool, AI computing center, and edge device proxy rendering module in this solution is the key to achieving real-time video analysis and reasoning.

[0046] 2. This application realizes real-time video analysis and reasoning through the collaborative work between the hybrid programming acceleration module, the virtual device pool, the AI ​​computing power center, and the edge device proxy rendering module. Specifically, the hybrid programming acceleration module generates video data call information based on the received analysis and reasoning task information, and sends it to the AI ​​computing power center. The AI ​​computing power center generates a video data call instruction based on the video data call information to call the physical video data in the virtual device pool. The virtual device pool obtains and preprocesses the physical video data according to the video data call instruction, and outputs standardized preliminary video data. After obtaining the standardized preliminary video data, hybrid programming analysis and reasoning are performed through the hybrid programming acceleration module to generate analysis and reasoning results. Finally, the edge device proxy rendering module renders the standardized preliminary video data according to the analysis and reasoning results to obtain the final target video data.

[0047] Through this collaborative process-based working mechanism, different modules can split complex video analysis and reasoning tasks, fully utilizing the advantages of each module and realizing dynamic matching of TASK-MODULE, thereby greatly shortening the entire analysis and reasoning time, enabling the system to output results according to real-time requirements. Therefore, the coordination and collaboration of the hybrid programming acceleration module, virtual device pool, AI computing center, and edge device proxy rendering module in this solution is the key to achieving real-time video analysis and reasoning.

[0048] 3. The C++ unit of this application includes a data decomposer, a multi-channel classifier, a decoder, a renderer, a drawer, an encoder, a packager and an integrator, which can pre-process the video data and output the first video data. Among them, the data decomposer decomposes the preliminary video data into video data packets and merges them into a queue; the multi-channel classifier extracts the data packets from the queue, performs multi-channel separation, and obtains a video frame queue and an audio data queue; the decoder decodes and converts the video frame queue; the renderer and the drawer render and draw the video frame; the encoder re-encodes the drawn video frame; the packager encapsulates the video frame and audio data respectively; and finally the integrator synthesizes the encapsulated audio and video stream to form the first video data. It can be seen that the various components in the C++ unit have carried out fine disassembly and professional processing of the video data, realizing the rapid decoding, rendering, encoding and reorganization of the video data. It gives full play to the execution efficiency advantage of C++, and greatly shortens the time of video preprocessing through pipeline parallel processing. Therefore, the collaborative work of the components in the C++ unit is the key to the hybrid programming acceleration module to achieve efficient video processing. It performs fine-grained parallel processing on video data, which not only improves the preprocessing speed, but also lays the foundation for subsequent analysis. The design of the C++ unit is an important part of the system to achieve real-time video analysis and reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the structure of an AI video analysis and reasoning acceleration system based on multi-terminal deployment provided in an embodiment of the present application; Figure 2 A schematic diagram of the structure of a hybrid programming acceleration module provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of a Python unit provided for this application; Figure 4 A structural diagram of an edge device proxy rendering module provided in this application. DETAILED DESCRIPTION

[0050] In order to enable technicians in this field to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0051] In the description of the embodiments of the present application, words such as "for example" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "for example" or "for example" is intended to present related concepts in a specific way.

[0052] In the description of the embodiments of the present application, the meaning of the term "multiple" refers to two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprise", "have" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0053] In order to facilitate understanding of the method and system provided by the embodiments of the present application, before introducing the embodiments of the present application, the background of the embodiments of the present application is first introduced.

[0054] Traditional AI video analysis systems usually rely on simple standard interfaces, such as a nine-square layout, which is easy to use but has limited scope and field of view. Such systems lack the ability to intelligently focus on and respond to key events. In an emergency, operators need to manually check all images to find the source of the problem, which prolongs the response time.

[0055] To solve the above problems, the existing technology uses advanced video analysis algorithms to achieve intelligent analysis of surveillance videos and real-time event detection and early warning through deep learning and computer vision. However, in actual applications, such algorithms still face challenges: first, compatibility and standardization issues between devices from different manufacturers increase the difficulty of algorithm deployment and integration; second, the ability to concurrently process massive video streams is insufficient, which may cause processing delays or omissions of information at critical moments.

[0056] In summary, traditional video analysis systems have problems such as simplified interfaces but limited scope and field of view, and slow response to key events. Existing intelligent video analysis algorithms have partially improved this area, but problems such as poor device compatibility and weak large-scale concurrent processing capabilities still affect practical applications. Further improving the video acquisition range, collaborative computing capabilities, and key task response speed is a technical direction that needs to be solved urgently.

[0057] After the background introduction of the above content, those skilled in the art can understand the problems existing in the prior art. The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments.

[0058] Based on the above background technology, please refer to Figure 1 , Figure 1 A schematic diagram of the structure of an AI video analysis and reasoning acceleration system based on multi-terminal deployment provided in an embodiment of the present application. The system can be implemented by a computer program or can be run as an independent tool application. Specifically, in an embodiment of the present application, the method can be applied on a server, but can also be applied to electronic devices such as servers. An AI video analysis and reasoning acceleration system based on multi-terminal deployment includes the following steps: The system includes a virtual device pool 1, an AI computing power center 2, a hybrid programming acceleration module 3, and an edge device proxy rendering module 4; A hybrid programming acceleration module is used to generate video data call information based on the received analysis and reasoning task information, and send the video data call information to the AI ​​computing center; Specifically, the hybrid programming acceleration module can generate the required video data call information based on the received video analysis and reasoning task information, and send it to the AI ​​computing center. The design consideration of this operation is that video analysis and reasoning is a complex process that requires a large amount of relevant video data as input to achieve. In order to reduce the complexity of subsequent acquisition of video data and improve efficiency, the hybrid programming acceleration module will immediately analyze and generate the video data call information required for the task after receiving the specific reasoning task. This information may include parameters such as video source, time period, resolution, etc. After generating the video data call information, the hybrid programming acceleration module will send this information to the AI ​​computing center. The computing center has the storage and management function of video data, and can quickly locate and extract the required video data after receiving the information. This avoids the need for other modules to manage the video data source by themselves, reducing the complexity of the system. By generating accurate video data call information in advance by the hybrid programming acceleration module, the entire video analysis and reasoning process can only obtain video data that is strongly related to the current task, rather than obtaining the entire data set. This significantly reduces the amount of data, reduces the computing load, and improves processing efficiency.

[0059] It can be said that this design mainly considers the optimization of the reasoning process, so that the video analysis system can develop towards precision and efficiency. The hybrid programming acceleration module can generate the required video data call information based on the received video analysis reasoning task information and send it to the AI ​​computing center. The design consideration of this operation is that video analysis reasoning is a complex process that requires a large amount of relevant video data as input to be realized. In order to reduce the complexity of subsequent video data acquisition and improve efficiency, the hybrid programming acceleration module will immediately analyze and generate the video data call information required for the task after receiving the specific reasoning task. This information can include parameters such as video source, time period, and resolution.

[0060] After generating the video data call information, the hybrid programming acceleration module will send this information to the AI ​​computing center. The computing center has the storage and management functions of video data, and can quickly locate and extract the required video data after receiving the information. This avoids the need for other modules to manage the video data source themselves, reducing the complexity of the system. By generating accurate video data call information in advance through the hybrid programming acceleration module, the entire video analysis and reasoning process can only obtain video data that is strongly related to the current task, rather than obtaining the entire data set. This significantly reduces the amount of data, reduces the computing load, and improves processing efficiency. It can be said that this design mainly considers the optimization of the reasoning process, so that the video analysis system can develop towards accuracy and efficiency.

[0061] The AI ​​computing center is used to receive video data call information and generate video data call instructions according to the video data call information; Specifically, after receiving the video data call information sent by the hybrid programming acceleration module, the AI ​​computing center will further generate video data call instructions based on this information. The design consideration of this process is that conversion and refinement are required from video data information to actual acquisition of video data. The call information sent by the hybrid programming acceleration module may only include the basic attributes and characteristics of the video, while the computing center itself has a complete video data index and storage mapping relationship. Therefore, the AI ​​computing center needs to convert and generate specific and operational video data call instructions based on the call information, such as index serial number, storage address, etc. After receiving the video data call information, the AI ​​computing center will start the parsing program to identify the information, extract key parameters as query conditions, and find matching video data in the center's database. Then, the system will automatically generate precise call instructions pointing to these video data, including detailed information such as storage location and reading method.

[0062] Through this step of instruction conversion, the external module only needs to provide business-related video data features without paying attention to underlying details such as data storage format and reading method. The computing center refines the instructions and converts external abstract calls into internal specific operations, greatly simplifying the interface and information exchange complexity between systems. This further improves the collaborative efficiency of video analysis and reasoning.

[0063] The virtual device pool is used to obtain physical video data according to the video data call instruction, and preprocess the physical video data to obtain standardized preliminary video data; specifically, the virtual device pool can obtain the corresponding physical video data based on the video data call instruction output by the AI ​​computing power center, and preprocess the video to standardize the video to form preliminary standardized video data. The design consideration of this process is that the original physical video data from different sources has different formats, and direct use will increase the complexity of subsequent analysis. The virtual device pool can directly obtain the required video source according to the instruction without paying attention to the video format issue. Then, through a unified preprocessing process, various types of physical video data are decoded, converted, encoded, and other processes are performed to output a standardized primary video data format.

[0064] Specifically, after receiving the call instruction, the virtual device pool can directly find the physical video device or storage of the instruction target and obtain the specified video data. Then, the system will start the automated preprocessing program to perform operations such as format conversion and encoding standardization on the video data. During the processing, you can specify the conversion to a format that can optimize video analysis, and obtain preliminary video data results with standardized content and encoding. The introduction of this preprocessing process can standardize and optimize the data before the video enters the analysis, unify data with long time sequences, complex formats, and different encodings into a standard state, and simplify subsequent processing. It also avoids redundant operations such as repeated format conversion of different video sources. It helps to improve the overall video analysis and processing efficiency.

[0065] Based on the above embodiment, as an optional embodiment, the virtual device pool is further used to abstract the physical video device into a virtual device, and register the virtual device into the virtual device pool.

[0066] Specifically, the consideration of this design is that physical video devices of different manufacturers and models are different, and direct calls will increase system complexity. Therefore, setting up a unified virtual device abstraction layer can achieve standardized management of devices.

[0067] Specifically, for each type of physical camera, storage device, etc., the virtual device pool will define a set of virtual device standards that align with its functions. During registration, the virtual device pool will collect information such as parameters and interfaces of the physical device, use the adapter to map it into the attribute form of the standard virtual device, establish a mapping relationship between the two, and register it in the pool.

[0068] In this way, when calling video data externally, you only need to send a call request to the virtual device pool, without having to worry about the specific physical device. The pool will convert and point to the corresponding physical device to complete the operation based on the registered mapping relationship. The introduction of this abstract layer realizes the standardization and unified access of physical video devices. It greatly reduces the complexity of calling and makes the system easy to expand and compatible with different devices. The external module only needs to interact with a small number of virtual devices, which simplifies the collection and management of video data.

[0069] A hybrid programming acceleration module is used to perform hybrid programming analysis and reasoning on standardized preliminary video data to obtain analysis and reasoning results; Specifically, the hybrid programming acceleration module can perform hybrid programming analysis and reasoning on the standardized preliminary video data, and finally obtain the analysis results of the video content. The design consideration of this link is that video analysis is a process with large computational intensity and requires flexible AI analysis algorithms. The hybrid programming method can make full use of the advantages of different programming languages ​​to improve analysis efficiency. Specifically, after the hybrid programming acceleration module receives the standardized preliminary video data, it can be pre-processed through programming languages ​​such as C++ to increase the data processing speed. Then, with the help of compatible units, the data and structure are converted to AI programming languages ​​such as Python for recognition. On the Python side, use its machine learning and deep learning algorithm tools to establish a model for video content analysis and understanding, perform reasoning on complex scenes, and obtain accurate analysis results.

[0070] This hybrid programming method not only takes advantage of the computing performance of languages ​​such as C++, but also utilizes the unique AI capabilities of languages ​​such as Python. Compared with a single programming language, it can enable efficient parallel and pipeline processing of complex video analysis tasks. It significantly reduces the inference time and can meet the real-time and efficiency requirements of video analysis.

[0071] Based on the above embodiment, as an optional embodiment, the hybrid programming acceleration module further includes: an SJF algorithm module; The SJF algorithm module is used to determine the estimated execution time of the analysis and reasoning tasks according to the analysis and reasoning task information, and select the analysis and reasoning task with the shortest estimated execution time as the next analysis and reasoning task to be executed.

[0072] Specifically, the hybrid programming acceleration module also includes an SJF algorithm module, which can determine the estimated execution time of the task based on the information of the analytical reasoning task, and select the task with the shortest estimated time as the next analytical reasoning task to be executed. The design consideration of this step is that the use of the short-job-first SJF algorithm can improve the real-time performance and average response time of task processing.

[0073] Specifically, the SJF algorithm module estimates the task execution time and implements short-job priority scheduling, which can improve module utilization, shorten the average waiting time, and enhance real-time analysis capabilities. Compared with first-come-first-served scheduling, it more efficiently manages and optimizes complex video analysis task flows, enabling the system to continuously, efficiently, and stably process a large number of concurrent analysis tasks. The introduction of the SJF scheduling strategy is an important measure to optimize the module task flow control.

[0074] Based on the above embodiment, as an optional embodiment, please refer to Figure 2 , the hybrid programming acceleration module includes a C++ unit 31, a data compatibility unit 32, and a Python unit 33; A C++ unit, used for preprocessing the preliminary video data to obtain first video data; Specifically, the hybrid programming acceleration module includes a C++ unit, which can preprocess the preliminary video data and output the first video data. The consideration of this design is that video data processing is a computationally intensive process, and the high-efficiency performance of C++ can speed up the preprocessing speed. Specifically, the C++ unit will perform preprocessing operations such as decoding, denoising, cropping, and adjusting the bit rate on the input preliminary video data. In these processes, the underlying call and parallel computing advantages of the C++ language can be used to achieve high-speed video data bit operations and conversions. After C++ preprocessing, the preliminary video data will be transformed into the first video data in an optimized format, laying the foundation for subsequent analysis and processing. The preprocessing link is undertaken by the C++ unit, which can greatly reduce the time consumption of basic data processing and enable complex subsequent analysis to be carried out faster. Compared with direct use of Python and other preprocessing, the C++ unit significantly improves the computing speed and parallel capability. It makes it possible for the overall system to achieve fast and pipelined processing, which is crucial to meet the real-time requirements of video analysis.

[0075] Based on the above embodiment, as an optional embodiment, the C++ unit includes: a data decomposer, a multi-way classifier, a decoder, a renderer, a drawer, an encoder, a packager and an integrator; A data decomposer, used for decomposing preliminary video data into video data packets, and placing the video data packets into a queue to obtain a video data packet queue; Specifically, the C++ unit includes a data decomposer, which can decompose the preliminary video data into video data packets and put the video data packets into a queue to obtain a video data packet queue. The design consideration of this step is that the volume of preliminary video data is often large and direct processing is not efficient enough. Setting a data decomposer to decompose it into small batches of video data packets can realize streaming processing and improve parallel efficiency.

[0076] Specifically, the data decomposer divides the input preliminary video data into multiple small packets according to rules such as time sequence and spatial resolution. Each packet contains video content over a period of time or space. Then, the system breaks these packets into columns, stores them in queues and maintains index relationships. In this way, the preliminary video data is decomposed into a queue of video data packets arranged in order. After this decomposition, the video data can be processed in parallel and diverted by packet, and different packets can be operated on different processing units at the same time to make full use of computing resources. It is also convenient to prioritize and sort the processing according to the criticality of the packet. Compared with the serial processing of the full amount of data, decomposition significantly improves the concurrency capability of C++ preprocessing and is an important means to achieve high-speed processing.

[0077] A multi-channel classifier is used to extract video data packets from the video data packet queue and perform multi-channel demultiplexing on the video data packets according to preset data types to obtain a video frame queue and an audio data queue; Specifically, the multi-channel classifier can extract video data packets from the video data packet queue, and demultiplex the video data packets according to the preset data type to obtain a video frame queue and an audio data queue. The consideration of this step is that the video data packet usually contains both video frames and audio data. In order to further optimize the processing flow, the two need to be separated and processed in parallel by type.

[0078] Specifically, the multi-channel classifier will sequentially extract video data packets, detect the data format in each packet, and determine different types of components such as video frames and audio data. Then, the classifier will organize the video frame part into a frame sequence and store it in the video frame queue, and organize the audio part into audio data fragments and store them in the audio data queue. After multi-channel separation, the two types of data, video and audio, can be separated for targeted processing. Video frames can be quickly decoded to obtain image information, and audio can be optimized by noise reduction. Compared with the overall processing of mixed data, multi-channel separation can achieve divide and conquer, greatly improving the parallel efficiency of C++ preprocessing and preparing for subsequent algorithm processing.

[0079] A decoder, used to decode the video frame data in the video frame queue to obtain decoded video frame data, and convert the video frame data into pixel data that can be used for image processing; Specifically, the decoder can decode the video frame data in the video frame queue, obtain the decoded video frame data, and convert it into pixel data that can be processed. The design consideration of this step is that the video frame is stored in a compressed and coded form and needs to be decoded to obtain the picture information. The conversion into pixel data can be optimized using image processing algorithms.

[0080] Specifically, the decoder plays a key role of connection and conversion in the video processing flow. It reads the video frame data output by the preprocessing link in sequence, calls the decoding algorithm to parse and decompress the compressed and encoded frame content, and restores the image and audio data to the original format that the algorithm can process. The decoded frame image is converted into a digital pixel matrix representing color and brightness, so that the subsequent modules can use the image as the processing unit to perform image analysis algorithms such as target detection and scene classification. The introduction of the decoder makes it possible to easily convert compressed and encoded video data into an algorithm-friendly mode, providing a basis for the application of optimization processing and content analysis algorithms. It connects the compression encoding link and the algorithm processing link, plays a key role in the video cloud processing process, and is one of the core components for realizing the utilization of video data.

[0081] A renderer, used for rendering pixel data to obtain rendered video frames; Specifically, the renderer can render the pixel data obtained by decoding and output the rendered video frame. The design consideration of this step is that rendering the video frame can enhance and adjust the picture and improve the visual effect. Specifically, the renderer obtains the pixel data after decoding and conversion, and inputs the defined image rendering model to perform image processing processes such as color adjustment, sharpening, and brightness optimization. This can remove video noise, enhance contrast and clarity, and output a colorful rendering effect. The rendered pixel data is then reorganized into aligned video frames.

[0082] The processing of the renderer can effectively improve the visual effect of the video frame and enhance the picture quality. High-quality frame input can also help subsequent algorithms extract more detailed features and improve the accuracy of analysis results. The renderer provides direct support for improving the quality of video content.

[0083] A renderer, used for performing final drawing on the rendered video frame to obtain a completed video frame; Specifically, the renderer can perform the final rendering of the rendered video frame to obtain the rendered video frame. The design consideration of this step is that further rendering based on rendering can embed richer visual information in the video frame to add assistance for subsequent analysis. Specifically, the renderer will obtain video frame data with better rendering effect. Then, graphic drawing operations such as border drawing, feature labeling, and time watermarking are performed on the basis of the frame image. These drawing contents can play a role in annotating, explaining and supplementing the image content. After the drawing is completed, a video frame with both rendering effect and embedded auxiliary information can be output. Compared with rendering alone, drawing can embed more features that are conducive to analysis in the image. This can provide additional support for the algorithm to understand the image content and improve the analysis effect. The drawing content can also serve as an explanation when the results are displayed. The application of the renderer can comprehensively improve the information content and analyzability of the video frame.

[0084] The encoder is used to re-encode the drawn video frame to obtain an encoded video data packet; Specifically, the encoder can re-encode the drawn video frame to obtain an encoded video data packet. The design consideration of this step is that the drawn video frame exists in the form of an original bitmap, which is large in size and inconvenient to store and transmit. Re-encoding can compress the data volume.

[0085] Specifically, the encoder obtains the bitmap data of the video frame after drawing enhancement. Then, the video encoding algorithm is used to perform directional prediction, DCT transformation, entropy coding and other processes to compress and package the frame information, and finally output a reduced-volume, transmittable encoded video data packet. This re-encoding process can greatly reduce the amount of video frame data and reduce storage and transmission costs. It is also convenient for subsequent storage and network transmission. Compared with the original bitmap, the encoded compressed data packet is more suitable as the output of C++ preprocessing. The introduction of the encoder enables the enhanced video frame to be efficiently compressed for output, and is a key module for realizing the data reduction of the processing results.

[0086] An encapsulator, used for encapsulating the encoded video data packet to obtain an encapsulated video data stream; Specifically, the encapsulator can encapsulate the encoded video data packet to obtain an encapsulated video data stream. The design consideration of this step is that the encoded video data packet is not a directly usable video stream format and needs to be further encapsulated into a standard format for easy reading and use. Specifically, the encapsulator obtains the video data packet output by the encoder. Then it adds synchronization words, timestamps, headers and other information, organizes the order of the packets, and encapsulates them into a frame and packet structure that conforms to the video stream format standard. In this way, a standardized encapsulated video data stream can be output. This encapsulation process realizes the conversion of the encoded video packet to the actual usable video stream format. The standardized stream format is convenient for storage and subsequent distribution and use. Compared with the initial stream before decoding, the stream output by the encapsulator has both compression and format optimization. The introduction of the encapsulator enables the C++ preprocessing unit to output a usable video stream that conforms to the standard.

[0087] The encapsulator is used to encapsulate the audio data packets in the audio data queue to obtain an encapsulated audio data stream; Specifically, the encapsulator can encapsulate the audio data packets in the audio data queue to obtain an encapsulated audio data stream. The design consideration of this step is that the original audio data packets need to be further format encapsulated to form a standard audio data stream.

[0088] Specifically, the encapsulator extracts audio data packets from the audio data queue. For each packet, it adds header information, timestamps, etc., and organizes the order relationship between packets. Finally, a package format that conforms to the audio stream format standard is formed, and the stream contains ordered audio data packets. In this way, an encapsulated audio data stream that can be directly read and used is obtained. This encapsulation process realizes the conversion of the original audio packet to the audio stream, making it a format that can be recognized and used by the audio processing system. The standardized encapsulated audio stream facilitates the storage, processing and transmission of the system. The introduction of the encapsulator module enables the audio data after multi-channel separation to be output in a standardized manner.

[0089] The integrator is used to integrate the packaged audio data stream and the packaged video data stream to obtain the first video data.

[0090] Specifically, the integrator can merge the encapsulated audio data stream and the encapsulated video data stream to finally obtain the first video data. The design consideration of this step is that the audio stream and the video stream are processed separately in the previous demultiplexing process, and a step is required to reintegrate them into a complete video data format.

[0091] Specifically, the integrator obtains the audio stream and video stream output by the encapsulator. It then matches and calibrates the two streams according to their timelines and synchronization information, and merges the two streams into a synchronized composite stream. Ultimately, the first video data containing a complete video frame and matching audio content is formed. This integration process realizes the re-aggregation of the separated audio and video stream data, and is a key step in generating the first video data that is similar to the preliminary video data but optimized. The introduction of the integrator enables the complex pre-processing process to ultimately output new video data that integrates the processing results, which can be used as input for subsequent algorithm analysis.

[0092] A data compatibility unit, used for converting the data structure of the first video data to obtain the second video data; Specifically, the data compatibility unit can convert the data structure of the first video data output by the C++ unit to obtain the second video data. The design consideration of this step is that there are differences in the organization of data between languages ​​such as C++ and Python, and direct transmission will cause mismatches. Setting up a data compatibility unit can realize data conversion and transmission between the two languages ​​so that they can be used together.

[0093] Specifically, the data compatibility unit uses preset components to analyze the first video data in C++ to obtain the core data structure and organization mode. Then use the adapter to convert the structure, memory layout, etc. in C++ into dictionaries, arrays, etc. in Python. This allows seamless migration of data modes between the two languages. Through the conversion of the data compatibility unit, it is ensured that the first video data can be output in the form of second video data recognized by Python. Subsequent Python units can process this data directly without having to worry about language-level mismatches. The compatibility of data structures greatly enhances the flexibility of the system and enables the efficient execution of mixed programming modes.

[0094] Based on the above embodiment, as an optional embodiment, the data compatibility unit is obtained by: The data structure of the first video data is converted into a data type in the Python unit by using the preset ctypes to obtain a data compatible unit.

[0095] Specifically, the data structure of the first video data is converted into the data type in the Python unit using the preset ctypes to obtain a data compatible unit. The design consideration of this step is that the video data structure processed by C++ may be incompatible with Python, and needs to be encapsulated and converted to realize data exchange between the two. Specifically, data types that meet the requirements of the Python interface will be predefined, such as image frame classes, etc. Then, through the ctypes module, the C++ video data, such as the pixel matrix of the image frame, etc., are converted and packaged into objects that can be recognized and processed by Python, such as numpy arrays in Python. This can eliminate the data structure differences between the two and realize the loading and calculation of C++ output content in Python. A compatible data format that can be directly called by the Python module is obtained. Compared with simple io exchange, using ctypes conversion can achieve zero-copy efficient conversion, avoid unnecessary data parsing and copying, and save data exchange costs. Introducing a data compatible unit is one of the key steps to achieve seamless connection between C++ and Python.

[0096] The Python unit is used to analyze and infer the second video data to obtain analysis and inference results.

[0097] Specifically, the Python unit can analyze and reason the second video data output by the data compatible unit, and finally obtain the analysis results of the video content. The consideration of this design is that Python has rich machine learning and deep learning algorithm tools developed by Anthropic, which is very suitable for analyzing and reasoning complex video data. Specifically, the Python unit will build deep learning models such as convolutional neural networks, input the second video data after compatible conversion, and perform video understanding tasks such as scene recognition, object detection, and motion tracking. In the process of model reasoning, Python's efficient AI library is used to achieve accurate analysis of video content. Finally, the detailed video analysis and reasoning results including scene description, target box, classification results, etc. can be output. Compared with pure C++ implementation, the Python unit can exert the power of AI algorithms to achieve more intelligent video analysis and obtain richer result information. The combination of the two can take into account both computing speed and analysis quality. The design of the Python unit provides the ability to analyze and understand videos, which is the key to realizing intelligent video processing. The output analysis results can provide support for a variety of upper-level applications.

[0098] Based on the above embodiment, as an optional embodiment, please refer to Figure 3 ,Python unit, including: preprocessing unit and analysis processing unit; A preprocessing unit, used for preprocessing the second video data to obtain preprocessed second video data; Specifically, the Python unit includes a preprocessing unit, which can preprocess the second video data to obtain the preprocessed second video data. The design consideration of this step is that the input quality and efficiency of subsequent video analysis can be improved through Python preprocessing. Specifically, the preprocessing unit will use Python's image processing library and other optimization operations to perform denoising, cropping, color adjustment and other optimization operations on the input second video data. Remove adverse effects and improve the clarity of video frames. And crop out useless areas to reduce the processing scale. After Python preprocessing, the effective information in the second video data is refined, the quality and feature expression capabilities are improved, and the preprocessed and optimized second video data is output. Compared with direct analysis, the introduction of preprocessing can improve the quality of input materials and reduce the interference of irrelevant content. Video analysis can be completed more efficiently and accurately. The design of the preprocessing unit improves the material availability of Python analysis.

[0099] The analysis and processing unit is used to input the preprocessed second video data into a preset analysis and reasoning model to obtain an analysis and reasoning result.

[0100] Specifically, the analysis and processing unit can input the preprocessed second video data into a preset analysis and reasoning model to obtain an analysis and reasoning result of the video content. The design consideration of this step is that the machine learning technology of Python can realize automated video analysis and understanding.

[0101] Specifically, the analysis and processing unit will build video analysis models such as convolutional neural networks and recursive neural networks. The second video data with improved quality will be passed into the model for processing in turn, and the deep learning ability of the model will be used to parse the video content to perform analysis tasks such as object detection, behavior recognition, and scene description. Finally, a specific video analysis reasoning result including object boxes, classification labels, scene information, etc. is formed. Compared with simple filtering and compression, the use of Python's AI technology can achieve smarter video understanding and processing. The introduction of the analysis and processing unit has greatly expanded the functional scope of the video processing system, elevating it from the data level to the semantic analysis level, and can output human-understandable video content analysis results. This provides support for complex video processing business needs.

[0102] The edge device proxy rendering module is used to render the standardized preliminary video data according to the analysis and reasoning results to obtain the target video data.

[0103] Specifically, the edge device proxy rendering module can render the preliminary standardized video data based on the results of video analysis reasoning and output the target video data. This design takes into account that different video analysis tasks may require different result presentation methods, and a single standardized video data may not meet the requirements. Setting up an independent rendering module can flexibly generate the result video.

[0104] Specifically, the rendering module receives the video analysis results containing information such as scenes and targets. Then, different video rendering templates are designed based on the key information extracted from the results, such as target frame protrusion and trajectory overlay. These customized rendering templates are applied to the standardized preliminary video, and the rendering synthesis is performed to output the target video data full of analysis information. This result-guided rendering method allows the video analysis system to output personalized result videos for different needs, expanding the flexibility of the analysis process and result display. Users do not need to design templates by themselves, and can intuitively obtain videos embedded with analysis information through the rendering module. The introduction of the rendering module lowers the threshold for use and improves the ease of use of the video analysis solution.

[0105] Based on the above embodiment, as an optional embodiment, please refer to Figure 4 ,The edge device proxy rendering module includes computing ,devices, streaming media servers and central servers; A central server is used to determine configuration information based on the analysis and reasoning results; Specifically, the edge device proxy rendering module includes a central server, which can determine the configuration information based on the video analysis and reasoning results. The design consideration of this step is that rendering needs to determine what rendering solution is most appropriate based on the analysis results, and the central server can be responsible for this decision-making process.

[0106] Specifically, the central server will receive the video analysis reasoning results including detection boxes, classification labels, etc. Then it will determine the optimal rendering configuration based on the key information extracted from the results, such as the detected target categories, quantities, location distribution, and scene attributes. Determine the rendering algorithm to be used, the location and range of the target area, the way to highlight key elements, etc. Finally, the configuration information containing customized rendering parameters such as algorithms and areas is formed. Compared with manual determination, the central server can automatically determine the reasonable rendering solution for the result by comprehensively considering the analysis results. Intelligent decision-making can generate accurate configuration information to guide subsequent rendering equipment to optimize the result presentation. The introduction of the central server improves the automation level of rendering configuration.

[0107] The computing device is used to render the preliminary video data according to the configuration information, and obtain the configuration information of the rendered preliminary video data including the algorithm UUID, the camera IP, and the ROI; Specifically, the computing device can render the preliminary video data according to the configuration information provided by the central server to obtain the rendered preliminary video data. The configuration information includes the algorithm UUID, camera IP, ROI, etc. The design consideration of this step is that it is necessary to use a customized solution to render the video based on the rendering decision of the central server to achieve the effect of optimizing the display of the analysis results.

[0108] Specifically, the computing device will obtain a configuration file containing information such as the algorithm, the source of the processed image, and key areas. Based on this, the specified algorithm model is loaded to identify the source camera of the image to be processed. Then, for the area of ​​interest in the configuration, the target recognition results are highlighted, such as target frame drawing, trajectory overlay, etc. After customized rendering, a rendering effect video incorporating analysis enhancement information can be output. Compared with a single rendering mode, the computing device can perform intelligent customized rendering by parsing the configuration information. Different analysis results can get the most matching visual presentation. The collaborative computing power approach improves the refinement of rendering and helps optimize the expression of results.

[0109] The streaming media server is used to convert the format of the rendered preliminary video data to obtain the target video data.

[0110] Specifically, the streaming media server can convert the format of the rendered preliminary video data to obtain the target video data. The design consideration of this step is that the format of the rendered video data may not be suitable for use in different systems and needs to be converted and standardized.

[0111] Specifically, the streaming media server obtains the rendered preliminary video data output by the computing device. It then encodes, compresses, and encapsulates it to convert the video into a format that supports network transmission and playback, such as MP4. It also ensures that the resolutions, bitrates and other parameter requirements of different usage scenarios are met. Finally, the target video data that can be directly applied is output. Compared with the original rendering results, further FORMAT can obtain video data output that is compatible with multiple systems and can be directly used. The introduction of the streaming media server can standardize the rendering results into the format required by the application environment, completing the streaming architecture from processing to application.

[0112] Based on the above embodiment, as an optional embodiment, the streaming media server is also used to transmit the target video data to the user end.

[0113] Specifically, the streaming media server can also transmit the target video data to the user end. The design consideration of this step is that the processed and rendered video is transmitted to the user, which can realize the end-to-end process from processing to application.

[0114] Specifically, the streaming media server obtains the formatted, directly usable target video data. Then, using the streaming media transmission protocol, the video stream data is pushed to the subscribed user terminal in real time through the network. The user terminal can obtain the real-time video stream that has been intelligently analyzed and optimized for rendering, and play it. Compared with the original video, the streaming media server realizes the instant distribution of intelligent enhancement results. Users can get a video experience embedded with analysis information. Streaming media transmission is a key step in connecting video cloud processing capabilities with application scenarios, completing a closed-loop application from cloud to end.

[0115] Based on the above embodiment, as an optional embodiment, the system further includes: a load distribution module; The load distribution module is used to classify the analysis and reasoning tasks, obtain the classification results, and push the analysis and reasoning tasks to the hybrid programming acceleration module of the corresponding device based on the computing power information and classification results of each device.

[0116] Specifically, the system also includes a load distribution module, which can classify video analysis and reasoning tasks, obtain classification results, and reasonably allocate tasks to corresponding hybrid programming acceleration modules based on device computing power information and classification results. The design consideration of this step is that analysis tasks of different types and complexities need to match processing modules with different computing powers to ensure processing efficiency. Specifically, the load distribution module will receive complex video analysis and reasoning tasks, such as target detection and scene analysis. It will first classify tasks by type and difficulty, marking tasks that require GPU computing, ordinary tasks, etc. Then, according to the computing power status of the module, such as GPU and CPU configuration, GPU-intensive tasks are allocated to GPU-containing modules, and ordinary analysis tasks are allocated to basic modules. In this way, the optimal matching of computing power and tasks is achieved to ensure the parallel processing efficiency of the system. Compared with simple equal distribution, refined task classification and matching can maximize the capabilities of different modules. The introduction of the load distribution module enables complex analysis tasks to be efficiently scheduled and processed, effectively improving the overall system analysis capabilities.

[0117] In the several implementations provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interfaces, and the indirect coupling or communication connection of the system or unit can be electrical or other forms.

[0118] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0119] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0120] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, mobile hard drives, magnetic disks or optical disks.

[0121] The above are only exemplary embodiments of the present disclosure and cannot be used to limit the scope of the present disclosure. That is, any equivalent changes and modifications made according to the teachings of the present disclosure are still within the scope of the present disclosure. After considering the disclosure of the specification and the truth of practice, those skilled in the art will easily think of other embodiments of the present disclosure.

[0122] This application is intended to cover any variation, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art not described in the present disclosure. The description and examples are to be regarded as exemplary only, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. An AI video analysis and reasoning acceleration system based on multi-terminal deployment, characterized in that: The system includes a virtual device pool, an AI computing power center, a hybrid programming acceleration module, and an edge device proxy rendering module; The hybrid programming acceleration module is used to generate video data call information according to the received analysis and reasoning task information, and send the video data call information to the AI ​​computing power center; The AI ​​computing center is used to receive the video data calling information and generate a video data calling instruction according to the video data calling information; The virtual device pool is used to obtain physical video data according to the video data calling instruction, and pre-process the physical video data to obtain standardized preliminary video data; The hybrid programming acceleration module is used to perform hybrid programming analysis and reasoning on the standardized preliminary video data to obtain analysis and reasoning results; The edge device proxy rendering module is used to render the standardized preliminary video data according to the analysis and reasoning results to obtain target video data.

2. The system according to claim 1, characterized in that The virtual device pool is also used to abstract the physical video device into a virtual device and register the virtual device into the virtual device pool.

3. The system according to claim 1, wherein the hybrid programming acceleration module comprises a C++ unit, a data compatibility unit, and a Python unit; The C++ unit is used to pre-process the preliminary video data to obtain first video data; The data compatibility unit is used to convert the data structure of the first video data to obtain the second video data; The Python unit is used to analyze and infer the second video data to obtain analysis and inference results.

4. The system according to claim 3, characterized in that The C++ unit includes: a data decomposer, a multi-way classifier, a decoder, a renderer, a drawer, an encoder, a packager and an integrator; The data decomposer is used to decompose the preliminary video data into video data packets, and put the video data packets into a queue to obtain a video data packet queue; The multi-channel classifier is used to extract the video data packets from the video data packet queue, and perform multi-channel separation on the video data packets according to preset data types to obtain a video frame queue and an audio data queue; The decoder is used to decode the video frame data in the video frame queue to obtain decoded video frame data, and convert the video frame data into pixel data that can be used for image processing; The renderer is used to render the pixel data to obtain a rendered video frame; The renderer is used to perform final rendering on the rendered video frame to obtain a rendered video frame; The encoder is used to re-encode the drawn video frame to obtain an encoded video data packet; The encapsulator is used to encapsulate the encoded video data packet to obtain an encapsulated video data stream; The encapsulator is used to encapsulate the audio data packets in the audio data queue to obtain an encapsulated audio data stream; The integrator is used to integrate the packaged audio data stream and the packaged video data stream to obtain the first video data.

5. The system according to claim 3, characterized in that The Python unit includes: a preprocessing unit and an analysis processing unit; The preprocessing unit is used to preprocess the second video data to obtain preprocessed second video data; The analysis and processing unit is used to input the preprocessed second video data into a preset analysis and reasoning model to obtain an analysis and reasoning result.

6. The system according to claim 1, characterized in that The edge device proxy rendering module includes a computing device, a streaming media server and a central server; The central server is used to determine configuration information according to the analysis and reasoning results; The computing device is used to render the preliminary video data according to the configuration information to obtain the rendered preliminary video data, wherein the configuration information includes an algorithm UUID, a camera IP, and a ROI; The streaming media server is used to convert the format of the rendered preliminary video data to obtain target video data.

7. The system according to claim 6, characterized in that The streaming media server is also used to transmit the target video data to the user end.

8. The system according to claim 6, characterized in that The system further comprises: a load distribution module; The load distribution module is used to classify the analysis and reasoning tasks, obtain classification results, and push the analysis and reasoning tasks to the hybrid programming acceleration module of the corresponding device based on the computing power information of each device and the classification results.

9. The system according to claim 8, characterized in that The hybrid programming acceleration module further includes: an SJF algorithm module; The SJF algorithm module is used to determine the estimated execution time of the analysis and reasoning task according to the analysis and reasoning task information, and select the analysis and reasoning task with the shortest estimated execution time as the next analysis and reasoning task to be executed.

10. The system according to claim 3, characterized in that The data compatibility unit is obtained in the following manner: The data structure of the first video data is converted into the data type in the Python unit by using the preset ctypes to obtain the data compatible unit.

Images