Real-time video analysis system of edge end

Abstract

The invention discloses a real-time video analysis system of an edge end, and belongs to the technical field of edge video analysis. The system comprises video analysis equipment, middleware and a field programmable logic gate array, the video analysis equipment and the field programmable logic gate array establish a remote procedure call for communication connection through the middleware; the video analysis equipment comprises a central processing unit and a graphics processor; the central processing unit is used for preprocessing a to-be-analyzed video; the graphics processor is used for analyzing the preprocessed video; and the field programmable logic gate array is used for distributing computing resources of the central processing unit and the graphics processing unit. The video analysis system provided by the invention realizes asynchronous transmission of system data by sharing the memory, can dynamically adjust the frequency and rate of data transmission according to network conditions and server load conditions, better adapts to different data flow conditions, improves the utilization rate of bandwidth, and realizes safe and stable data transmission.

Description

Technical Field

[0001] The present invention belongs to the technical field of edge video analysis and relates to a real-time video analysis system at an edge end. Background Art

[0002] A real-time video analysis system usually consists of a smart camera and an edge device equipped with a central processing unit (CPU) and a graphics processing unit (GPU). It generates video in the smart camera at the edge and performs video analysis at the edge to achieve real-time response and / or privacy protection. The real-time video analysis system can identify events involving spatial or temporal factors that occur in a time-limited video stream. This method combines the knowledge of computer vision and deep learning to achieve advanced processing of visual data and then analyze the important events contained therein. Real-time video analysis technology at the edge has shown great potential in many fields, with applications ranging from dynamic environment monitoring to complex production processes. However, real-time video analysis technology at the edge also faces technical problems such as insufficient real-time, security and stability of video and image data transmission within the video analysis system. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a real-time video analysis system at the edge, thereby improving the efficiency and security of data transmission in the real-time video analysis system.

[0004] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0005] The present invention provides a real-time video analysis system at the edge, comprising: a video analysis device, a middleware and a field programmable logic gate array;

[0006] The video analysis device and the field programmable logic gate array establish a communication connection through a remote procedure call via a middleware;

[0007] The video analysis device includes a central processing unit and a graphics processing unit;

[0008] The central processing unit is used to pre-process the video to be analyzed;

[0009] The graphics processor is used to analyze and process the pre-processed video to obtain an analysis result of the video to be analyzed;

[0010] The field programmable logic gate array is used to allocate computing resources of the central processing unit and the graphics processing unit.

[0011] Furthermore, the field programmable gate array is a dual-mirror field programmable gate array.

[0012] Furthermore, the hardware functions implemented by programming the field programmable gate array include: controlling the on, off and sleep states of the field programmable gate array; managing the mirror of the field programmable gate array; and processing data in the field programmable gate array.

[0013] Furthermore, analyzing and processing the preprocessed video includes: inputting the preprocessed video into a pre-established neural network model for model inference.

[0014] Further, the remote procedure call includes a stub component and a skeleton component;

[0015] The stub component is used to package the data to be transmitted in the video analysis device into a specific data structure, send it to the skeleton component through the middleware, and receive the data returned by the middleware;

[0016] The skeleton component is used for unpacking the received data and sending the data to the field programmable logic gate array, and sending the data returned by the field programmable logic gate array to the stub component through the middleware.

[0017] Furthermore, the stub component encapsulates the algorithm program required for video analysis as a function and provides a corresponding function interface;

[0018] The skeleton component provides multiple data processing modes for middleware and field programmable logic gate arrays, including: a one-time computing mode that produces a single result from a single input, and a streaming computing mode that continuously processes a series of data.

[0019] Furthermore, the video analysis device utilizes a microprocessor and a field programmable gate array to perform data transmission via middleware.

[0020] Furthermore, the middleware performs asynchronous data transmission by operating the shared memory.

[0021] Furthermore, the microprocessor and the field programmable logic gate array write and / or read the data in the shared memory in an asynchronous manner.

[0022] Further, it also includes an asynchronous data transfer controller;

[0023] The asynchronous data transfer controller supports the interface requirements of the microprocessor and the field programmable logic gate array; and adjusts the request priority of the microprocessor and the field programmable logic gate array by configuring the arbitration scheme of the asynchronous data transfer controller.

[0024] Compared with the prior art, the present invention has the following beneficial effects:

[0025] The real-time video analysis system at the edge provided by the present invention establishes a remote procedure call style interaction scheme through middleware to perform communication connection between the video analysis device and the field programmable gate array, and the middleware performs asynchronous data transmission through shared memory, so that the system can dynamically adjust the transmission parameters according to the network and server load conditions, effectively improve the utilization rate of bandwidth, and make full use of network resources; optimize data transmission according to actual conditions, and can improve the overall performance and responsiveness of the video analysis system, whether it is a stable network environment or network fluctuations, it can ensure the stability of data transmission; during the transmission process, it can reduce the risk of unauthorized access to data and protect the privacy of users. The present invention decouples the development of video analysis applications and the hardware development of field programmable gate arrays, and proposes a unified programming paradigm for dual-mirror field programmable gate arrays, so that the application is insensitive to the hardware details of the field programmable gate array, and only needs to schedule the computing resources of the field programmable gate array, without wasting development time costs on differentiated programming of different types of field programmable gate arrays. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A schematic diagram of the framework of a real-time video analysis system at an edge end provided by an embodiment of the present invention;

[0027] Figure 2 Schematic diagram of the framework of the real-time video analysis method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. The same reference numerals in the accompanying drawings indicate the same or similar components or parts. It should be understood by those skilled in the art that these drawings are not necessarily drawn to scale. The embodiments of the present application and the specific features in the embodiments are detailed descriptions of the technical solution of the present application, rather than limitations on the technical solution of the present application. In the absence of conflict, the embodiments of the present application and the technical features in the embodiments can be combined with each other.

[0029] The term "and / or" herein is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " herein generally indicates that the front and back associated objects are in an "or" relationship. The orientation or position relationship indicated by the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "inside", "outside", etc. is based on the orientation or position relationship shown in the accompanying drawings, and is only used to explain the relative position relationship, movement, etc. between the components in a certain specific posture. If the specific posture changes, the directional indication will also change accordingly. It is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present disclosure / application.

[0030] In addition, the terms "first", "second", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0031] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood by specific circumstances.

[0032] like Figure 1 As shown, an embodiment of the present invention provides a real-time video analysis system at the edge, and the system includes: a video analysis device, a middleware, and a field programmable gate array (FPGA).

[0033] like Figure 2 As shown, in the embodiment of the present invention, the video analysis device analyzes the real-time video mainly including the following steps:

[0034] Step 1: Get the video to be analyzed and the corresponding video analysis task;

[0035] Step 2: According to the video analysis task, a video resolution set and a neural network model set to be selected are obtained; according to the video to be analyzed, the video resolution set and the neural network model set, the computing time of the central processing unit (CPU) and the graphics processing unit (GPU) in the edge device is divided by the Bandit algorithm or other algorithms, and a video resolution is selected from the video resolution set, and a neural network model is selected from the neural network model set;

[0036] Step 3: Based on the divided computing time of the central processing unit and the graphics processing unit, as well as the selected video resolution and neural network model, the analysis result of the video to be analyzed is obtained.

[0037] The CPU is used to preprocess the video to be analyzed, which includes filtering out unnecessary video frames or extracting only the region of interest (ROI) in the video frame. Video preprocessing involves a large number of operations such as searching, sorting, and matching, and its computational workload belongs to instruction-intensive computing (IIC). The GPU is used to input the preprocessed video into the video of the neural network model for model inference, and finally obtain the analysis results of the video. The operations involved in model inference are simple but the amount of data is large, so the GPU performs data-intensive computing (DIC).

[0038] As a reconfigurable hardware, the field programmable gate array provides higher flexibility and customization than the central processing unit and the graphics processing unit. It can be reprogrammed at runtime to dynamically reallocate hardware resources. Therefore, in view of the dual computing resource scheduling problem of the central processing unit and the graphics processing unit, the present invention uses the field programmable gate array as a bridge between the two computing resources to allocate the computing resources of the central processing unit and the graphics processing unit. The field programmable gate array controls the central processing unit to perform preprocessing operations and the graphics processing unit to perform model reasoning operations based on the divided computing time of the central processing unit and the graphics processing unit.

[0039] The present invention uses a field programmable gate array to allocate the computing resources of the central processing unit and the graphics processing unit, and the field programmable gate array can realize dual-mirror switching. There are multiple models of dual-mirror series FPGAs, such as Intel Max10, Xilinx Artix-7, etc. Due to their differences in low-level specifications (such as the number of pins and instruction sets), the programming of different field programmable gate arrays is closely coupled with their specific models. Taking Intel Max10 and Xilinx Artix-7 to realize mirror switching as an example, mirror switching is to program the specified mirror into the logic unit of the field programmable gate array for executing the functions implemented in the mirror.

[0040] The image switching process in Intel Max10 is: first load the image file IMAGE0 into the image area, and Max10 stores the image in a fixed memory area named the master / slave image area. At this time, the image file is stored in the slave image area (SLAVE IMAGE); then initialize the write / read parameters, set the read width to 16 bits, the write width to 4 bits, and the maximum data width allowed for the field programmable logic gate array to read and write an image is 32 bits; finally, read the image from the image area and write the image to the FPGA logic unit supported by Intel's module. Max10 completes this process by sending signals to predefined output pins. Specifically, Max10 sends a signal 1 to pin BOOT SEL and a signal 0 to pinCFG_SEL.

[0041] The process of switching the mirror AGE0 in Xilinx Artix-7 is: first initialize the write / read parameters to set the read width and write width; then read the mirror from the memory and write the mirror to the FPGA logic unit. Xilinx Artix-7 completes this process through Xilinx's read module (Xilinx dual-boot read) and write module (Xilinx dual-boot write).

[0042] The above example illustrates that the dual-mirror FPGA programming paradigm is sensitive to hardware, but for the video analysis task of the present invention, it is only necessary to schedule the computing resources of the FPGA, so it is not desirable to waste development time on the differentiated programming of different models of FPGAs. Therefore, the present invention proposes a hardware function logic abstraction method to decouple the video analysis application development from the FPGA hardware development, and proposes a unified programming paradigm for the dual-mirror FPGA, so that the application is insensitive to the hardware details of the FPGA.

[0043] The computational functions of video analysis applications can be divided into: basic control, filtering, interference, background subtraction, and cropping bounding boxes. Therefore, the requirements for hardware functions can be summarized as follows: 1) hardware setting functions to control the state of the field programmable logic gate array, such as on, off, and sleep mode; 2) image management functions to manage the images of the field programmable logic gate array; 3) data processing functions to process the data in the field programmable logic gate array.

[0044] The present invention uses a field programmable logic gate array to allocate the computing resources of the central processing unit and the graphics processing unit. Therefore, a communication connection is required between the video analysis device and the field programmable logic gate array for data transmission. Specifically, each thread constructs a sending thread and a receiving thread to establish a communication connection. The sending thread consumes clock cycles to send data, and the receiving thread consumes clock cycles to receive data. The sending or receiving thread will block the opposite processor until the transmission is completed to ensure the correctness of the communication.

[0045] The video analysis device of the present invention utilizes a microprocessor and a field programmable logic gate array to transmit data through middleware.

[0046] When the clock frequencies of the microprocessor and the field programmable logic gate array do not match, this mechanism manifests itself as fast processor idleness. For example, the common clock frequency of a typical microprocessor such as STM32f1 is 64MHz, and the clock frequencies of Intel Max10 and Xilinx Artix-7 are 473.5 MHz and 464MHz respectively. The clock idling waste is very large, that is, four-fifths of the FPGA clock cycle will be wasted. Therefore, the present invention designs an asynchronous data transmission mechanism, and the microprocessor and the field programmable logic gate array transmit data by writing and / or reading shared memory in an asynchronous manner. Specifically, after a processor writes data to this shared memory, an interrupt will be triggered to allow another processor to read the data. After the reading is completed, the processor will be released for other computing tasks.

[0047] In order to support such a mechanism, the present invention designs an asynchronous data transfer controller and ensures that it supports the interface requirements of the microprocessor and field programmable logic gate array used in the present invention, such as bus width, data transfer rate and addressing capability. In order to prevent conflicts between the field programmable logic gate array and the microprocessor, the priorities of the two are reorganized. By configuring the arbitration scheme of the asynchronous data transfer controller, the priorities of the competing requests of the microprocessor and the field programmable logic gate array are adjusted to minimize contention and thus reduce delays. The microprocessor and the field programmable logic gate array can perform their own independent operations, reducing the need for frequent synchronization and improving the overall performance of the real-time video analysis system.

[0048] While abstracting the hardware functions, the present invention uses a framework design pattern to create a remote procedure call (RPC) style interaction scheme to call the dual-image switching function and simplify the process of switching different images. The remote procedure call method has the advantages of being simple and practical, so many heterogeneous systems adopt this transparent data communication and control of hardware IP cores.

[0049] Remote procedure calls usually include two key components: stub components and skeleton components. Stub components usually refer to a stub program that acts as a proxy between the client and the remote service. The client sends a request to the remote server by calling the local stub component. The stub component is responsible for packaging and sending the request parameters, and waiting for the server to complete the processing and return the result to the client. The skeleton component corresponds to the stub component and usually refers to a framework or component on the server side. It receives requests from the stub component, unpacks the request parameters, and calls the actual service logic. After the processing is completed, the skeleton component is responsible for packaging and sending the results back to the client's stub component.

[0050] In the present invention, the stub component is used to package the data to be transmitted in the video analysis device into a specific data structure, send it to the skeleton component through the middleware, and receive the data returned by the middleware; the skeleton component is used to unpack the received data and send it to the field programmable logic gate array, and send the data returned by the field programmable logic gate array to the stub component through the middleware.

[0051] Figure 1 The workflow of the middleware of the present invention is shown: 1) The video analysis device calls the function with parameters provided by the stub component; 2) The stub component packages the data to the middleware; 3) The middleware kernel deployed on the edge device transmits the data to the field programmable logic gate array by operating the shared memory; 4) The middleware kernel in the field programmable logic gate array transmits the data to the corresponding skeleton component of the accelerator; 5) The data is unpacked by the skeleton and passed to the accelerator. After completing the transmission task, it returns to the original state and prepares for the next task.

[0052] In addition, the stub component and the skeleton component provide a cleaner abstract model for the FPGA image and video analysis applications. The stub component provides a pure abstract model for the upper-level video analysis application, which encapsulates the hardware-level functions and provides the function interface required by each application. Different video analysis systems can directly call the functions provided by the stub component when using their own algorithms to analyze the video. The skeleton component provides an abstraction between the middleware and the FPGA accelerator, which can realize different data processing modes, such as a one-time calculation mode that produces a single result from a single input; a streaming calculation mode that continuously processes a series of data, etc.

[0053] The middleware connects the stub component and the skeleton component, and its core is the above-mentioned asynchronous communication mechanism based on shared memory. The middleware kernel writes data such as function parameters or return values ​​into a specific area and notifies the stub component and the skeleton component through an interrupt signal. In addition, the middleware kernel is also responsible for switching the skeleton component of the field programmable logic gate array. When the video analysis device calls the function provided by the stub component, it uses an accelerator other than the accelerator loaded in the field programmable logic gate array. The kernel is responsible for executing the mirror switching step of the field programmable logic gate array and loading the accelerator and skeleton component corresponding to the called stub function onto the field programmable logic gate array. In addition, the middleware kernel needs to manage the functional abstraction of the field programmable logic gate array like FPGA_ON. When encountering a function call, the kernel will find the address corresponding to the function and transfer the data to the corresponding memory address, thereby simplifying the complexity of the field programmable logic gate array in the use process.

[0054] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of this disclosure / application.

Claims

1. A real-time video analysis system at the edge, characterized in that: include: Video analytics equipment, middleware, and field-programmable gate arrays; The video analysis device and the field programmable logic gate array establish a communication connection through a remote procedure call via a middleware; The video analysis device includes a central processing unit and a graphics processing unit; The central processing unit is used to pre-process the video to be analyzed; The graphics processor is used to analyze and process the pre-processed video to obtain an analysis result of the video to be analyzed; The field programmable logic gate array is used to allocate computing resources of the central processing unit and the graphics processing unit.

2. The real-time video analysis system at the edge according to claim 1, characterized in that: The field programmable logic gate array is a double-mirror field programmable logic gate array.

3. The real-time video analysis system at the edge according to claim 2, characterized in that: The hardware functions implemented by programming the field programmable logic gate array include: controlling the on, off and sleep states of the field programmable logic gate array; managing the mirror of the field programmable logic gate array; and processing data in the field programmable logic gate array.

4. The real-time video analysis system at the edge according to claim 1, characterized in that: The analysis and processing of the preprocessed video includes: inputting the preprocessed video into a pre-established neural network model for model inference.

5. The real-time video analysis system at the edge according to claim 1, characterized in that: The remote procedure call includes a stub component and a skeleton component; The stub component is used to package the data to be transmitted in the video analysis device into a specific data structure, send it to the skeleton component through the middleware, and receive the data returned by the middleware; The skeleton component is used for unpacking the received data and sending the data to the field programmable logic gate array, and sending the data returned by the field programmable logic gate array to the stub component through the middleware.

6. The real-time video analysis system at the edge according to claim 5, characterized in that: The stub component encapsulates the algorithm program required for video analysis as a function and provides a corresponding function interface; The skeleton component provides multiple data processing modes for middleware and field programmable logic gate arrays, including: a one-time computing mode that produces a single result from a single input, and a streaming computing mode that continuously processes a series of data.

7. The real-time video analysis system at the edge according to claim 5, characterized in that: The video analysis device utilizes a microprocessor and a field programmable logic gate array to transmit data through a middleware.

8. The real-time video analysis system at the edge according to claim 7, characterized in that: The middleware performs asynchronous data transmission by operating the shared memory.

9. The real-time video analysis system at the edge according to claim 8, characterized in that: The microprocessor and the field programmable logic gate array write and / or read the data in the shared memory in an asynchronous manner.

10. The real-time video analysis system at the edge according to claim 9, characterized in that: Also included is an asynchronous data transfer controller; The asynchronous data transfer controller supports the interface requirements of the microprocessor and the field programmable logic gate array; and adjusts the request priority of the microprocessor and the field programmable logic gate array by configuring the arbitration scheme of the asynchronous data transfer controller.

Images