Intelligent camera design method based on ARM and FPGA architecture

Abstract

According to the design method of the intelligent camera based on the ARM + FPGA architecture, the high-level data processing capacity of an ARM processor and the high-performance computing capacity of an FPGA are combined, and optimization of a camera system in the aspects of performance, flexibility and cost is achieved. According to the technical scheme, the problem of complexity of a traditional FPGA camera in AI algorithm deployment can be effectively solved, meanwhile, the processing capacity of an ARM camera is improved, and the ARM camera has higher universality and adaptability.

Description

Technical Field

[0001] The present invention relates to the field of smart cameras, and in particular to smart camera technology based on ARM+FPGA architecture. Specifically, the present invention combines the advantages of ARM processors and FPGAs to improve the performance and efficiency of cameras in data processing and intelligent algorithm deployment. Background Art

[0002] Traditional cameras based on FPGA architecture are widely used in various image processing and computing tasks due to their high performance, low latency and low power consumption. However, FPGA has great development complexity and long development cycle in deploying AI algorithms, and its ecosystem is not as perfect as GPU or NPU solutions. This makes FPGA-based smart cameras face challenges in the process of implementing modern AI technology.

[0003] On the other hand, although ARM-based smart cameras have strong processing capabilities and flexibility, they usually lack the advantages of FPGA in parallel processing and real-time response. Therefore, how to effectively combine ARM and FPGA to balance efficient data processing and rapid deployment of intelligent algorithms has become an important research topic in the field of smart cameras.

[0004] This paper proposes a smart camera based on ARM+FPGA architecture, which aims to combine the high-level data processing capabilities of ARM processors with the high-performance computing capabilities of FPGAs to optimize the performance, flexibility and cost of the camera system. This innovative combination can effectively solve the complexity problem of traditional FPGA cameras in the deployment of AI algorithms, while improving the processing capabilities of ARM cameras, making them more versatile and adaptable. Summary of the invention

[0005] The present invention relates to a smart camera design method based on ARM and FPGA architecture, comprising the following steps:

[0006] S1. AI algorithm deployment steps: The AI ​​algorithm deployment process includes model conversion, compilation and deployment, testing and debugging, and integration and deployment;

[0007] S2 high-speed data transmission step: according to the preset data format and the corresponding protocol, establish a transmission link, through the MIPI interface, to achieve high-speed data transmission between the ARM processor and the FPGA; the high-speed data transmission adopts the MIPI CSI-2 protocol or MIPIDSI (display serial interface);

[0008] S3.FFMPEG video encoding and decoding steps: The ARM processor uses the FFMPEG library to encode and decode the video data;

[0009] S4. Serial port data receiving and sending step: The ARM processor sends and receives data through its built-in serial port interface, and the serial port communication of the serial port interface can be implemented through multi-threading.

[0010] Further, in step S1, the model conversion further includes: building a self-developed algorithm model, using Rockchip's RKNN-Toolkit2 tool to convert the self-developed algorithm model into the RKNN format, the conversion includes first converting the self-developed algorithm model from the source format to the ONNX format, and then using the RKNN-Toolkit2 tool to further convert it into the RKNN format, the compilation and deployment further includes installing related libraries and runtime environments, configuring a cross-compilation tool chain, and compiling libraries and frameworks that support target detection, and compiling and deploying them on the RK series platform; after completion, uploading the converted RKNN model to the development board, and running and testing it on the target platform.

[0011] The testing and debugging further include performance and memory evaluation of the model to ensure that its performance on the development board meets application requirements; verifying the actual effect of the model through system testing to ensure the stability and reliability of the model in the production environment; the integration and deployment further include integrating the tested algorithm model into actual applications, including integrating the model and related codes into the target application to ensure that the application can correctly call the model for reasoning and process the results; finally, a comprehensive system test is performed to verify the effect of the integration.

[0012] Further, in step S2, the preset data format includes an image data format and a data packet format, wherein for the image data format, the FPGA processor encodes the image data in a corresponding format and then sends it to the ARM through the MIPI CSI-2 interface; for the data packet format, the data transmitted by the MIPI CSI-2 interface is usually sent in the form of data packets, each data packet containing data and related control information.

[0013] The establishment of the transmission link further includes: data link configuration and clock signal configuration; wherein the data link configuration includes configuring the MIPI CSI-2 interface to include multiple data links, and data is transmitted in serial mode through the multiple data links; the FPGA processor sends image data through MIPI CSI-2, and the ARM receives and processes the data; the clock signal configuration includes: a clock signal line for synchronizing data transmission; the ARM processor and the FPGA synchronize the clock signal according to the clock signal line to ensure the correct reception and processing of the data.

[0014] Furthermore, the corresponding format includes a YUV format or an RGB format.

[0015] Further, in step S3, the ARM processor uses the FFMPEG library to encode and decode the video data, which further includes:

[0016] 1) Input source step: use the real-time camera data of the FPGA front end as the input source;

[0017] 2) Encoding step: Select the appropriate video codec and set encoding parameters according to the requirements of the target streaming platform;

[0018] 3) Packaging step: select a container format, usually a container format that complies with streaming media standards, such as FLV or MP4; configure stream parameters, and set appropriate stream parameters to ensure that the data can be correctly processed by the streaming media server;

[0019] 4) Push streaming step: select the protocol and determine the streaming protocol, which includes RTMP or HLS; configure the correct streaming URL, including the protocol, server address and stream name.

[0020] Further, the video codec includes H.264 (libx264) and H.265 (libx265),

[0021] The encoding parameters include bit rate, frame rate and resolution.

[0022] Further, in step S4, the serial communication includes:

[0023] 1) Sending data: When sending data, first configure the serial port parameters, then use the RS-422 serial port to send data. The processor converts the data into a differential signal and transmits it through the serial port interface;

[0024] 2) Receiving data: When receiving data, the processor obtains the differential signal from the RS-422 serial port interface and converts it into raw data.

[0025] Furthermore, the serial port parameters include: baud rate, data bits and stop bits.

[0026] Compared with cameras with traditional FPGA architecture, the design method of the smart camera based on ARM+FPGA architecture has a newly added ARM processor that can be responsible for the main control and high-level data processing of the system; compared with smart cameras with FPGA architecture, the present invention can deploy the most cutting-edge AI algorithms on the ARM processor quickly, efficiently and at low cost; compared with smart cameras based on ARM architecture, the present invention has stronger versatility, and FPGA camera modules with the same interface can be freely combined with ARM smart modules to meet the needs of different usage scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0028] The structures, proportions, sizes, etc. illustrated in the drawings of this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with this technology. They are not used to limit the conditions under which this application can be implemented, and therefore have no substantive technical significance. Any structural modification, change in proportion or adjustment of size, without affecting the effects and purposes that can be achieved by this application, should still fall within the scope of the technical contents disclosed in this application.

[0029] Figure 1 ARM+FGPA system architecture diagram in a specific implementation manner of the present invention.

[0030] Figure 2 This is a flowchart of the AI ​​algorithm deployment on the ARM processor in a specific implementation manner of the present invention.

[0031] Figure 3 This is the FFMPEG streaming flow chart in the specific implementation mode of the present invention.

[0032] Figure 4 This is a diagram of RTMP streaming results of deploying RK development board for yolov5 target detection and sort tracking in a specific implementation manner of the present invention.

[0033] Figure 5 This is a diagram of serial port multi-threaded data reception and transmission in a specific implementation manner of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the embodiments in the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0035] The core architecture of the present invention is composed of the ARM processor and FPGA of Rockchip RK series chips. The two are closely connected through the MIPI interface to ensure high-speed and stable data transmission. The ARM processor is responsible for the overall control of the system and high-level data processing, including the deployment of AI algorithms, serial port data transmission and reception, and video encoding and decoding operations. FPGA provides efficient parallel computing capabilities to assist the ARM processor in completing complex data processing tasks.

[0036] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0037] The present invention proposes a smart camera design method based on ARM and FPGA architecture, which specifically includes the following steps:

[0038] S1.AI Algorithm Deployment

[0039] ARM processors, with their superior computing power, can quickly deploy and execute the latest AI algorithms. This advantage enables the system to process and analyze video data in real time, and implement intelligent functions such as target detection, recognition, and tracking. The process of AI algorithm deployment mainly includes model conversion, compilation and deployment, testing and debugging, and integration and deployment.

[0040] (1) Model conversion

[0041] Use the RKNN-Toolkit2 tool officially provided by Rockchip to convert the self-developed algorithm model into the RKNN format. RKNN is a dedicated model format used by Rockchip NPU platforms (such as development boards), with a file extension of .rknn. The conversion process includes first converting the model from the source format (such as PyTorch's .pth) to the ONNX format, and then further converting it to the RKNN format using the RKNN-Toolkit2 tool, and performing necessary model quantization to optimize performance.

[0042] (2) Compilation and deployment

[0043] When compiling and deploying on the RK series platform, you need to install the relevant libraries and runtime environment, configure the cross-compilation tool chain, and compile the libraries and frameworks that support target detection. Subsequently, upload the converted RKNN model to the development board and run and test it on the target platform.

[0044] (3) Testing and debugging

[0045] Evaluate the model's performance and memory to ensure that its performance on the development board meets application requirements. Verify the model's actual effect through system testing to ensure the model's stability and reliability in the production environment.

[0046] (4) Integration and deployment

[0047] Integrate the tested algorithm into the actual application, including integrating the model and related code into the target application, ensuring that the application can correctly call the model for reasoning and process the results. Finally, conduct a comprehensive system test to verify the effect of the integration.

[0048] S2. High-speed data transmission

[0049] Through the MIPI interface, high-speed data transmission is achieved between the ARM processor and the FPGA. This ensures real-time transmission of video data and AI processing results, and improves the overall performance and responsiveness of the system. MIPI is a standardized interface for high-speed data transmission in mobile devices and embedded systems. It includes a variety of protocols and interface standards, the most commonly used of which are MIPI CSI-2 (Camera Serial Interface 2) and MIPI DSI (Display Serial Interface). MIPI CSI-2 is dedicated to the transmission of image data and can support high-resolution and high-frame-rate image data transmission. MIPI CSI-2 and DSI typically use the MIPI D-PHY physical layer standard to support high-speed differential signal transmission.

[0050] (1) Data format and protocol

[0051] Image data format: The FPGA processor encodes the image data in a certain format (such as YUV, RGB), and then sends it to ARM through the MIPI CSI-2 interface.

[0052] Data packet format: Data transmitted by the MIPI CSI-2 interface is usually sent in the form of data packets, each of which contains data and related control information.

[0053] (2) Transmission link

[0054] Data link configuration: The MIPI CSI-2 interface includes multiple data links, through which data is transmitted serially, providing high-bandwidth transmission capabilities. The FPGA processor sends image data through MIPI CSI-2, and the ARM receives and processes the data.

[0055] Clock signal: The MIPI interface also includes a clock signal line for synchronizing data transmission. Both the ARM processor and the FPGA need to synchronize these clock signals to ensure that the data is received and processed correctly.

[0056] S3.FFMPEG video codec

[0057] With the help of FFMPEG library, ARM processors can efficiently perform encoding and decoding operations on video data. This supports the input and output of multiple video formats and meets the multimedia processing requirements in different application scenarios.

[0058] (1) Input source

[0059] Use the FPGA front-end real-time camera data as input source.

[0060] (2) Coding

[0061] Select the appropriate codec based on the requirements of the target streaming platform. Common video codecs include H.264 (libx264) and H.265 (libx265). Set encoding parameters such as bit rate, frame rate, resolution, etc. to ensure the quality and stability of the stream.

[0062] (3) Packaging

[0063] Select a container format, usually one that complies with streaming standards, such as FLV or MP4. Configure streaming parameters, set appropriate streaming parameters to ensure that data can be properly processed by the streaming server.

[0064] (4) Push Stream

[0065] Select the protocol and confirm the streaming protocol. Common ones include RTMP and HLS. RTMP is the most commonly used streaming protocol. Configure the streaming URL. Configure the correct streaming URL, which usually includes the protocol, server address, and stream name.

[0066] S4.Serial port data transmission and reception

[0067] The built-in serial communication function of the ARM processor enables the system to exchange data with other external devices. This not only enhances the scalability of the system, but also facilitates remote control and data acquisition. The following is a detailed description of serial data transmission and reception, taking the 422 serial port as an example.

[0068] ARM processors send and receive data through their built-in serial ports. This usually involves sending data to external devices through the serial port, or receiving data from external devices and processing it. In order to improve the efficiency and response speed of data transmission and reception, serial communication can be implemented through multithreading. Multithreading can process multiple serial port data streams in parallel, thereby optimizing system performance and resource utilization.

[0069] (1) Sending data

[0070] When using the RS-422 serial port to send data, the processor converts the data into a differential signal and transmits it through the serial port interface. By configuring the serial port's baud rate, data bits, stop bits and other parameters, accurate data transmission is ensured.

[0071] (2) Receiving data

[0072] When receiving data, the processor obtains the differential signal from the RS-422 serial port interface and converts it into raw data. The processor needs to be able to correctly parse and process this data to achieve effective data transmission.

[0073] In this specification, each embodiment is described in a progressive, parallel, or progressive and parallel manner. Each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0074] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that an article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such article or device. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the article or device including the above elements.

[0075] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A smart camera design method based on ARM and FPGA architecture, characterized by: The steps include: S1. AI algorithm deployment steps: The AI ​​algorithm deployment process includes model conversion, compilation and deployment, testing and debugging, and integration and deployment; S2 high-speed data transmission step: according to the preset data format and the corresponding protocol, establish a transmission link, through the MIPI interface, to achieve high-speed data transmission between the ARM processor and the FPGA; the high-speed data transmission adopts the MIPI CSI-2 protocol or the display serial interface protocol (MIPIDSI); S3.FFMPEG video encoding and decoding steps: The ARM processor uses the FFMPEG library to encode and decode the video data; S4. Serial port data transmission and reception step: The ARM processor transmits and receives data through its built-in serial port interface, and the serial port communication of the serial port interface can be implemented through multi-threading.

2. The design method according to claim 1, characterized in that: In step S1, the model conversion further includes: building a self-developed algorithm model, using Rockchip's RKNN-Toolkit2 tool to convert the self-developed algorithm model into the RKNN format, the conversion includes first converting the self-developed algorithm model from the source format to the ONNX format, and then using the RKNN-Toolkit2 tool to further convert it into the RKNN format, the compilation and deployment further includes installing related libraries and runtime environments, configuring a cross-compilation tool chain, and compiling libraries and frameworks that support target detection, and compiling and deploying on the RK series platform; after completion, uploading the converted RKNN model to the development board, and running and testing it on the target platform; The testing and debugging further include performance and memory evaluation of the model to ensure that its performance on the development board meets application requirements; verifying the actual effect of the model through system testing to ensure the stability and reliability of the model in the production environment; the integration and deployment further include integrating the tested algorithm model into actual applications, including integrating the model and related codes into the target application to ensure that the application can correctly call the model for reasoning and process the results; finally, a comprehensive system test is performed to verify the effect of the integration.

3. The design method according to claim 1, characterized in that: In step S2, the preset data format includes an image data format and a data packet format, wherein for the image data format, the FPGA processor encodes the image data in a corresponding format and then sends it to the ARM through the MIPI CSI-2 interface; for the data packet format, the data transmitted by the MIPI CSI-2 interface is usually sent in the form of data packets, each of which contains data and related control information; The establishment of the transmission link further includes: data link configuration and clock signal configuration; wherein the data link configuration includes configuring the MIPI CSI-2 interface to include multiple data links, and data is transmitted in serial mode through the multiple data links; the FPGA processor sends image data through MIPI CSI-2, and the ARM receives and processes the data; the clock signal configuration includes: a clock signal line for synchronizing data transmission; the ARM processor and the FPGA synchronize the clock signal according to the clock signal line to ensure the correct reception and processing of the data.

4. The design method according to claim 3, characterized in that: The corresponding format includes a YUV format or an RGB format.

5. The design method according to claim 1, characterized in that: In step S3, the ARM processor uses the FFMPEG library to encode and decode the video data, further comprising: 1) Input source step: use the real-time camera data of the FPGA front end as the input source; 2) Encoding step: Select the appropriate video codec and set encoding parameters according to the requirements of the target streaming platform; 3) Packaging step: select a container format, usually a container format that complies with streaming media standards, such as FLV or MP4; configure stream parameters, and set appropriate stream parameters to ensure that data can be correctly processed by the streaming media server; 4) Push streaming step: select the protocol and determine the streaming protocol, which includes RTMP or HLS; configure the correct streaming URL, including the protocol, server address and stream name.

6. The design method according to claim 5, characterized in that: The video codecs include H.264 (libx264) and H.265 (libx265), and the encoding parameters include bit rate, frame rate and resolution.

7. The design method according to claim 1, characterized in that: In step S4, the serial communication includes: 1) Sending data: When sending data, first configure the serial port parameters, then use the RS-422 serial port to send data. The processor converts the data into a differential signal and transmits it through the serial port interface; 2) Receiving data: When receiving data, the processor obtains the differential signal from the RS-422 serial port interface and converts it into raw data.

8. The design method according to claim 7, characterized in that: The serial port parameters include: baud rate, data bits and stop bits.

Images