A real-time target detection method and system for RISC-V architecture

Abstract

The present invention provides a real-time target detection method and system for RISC-V architecture, which relates to the field of embedded systems. The environmental dependencies, CNToolkit and firmware of Cambrian CNStream data stream processing SDK are sequentially transplanted into the RISC-V architecture; Cambrian CNStream data stream processing SDK is transplanted; offline model files are loaded; embedded devices and MLU 220M.2 are initialized; source data is collected and streamed into Cambrian CNStream data stream processing SDK using RTSP; Cambrian CNStream data stream processing SDK processes the source data to realize target detection function; target detection results are displayed by streaming to VLC media player via RTSP. The present invention realizes real-time target detection function for RISC-V architecture by using Sifive Unmatched embedded device and domestic smart chip Cambrian edge artificial intelligence acceleration card MLU 220M.2, and Cambrian CNStream data stream processing SDK.

Description

Technical Field

[0001] The present invention relates to the field of embedded systems, and in particular to the field of embedded AI applications. Specifically, a real-time target detection method and system for a RISC-V architecture are proposed. Background Art

[0002] An instruction set is a collection of instructions used by a CPU to perform computations and control a computer system. Based on the complexity of the instruction set, it can be categorized as either a complex instruction set computer (CISC) or a reduced instruction set computer (RISC). The RISC-V architecture is an open instruction set architecture (ISA) based on the principles of a reduced instruction set computer (RISC-V). Before 2010, the mainstream CPU architectures were x86 and ARM. With the emergence of RISC-V, compared to x86 and ARM, this architecture boasts a smaller architecture, support for modular, configurable instruction subsets, support for extensible instructions, a smaller number of instructions, and reduced hardware implementation complexity, making it the third most popular CPU architecture.

[0003] In addition to the aforementioned advantages, RISC-V also features Physical Memory Protection (PMP) and Multi-domain Trusted Execution Environment (MTE) security technologies. Introducing artificial intelligence (AI) into edge systems (IoT devices) based on the RISC-V architecture can reduce power consumption, shorten response times, and improve the security of data storage and processing compared to AI applications in cloud systems. Consequently, RISC-V is primarily used in edge computing devices, with typical applications including the Kendryte K210 chip released by Canaan and the Sophon BM1880 chip released by BITMAIN.

[0004] The use of RISC-V in edge device AI can improve the PPA (performance, power consumption, and area) and flexibility of edge computing chips. However, its support for real-time object detection is insufficient. Summary of the Invention

[0005] The technical problem to be solved by this invention is how to implement a real-time target detection function by running a deep learning-based real-time target detection algorithm on the RISC-V architecture. To solve this technical problem, the present invention provides a real-time target detection method and system for the RISC-V architecture. This method uses a deep learning-based target detection method, connects to an IP camera, and is equipped with a Sifive Unmatched embedded device and a domestically produced Cambrian edge AI accelerator card, the MLU 220M.2. The system outputs the position of the detected object in the video stream and classifies it to obtain a label with a score, thus completing the real-time target detection function.

[0006] The present invention solves the above technical problems through the following technical solutions:

[0007] A real-time target detection method for a RISC-V architecture includes the following steps:

[0008] 1) Port the environment dependencies of the Cambrian CNStream data stream processing SDK to the RISC-V architecture;

[0009] 2) Port the CNToolkit and firmware required by the Cambrian CNStream data stream processing SDK to the RISC-V architecture;

[0010] 3) Porting the Cambrian CNStream data stream processing SDK to the RISC-V architecture;

[0011] 4) Load the Cambrian offline model file yolov3.cambricon;

[0012] 5) Initialize the embedded device Sifive Unmatched development board and Cambricon edge AI accelerator card MLU220M.2;

[0013] 6) Collect source data through IP cameras and push it to the Cambrian CNStream data stream processing SDK via RTSP;

[0014] 7) Use the Cambrian CNStream data stream processing SDK to process the source data collected by the IP camera to achieve real-time target detection function;

[0015] 8) The target detection results are streamed to the VLC media player via RTSP for display.

[0016] Preferably, the environment dependencies in step 1) include: computer vision library OpenCV, command line parameter processing library gflags, Google log library glog, font format library freetype6, Chinese character library ttf-wqy-zenhei, compilation tool cmake, multimedia library sdl2, socket layer cryptography library openssl, high-performance library Librdkafka used by hardware, and streaming media transmission protocol library live555.

[0017] Preferably, the environment dependency in step 1) is transplanted to the RISC-V architecture through cross-compilation.

[0018] Preferably, the CNToolKit in step 2) includes: Cambrian hardware codec library CNCodec, Cambrian device interface library CNDev, Cambrian driver interface library CNDrv and Cambrian runtime library CNRT.

[0019] Preferably, the CNToolkit and firmware in step 2) are transplanted to the RISC-V architecture through cross-compilation.

[0020] Preferably, the Cambrian CNStream data stream processing SDK in step 3) is ported to the RISC-V architecture through cross-compilation.

[0021] Preferably, in step 4), when loading the Cambrian offline model file, the caffe format of the yolov3 model is converted to the cambricon format used by the Cambrian CNStream data stream processing SDK.

[0022] Preferably, the step of converting the caffe format to the cambricon format in step 4) includes:

[0023] Install cambricon_caffe docker;

[0024] Convert yolov3.cfg and yolov3.weights to yolov3.prototxt and yolov3.caffemodel;

[0025] Modify yolov3.prototxt as needed;

[0026] Use generate_quantized_pt to quantize yolov3.prototxt and yolov3.caffemodel;

[0027] Use the offline conversion tool caffe genoff to process the quantized yolov3 model and generate the yolov3.cambricon offline model file.

[0028] Preferably, the step of initializing the embedded device Sifive Unmatched development board in step 5) includes:

[0029] Load the boot program BootLoader;

[0030] Load the Linux kernel;

[0031] Mount the root file system rootfs;

[0032] Create an init process;

[0033] Enter the RISC-V architecture terminal.

[0034] Preferably, the step of initializing the Cambrian edge AI accelerator card MLU 220M.2 in step 5) includes:

[0035] Enter the RISC-V architecture terminal and find the path where the MLU 220M.2 driver is located;

[0036] Run the source command to make the MLU 220M.2-related dependency libraries take effect in the RISC-V architecture.

[0037] Preferably, the IP camera in step 6) is in the same network segment as the Sifive Unmatched development board with the Cambrian edge artificial intelligence acceleration card MLU 220M.2.

[0038] Preferably, in step 7), the function of implementing target detection using the Cambrian CNStream data stream processing SDK includes:

[0039] Use the built-in modules in Cambrian CNStream: DataSource module, Inferencer module, OSD module and RTSP Sink module, write the parameter configuration of the built-in modules, use the pipeline mode to connect the built-in modules in series, and build a json file;

[0040] Save the data source collected by the IP camera to the path where the files.list_video file is located and write it to the data_path parameter in the shell script;

[0041] Write the path of the josn file constructed above into the config_fname parameter in the shell script;

[0042] Write the path where the yolov3.cambricon model file is located into the MODEL_PATH parameter in the shell script;

[0043] Start the shell script to implement the target detection function of the Cambrian CNStream data stream processing SDK.

[0044] Preferably, the VLC media player in step 8) is installed in a computer with a Windows operating system, which is in the same network segment as the IP camera and the SifiveUnmatched development board with the Cambrian edge artificial intelligence accelerator card MLU 220M.2.

[0045] A real-time target detection system for a RISC-V architecture is implemented based on the above method and includes: a RISC-V architecture, an IP camera, and a VLC media player; the RISC-V architecture is constructed through steps 1) to 5) of the above method and has the data processing and target detection functions of step 7) of the above method; the IP camera is used to collect source data and stream it to the Cambrian CNStream data stream processing SDK of the RISC-V architecture via RTSP; and the VLC media player is used to display the target detection results streamed from the RISC-V architecture via RTSP.

[0046] Compared with the prior art, the present invention has the following advantages:

[0047] (1) The present invention proposes a method for achieving real-time target detection under the RISC-V architecture;

[0048] (2) Compared with the traditional method of implementing real-time target detection on a GPU platform, the present invention has the characteristics of low power consumption and small size;

[0049] (3) This invention can accelerate the implementation of AI products in the field of edge computing using the RISC-V architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a flow chart of a real-time target detection method for RISC-V architecture proposed by the present invention;

[0051] Figure 2 is a relationship diagram between software dependencies mentioned in an embodiment of the present invention;

[0052] Figure 3 This is the overall architecture diagram of a real-time target detection system for RISC-V architecture proposed in this invention. DETAILED DESCRIPTION

[0053] In order to make the above features and advantages of the present invention more clearly understood, embodiments are given below with reference to the accompanying drawings for detailed description.

[0054] like Figure 1 As shown, this embodiment provides a real-time target detection method for a RISC-V architecture, comprising the following steps:

[0055] S1: Port the environment dependencies of the Cambrian CNStream data stream processing SDK to the RISC-V architecture;

[0056] S2: Port the CNToolkit and firmware required by the Cambrian CNStream data stream processing SDK to the RISC-V architecture;

[0057] S3: Port the Cambrian CNStream data stream processing SDK to the RISC-V architecture;

[0058] S4: Load the Cambrian offline model file yolov3.cambricon;

[0059] S5: Initialize the embedded device Sifive Unmatched development board and Cambricon edge AI accelerator card MLU220M.2;

[0060] S6: Collect source data through IP cameras and push it to the Cambricon CNStream data stream processing SDK via RTSP;

[0061] S7: Cambricon CNStream data stream processing SDK processes the source data collected by IP cameras to implement target detection function;

[0062] S8: The target detection results are streamed to the VLC media player via RTSP for display.

[0063] Step S1: Porting environment dependencies. In Ubuntu20.04, x86_64-bit machines, the environment dependencies that need to be ported for the Cambrian CNStream data stream processing SDK need to be written according to the syntax pattern of the YOCTO system, and ported to the RISC-V architecture through cross-compilation for the Cambrian CNStream data stream processing SDK to call and implement corresponding functions. The environment dependencies include: computer vision library OpenCV, command line parameter processing library gflags, Google log library glog, font format library freetype6, Chinese character library ttf-wqy-zenhei, compilation tool cmake, multimedia library sdl2, socket layer cryptography library openssl, high-performance library Librdkafka for hardware use, and streaming media transmission protocol library live555. Name the above files in the form of library name_version number.bb and store them in the meta-cambricon / recipes-multimedia / path, such as Figure 2 As shown, it is used for subsequent porting and calling of the Cambrian CNStream data stream processing SDK.

[0064] Step S2: Port CNToolkit and firmware. In the system architecture of step S1, build the RISC-V 64 compilation environment, download and install the riscv64-oe-linux cross-compilation tool chain. Use riscv64-oe-linux-gcc and riscv64-oe-linux-g++ to cross-compile the CNToolKit of the Cambrian CNStream data stream processing SDK. CNToolKit includes four runtime libraries: Cambrian hardware codec library CNCodec, Cambrian device interface library CNDev, Cambrian driver interface library CNDrv and Cambrian runtime library CNRT. Compress the include and lib folders generated in the installation directory of each runtime library into a tar package and put it in the meta-cambricon / recipes-support / cambricon / files path. According to the syntax pattern of the YOCTO system, write the cambricon_git.bb recipe file to read the tar package of CNToolKit and firmware generated by cross compilation, such as Figure 2 The figure is provided for subsequent porting and calling by the Cambricon CNStream Data Stream Processing SDK. By cross-compiling and porting the CNToolkit, the Cambricon CNStream Data Stream Processing SDK can access the hardware device information of the Cambricon Edge AI Accelerator Card MLU 220M.2 by calling the API of the above library; by cross-compiling and porting the firmware, the Cambricon Edge AI Accelerator Card MLU 220M.2 can be initialized.

[0065] Step S3: Port Cambricon CNStream. In the system architecture of step S1, write the bbclass recipe file of Cambricon CNStream data stream processing SDK according to the syntax mode of the YOCTO system, generate CNStream_5.4.0.bb and place it in the meta-cambricon / recipes-cambricon / images path. Through the DEPENDS keyword, you can specify the environment dependencies, CNToolkit and firmware required by the SDK at runtime, that is, include the bbclass recipe files written in steps 1 and 2. The dependency relationship between software packages, such as Figure 2 As shown. Based on the KAS compilation tool and bitebake compilation command, write the cambricon-demo-coreip-cli.yml file and build the demo-coreip-cli system image. The compilation command is as follows:

[0066] kas build cambricon-demo-coreip-cli.yml

[0067] In this process, use IMAGE_INSTALL:append="CNStream" to cross-compile the Cambrian CNStream data stream processing SDK to the RISC-V64 operating system.

[0068] Step S4: Load the Cambricon offline model file. The Cambricon CNStream data stream processing SDK does not support model files in the Caffe format. Therefore, in the system architecture of step S1, the Caffe format of the Yolov3 model needs to be converted to the Cambricon format used by the Cambricon CNStream data stream processing SDK. The specific conversion steps are as follows:

[0069] (1) Install cambricon_caffe docker;

[0070] (2) Convert yolov3.cfg and yolov3.weights to yolov3.prototxt and yolov3.caffemodel;

[0071] (3) Modify yolov3.prototxt as needed;

[0072] (4) Use generate_quantized_pt to quantize yolov3.prototxt and yolov3.caffemodel;

[0073] (5) Use the offline conversion tool caffe genoff to process the quantized model and generate the yolov3.cambricon offline model file.

[0074] Step S5: Initialize the hardware. Copy the demo-coreip-cli-unmatched.wic.xz compiled in step S3 to the SD card. Execute the following command:

[0075] xzcat demo-coreip-cli-unmatched.wic.xz|sudo dd of= / dev / sdX bs=512Kiflag=fullblock oflag=direct conv=fsync status=progress

[0076] Use fdisk -l to view sdX and change it to the actual device file of the SD card.

[0077] Insert the SD card into the MicroSD card slot of the Sifive Unmatched development board, turn on the power, start the Sifive Unmatched development board, select the COM3 port, set the baud rate to 115200, and enter the RISC-V architecture. At this point, initialize the embedded device Sifive Unmatched development board. The specific steps are as follows:

[0078] (1) Load the boot program BootLoader;

[0079] (2) Load the Linux kernel;

[0080] (3) Mount the root file system rootfs;

[0081] (4) Create the init process;

[0082] (5) Enter the RISC-V architecture terminal.

[0083] After that, initialize the Cambricon edge AI accelerator card MLU 220M.2. The specific steps are as follows:

[0084] (1) Enter the file path where the MLU220 M.2 firmware driver is located;

[0085] (2) Execute . / load_auto to copy the firmware to the RISC-V architecture;

[0086] (3) Execute the source . / env.sh command to make the MLU 220M.2 related dependency libraries effective in the RISC-V architecture.

[0087] Step S6: Collect real-time data. Figure 3 As shown in this step, a Hikvision IP camera is used and placed on the same network segment as the Sifive Unmatched development board with the Cambricon edge AI accelerator card MLU 220M.2. The IP camera collects data and stores it in the following format: files.list_video:

[0088] rtsp: / / username:password@ <ipaddress> / <videotype> / ch <number> / <streamtype>

[0089] The data is pushed to the Cambrian CNStream data stream processing SDK via RTSP.

[0090] Step S7: Implement target detection. In this step, the steps of implementing target detection using the Cambrian CNStream data stream processing SDK are as follows:

[0091] (1) Build a json configuration. Use the DataSource module, Inferencer module, OSD (On Screen Display) module, and RTSP (Real Time Streaming Protocol) Sink module provided by Cambrian CNStream, connect them in series using the pipeline mode, write the parameter configuration of each module, form upstream and downstream dependencies, and build a json file for Cambrian CNStream to call;

[0092] (2) Load the video file. Save the data source captured by the IP camera to the path where the file is located in files.list_video and write it into the data_path parameter in the shell script;

[0093] (3) Load the josn file. Write the path of the josn file generated in (1) into the config_fname parameter in the shell script;

[0094] (4) Load the Cambricon offline model file. Write the path of the converted yolov3.cambricon model file to the MODEL_PATH parameter in the shell script;

[0095] (5) Implement target detection. Start the shell script to implement the target detection function of the Cambrian CNStream data stream processing SDK.

[0096] Step S8: Display the results. The target detection results obtained in step S7, i.e., the image frames with position frames and scores tags, are pushed to the VLC media player via RTSP for display. Figure 3 As shown, VLC media player is installed on a Windows computer. This computer must be on the same network segment as the IP camera and the Sifive Unmatched development board equipped with the Cambricon MLU220M.2 edge AI accelerator card. Open VLC media player and enter the development board's RTSP stream address to display target detection results in real time.

[0097] Based on the above method, this embodiment provides a real-time target detection system for RISC-V architecture, including: RISC-V architecture, IP camera and VLC media player, such as Figure 3 The RISC-V architecture is constructed through steps 1) to 5) and includes the data processing and target detection capabilities of step 7). The IP camera is used to collect source data and stream it to the Cambrian CNStream data stream processing SDK for the RISC-V architecture via RTSP. The VLC media player is used to display the target detection results streamed from the RISC-V architecture via RTSP.

[0098] Although the present invention has been disclosed as above by way of embodiments, they are not intended to limit the present invention. Any appropriate modification or equivalent substitution of the technical solution of the present invention by a person skilled in the art should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on that defined in the claims.< / streamtype> < / number> < / videotype> < / ipaddress>

Claims

1. A real-time target detection method for RISC-V architecture, characterized in that: The following steps are involved: 1) Port the environment dependencies of the Cambrian CNStream data stream processing SDK to the RISC-V architecture; 2) Port the CNToolkit and firmware required by the Cambrian CNStream data stream processing SDK to the RISC-V architecture; 3) Porting the Cambrian CNStream data stream processing SDK to the RISC-V architecture; 4) Load the Cambrian offline model file yolov3.cambricon; 5) Initialize the embedded device Sifive Unmatched development board and Cambricon edge AI accelerator card MLU220M.2; 6) Collect source data through IP cameras and push it to the Cambrian CNStream data stream processing SDK via RTSP; 7) Use the Cambrian CNStream data stream processing SDK to process the source data collected by the IP camera to implement the target detection function; 8) The target detection results are streamed to the VLC media player via RTSP for display.

2. The method according to claim 1, wherein The environment dependencies in step 1) include: computer vision library OpenCV, command line parameter processing library gflags, Google log library glog, font format library freetype6, Chinese character library ttf-wqy-zenhei, compilation tool cmake, multimedia library sdl2, socket layer cryptography library openssl, high-performance library Librdkafka used by hardware, and streaming media transmission protocol library live555; the environment dependencies are ported to the RISC-V architecture through cross-compilation.

3. The method according to claim 1, wherein The CNToolKit in step 2) includes: Cambrian hardware codec library CNCodec, Cambrian device interface library CNDev, Cambrian driver interface library CNDrv and Cambrian runtime library CNRT; the CNToolkit and firmware are ported to the RISC-V architecture through cross-compilation.

4. The method according to claim 1, wherein The Cambrian CNStream data stream processing SDK described in step 3) is ported to the RISC-V architecture through cross-compilation.

5. The method according to claim 1, wherein In step 4), when loading the Cambricon offline model file, convert the Caffe format of the Yolov3 model to the Cambricon format used by the Cambricon CNStream data stream processing SDK. The conversion steps include: Install cambricon_caffe docker; Convert yolov3.cfg and yolov3.weights to yolov3.prototxt and yolov3.caffemodel; Modify yolov3.prototxt as needed; Use generate_quantized_pt to quantize yolov3.prototxt and yolov3.caffemodel; Use the offline conversion tool caffe genoff to process the quantized yolov3 model and generate the yolov3.cambricon offline model file.

6. The method according to claim 1, wherein The steps for initializing the embedded device SifiveUnmatched development board in step 5) include: Load the boot program BootLoader; Load the Linux kernel; Mount the root file system rootfs; Create an init process; Enter the terminal of RISC-V architecture; The steps to initialize the Cambricon edge AI accelerator card MLU 220M.2 include: Enter the RISC-V architecture terminal and find the path where the MLU 220M.2 driver is located; Run the source command to make the MLU 220M.2-related dependency libraries take effect in the RISC-V architecture.

7. The method according to claim 1, wherein The IP camera described in step 6) is in the same network segment as the Sifive Unmatched development board with the Cambricon edge AI accelerator card MLU 220M.

2.

8. The method according to claim 1, wherein In step 7), the functions of using the Cambrian CNStream data stream processing SDK to implement target detection include: Use the built-in modules in Cambrian CNStream: DataSource module, Inferencer module, OSD module and RTSP Sink module, write the parameter configuration of the built-in modules, use the pipeline mode to connect the built-in modules in series, and build a json file; Save the data source collected by the IP camera to the path where the files.list_video file is located and write it to the data_path parameter in the shell script; Write the path of the josn file constructed above into the config_fname parameter in the shell script; Write the path where the yolov3.cambricon model file is located into the MODEL_PATH parameter in the shell script; Start the shell script to implement the target detection function of the Cambrian CNStream data stream processing SDK.

9. The method according to claim 1, wherein The VLC media player described in step 8) is installed on a computer with a Windows operating system, which is in the same network segment as the IP camera and the Sifive Unmatched development board with the Cambrian edge artificial intelligence accelerator card MLU 220M.

2.

10. A real-time target detection system for RISC-V architecture, implemented based on the method according to any one of claims 1 to 9, characterized in that: include: A RISC-V architecture, an IP camera, and a VLC media player; the RISC-V architecture is constructed through steps 1) to 5) of the method and has the data processing and target detection functions of step 7) of the method; the IP camera is used to collect source data and stream it to the Cambrian CNStream data stream processing SDK of the RISC-V architecture via RTSP; the VLC media player is used to display the target detection results streamed from the RISC-V architecture via RTSP.

Images