Real-time image recognition system and method based on PSoC chip and neural network

Abstract

This invention provides a real-time image recognition system based on a PSoC chip, comprising: a CPU processor; a network model receiving module that receives configuration information and caches it in a first memory via the CPU processor; an image preprocessing module that receives acquired image data and preprocesses it according to the configuration information; a high-speed storage control module that caches the preprocessed image data in a second memory and retrieves the configuration information from the first memory via the CPU processor and caches it in the second memory; an AI acceleration hard core that retrieves the configuration information and preprocessed image data from the second memory via the high-speed storage control module, recognizes the image data based on a convolutional neural network model corresponding to the configuration information, and caches the recognition result in the second memory; and the high-speed storage control module sends the recognition result from the second memory to the first memory. This invention also provides a real-time image recognition method.

Description

Technical Field

[0001] This invention relates to the field of intelligent image recognition technology, and in particular to a real-time image recognition system and method based on a PSoC (Programmable System-On-Chip) chip and a neural network. Background Technology

[0002] With the development of artificial intelligence technology, deep convolutional neural networks have been widely used in image recognition, speech recognition, pattern analysis, prediction and estimation, and other scenarios, successfully solving many practical problems that traditional algorithms struggle with. Their powerful feature learning capabilities and multi-degree-of-freedom network structure determine that deep convolutional neural network algorithms are computationally intensive and have a fast update and iteration speed.

[0003] Considering the power consumption limitations of general-purpose graphics processing units (GPUs), current edge computing hardware platforms for convolutional neural networks mainly consist of general-purpose processors (CPUs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs). Their basic design principles are programmability, versatility, and reconfigurability. Given the relatively low performance-to-power ratio of general-purpose CPUs and the high R&D costs and time lag of ASICs, efficiently implementing neural network algorithms has become an urgent industry need. FPGAs offer advantages such as flexible configuration, high parallelism, and high energy efficiency, making them highly suitable for industries with rapidly changing markets. Furthermore, FPGA devices can gradually incorporate modules such as DSP (Digital Signal Processor), ARM (Advanced RISC Machines), and AI (Artificial Intelligence) hard cores to form reconfigurable PSoCs (Programmed System-on-Chips) for implementing more complex network algorithms. How to combine the characteristics of PSoC chips with the parallelism and pipelining of deep convolutional neural network computation models to fully utilize limited on-chip resources for efficient collaborative deployment remains a problem to be solved. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time image recognition system and method based on a PSoC chip, which can fully utilize the computing resources of the AI ​​acceleration hard core in the PSoC chip, and combine the potential parallelism and pipelined nature of the deep convolutional neural network model calculation process, and respond with on-chip programmable logic resources, so as to carry out reasonable and efficient hardware and software deployment, and realize image recognition of deep convolutional neural networks with high performance and low power consumption.

[0005] To achieve the above objectives, the present invention provides a real-time image recognition system for neural networks based on a PSoC chip, comprising: a PSoC heterogeneous multi-core processor, a network model receiving module, a first memory, and a second memory; the PSoC heterogeneous multi-core processor comprises: a CPU processor, an AI acceleration hard core, and an FPGA module;

[0006] The network model receiving module is used to receive configuration information sent by the host computer and cache it in the first memory through the CPU processor; the configuration information includes: an instruction file, used to indicate the operation steps of functions in the convolutional neural network model program code; a first data file, which contains the weight parameters of each layer of the convolutional neural network model; a first configuration file, used to configure the AI ​​acceleration hard core; and a second configuration file, which defines the parameters for preprocessing image data.

[0007] The FPGA module includes: an image preprocessing module and a high-speed storage control module;

[0008] The image preprocessing module receives externally acquired image data and preprocesses the received image data based on the second configuration file;

[0009] The high-speed storage control module caches the preprocessed image data in the second memory; the high-speed storage control module also obtains the configuration information in the first memory through the CPU processor and caches it in the second memory;

[0010] The AI ​​acceleration hard core obtains configuration information and preprocessed image data from the second memory through the high-speed storage control module, identifies the image data based on the configuration information, and caches the identification results in the second memory; the high-speed storage control module then sends the identification results in the second memory to the CPU processor.

[0011] Optionally, the PSoC chip-based neural network real-time image recognition system further includes a control center; the control center includes: an image preprocessing controller, a high-speed storage controller, and a status monitor;

[0012] The CPU processor configures the high-speed storage control module with instructions, addresses, and data lengths for reading data from the second memory and for writing data to the second memory through the high-speed storage controller; the CPU processor also configures the high-speed storage control module with instructions, addresses, and data lengths for reading data from the first memory and for writing data to the first memory through the high-speed storage controller.

[0013] The status monitor is used to start the AI ​​acceleration hard core and read back multiple states of the AI ​​acceleration hard core, including: startup state, configuration file to be injected state, image data to be injected state, and image data recognition completion state. Based on the read-back startup state, configuration file to be injected state, and image data to be injected state, the status monitor triggers the AI ​​acceleration hard core to obtain preprocessed image data and configuration information from the second memory. Based on the read-back image data recognition completion state, the status monitor triggers the AI ​​acceleration hard core to send the recognized image data to the second memory.

[0014] The image preprocessing controller triggers the image preprocessing module to retrieve the second configuration file from the second memory and write the preprocessed image data into the second memory based on the startup state. Based on the startup state and the image data recognition completion state, the image preprocessing module is triggered to receive externally acquired image data.

[0015] Optionally, the AI ​​acceleration hard core contains multiple status registers; the status monitor reads back the status corresponding to the AI ​​acceleration hard core through the status registers.

[0016] Optionally, the real-time image recognition system based on the PSoC chip also includes off-chip memory, which is non-volatile memory; the CPU processor also writes the configuration information to the off-chip memory; the first and second memories are volatile memories; when the image recognition system is powered on, the CPU processor caches the configuration information in the off-chip memory to the first and second memories.

[0017] Optionally, after the image recognition system is powered on, the status monitor also compares the configuration information in the first memory and the off-chip memory to determine whether there is an error in loading the configuration information into the first memory.

[0018] Optional, the high-speed storage control module includes:

[0019] The first memory access module is communicatively connected to the CPU processor and is used to obtain the instructions, address, and data length configured by the CPU processor to read data from the first memory, as well as the instructions, address, and data length to write data to the first memory.

[0020] The second memory access module is communicatively connected to the CPU processor and is used to obtain the instructions, addresses, and data lengths configured by the CPU processor for reading data from the second memory, as well as the instructions, addresses, and data lengths for writing data to the second memory.

[0021] The data interaction module includes: a first memory data cache module, a second memory data cache module, and an interaction state machine; the first memory data cache module and the second memory data cache module are respectively used to cache data read from the first memory and the second memory; the interaction state machine is used to exchange data read from the first memory data cache module and the second memory data cache module.

[0022] The second memory read / write controller is used to read data from the second memory and cache it in the second memory data cache module according to the instructions, address and data length configured by the CPU processor to read data from the second memory; the second memory read / write controller is also used to write the exchanged data in the second memory data cache module into the second memory according to the instructions, address and data length configured by the CPU processor to write data to the second memory.

[0023] Optionally, the PSoC chip-based neural network real-time image recognition system further includes: a first memory read / write controller, which is communicatively connected to the first memory access module and the data interaction module;

[0024] The first memory read / write controller is used to read data from the first memory and cache it in the first memory data cache module according to the instruction, address and data length configured by the CPU processor to read data from the first memory; the first memory read / write controller is also used to write the exchanged data in the first memory data cache module into the first memory according to the instruction, address and data length configured by the CPU processor to write data to the first memory.

[0025] Optionally, the CPU processor can also frame the identified image information in the image data, convert the framed image data into HDMI format, and send it to the display for visualization.

[0026] This invention also provides a real-time image recognition method, implemented using a neural network real-time image recognition system based on a PSoC chip as described in this invention, comprising the following steps:

[0027] S1. The network model receiving module receives the configuration information sent by the host computer and caches it in the first memory through the CPU processor; the high-speed storage control module retrieves the configuration information in the first memory and caches it in the second memory.

[0028] S2. The image preprocessing module receives externally acquired image data, preprocesses the acquired image data based on configuration information, and caches the preprocessed image data in the second memory.

[0029] S3. The control center activates the AI ​​acceleration hard core, triggering it to retrieve configuration information and image data from the second memory. Based on the retrieved configuration information, the AI ​​acceleration hard core uses a deep convolutional neural network to recognize the image data and stores the recognized image data in the second memory.

[0030] S4. The high-speed storage control module sends the identified image data in the second memory to the first memory; the CPU frames the image information in the identified image data in the first memory and converts it into HDMI format for display on the monitor.

[0031] Optionally, in step S1, the CPU processor configures the high-speed storage control module with instructions, addresses, and data lengths for reading data from the first memory and writing data to the second memory through the control center; and triggers the high-speed storage control module to read configuration information from the first memory and write it to the second memory through the control center.

[0032] In step S2, when the control center reads that the AI-accelerated hard core has completed the recognition of one frame of image data, the new module in the control center triggers the image preprocessing module to receive a new frame of image acquired from the outside.

[0033] In data step S4, the control center configures the high-speed storage control module with instructions, addresses, and data lengths for reading data from the second memory and writing data to the first memory based on the CPU processor, triggering the high-speed storage control module to read the identified image data from the second memory and write it to the first memory.

[0034] Compared with the prior art, the beneficial effects of the PSoC chip-based neural network real-time image recognition system and method of the present invention are as follows:

[0035] 1) The recognition system of this invention injects configuration information corresponding to the deep convolutional neural network algorithm and uses an AI-accelerated hard core to recognize image data based on this configuration information, thus achieving a real-time embedded implementation of the deep convolutional neural network algorithm without being limited by the type of network algorithm. In this invention, the utilization rate of the AI-accelerated hard core exceeds 80%, with an average computing power of 1.6 TOPS. Compared to traditional image data recognition using a CPU processor, this invention significantly improves the speed of intelligent image recognition and reduces computational costs.

[0036] 2) The control logic of the recognition system of this invention is implemented by the control center based on the real-time readback of the AI-accelerated hard core status. The control center also acts as the scheduling center. The control center initiates the AI-accelerated hard core, triggers the image preprocessing module to receive a new frame of image data, triggers the AI-accelerated hard core to read image data and configuration information from the second memory, triggers the high-speed storage control module to read the configuration information from the first memory and cache it in the second memory, and triggers the high-speed storage control module to read the recognized image data from the second memory and cache it in the first memory. The control logic of this invention is highly efficient. Since the control logic does not need to be implemented by the CPU processor, the burden on the CPU processor is greatly reduced, and the data processing speed is improved.

[0037] 3) In this invention, the instructions, starting address, and data length for reading / writing the first storage module and the second storage module are all configured by the CPU processor. The high-speed storage control module communicates with the first and second storage modules through different interfaces and realizes data exchange between the first and second storage modules within the high-speed storage control module, which greatly improves the data transmission efficiency within the system.

[0038] 4) Furthermore, the present invention has instruction extensibility, and additional computing units can be quickly developed through the CPU processor to assist in calculations in order to support new deep convolutional neural networks. Attached Figure Description

[0039] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description will be briefly introduced below. Obviously, the drawings in the following description are one embodiment of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort:

[0040] Figure 1 This is a schematic diagram of the structure of the real-time image recognition system based on a PSoC chip according to the present invention.

[0041] Figure 1A This is a schematic diagram of the structure of the PSoC heterogeneous multi-core processor in an embodiment of the present invention;

[0042] Figure 2 This is a flowchart of the compilation process of the Icraft compiler in an embodiment of the present invention;

[0043] Figure 3 This is a schematic diagram of the address space partitioning of the off-chip memory in an embodiment of the present invention;

[0044] Figure 4 This is a schematic diagram of the control center in an embodiment of the present invention;

[0045] Figure 5This is a schematic diagram of the high-speed storage control module in an embodiment of the present invention;

[0046] Figure 6 In this embodiment of the invention, the data in the first memory is written to the AXI4-HP interface in 8 parts to obtain the timing diagram;

[0047] Figure 7 This is a timing diagram of the second memory reading the AXI4-HP interface in an embodiment of the present invention;

[0048] Figure 8 , Figure 9 This is a flowchart of the real-time image recognition method based on a deep convolutional neural network using a PSoC chip according to the present invention.

[0049] Figure 10 This is a schematic diagram of the image data after the frame has been added. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0052] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0053] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0054] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0055] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0056] This invention provides a real-time image recognition system based on a PSoC chip and a neural network, such as... Figure 1 As shown, it includes: a PSoC heterogeneous multi-core processor, a network model receiving module 3, a first memory PS-DDR (volatile memory), a second memory PL-DDR (volatile memory), and off-chip memory 9 (non-volatile memory). The PSoC heterogeneous multi-core processor includes: a CPU processor 2, an AI acceleration hard core, and an FPGA module. The FPGA module includes: an image preprocessing module 5, a high-speed storage control module 4, and a control center 6. The CPU processor and the network model receiving module 3 serve as the PS end of the recognition system of this invention, and the FPGA module serves as the PL end.

[0057] like Figure 1 , Figure 1A As shown, the PS (Processing System) side of the PSoC heterogeneous multi-core processor mainly includes a CPU processor 2, on-chip RAM (OCM), a first memory read / write controller PS-DDR-Ctrl, a DMA controller, custom timers, a watchdog timer (WDT) central interconnect bus, and peripheral interfaces (CAN / Ethernet / serial port). The PL (Programmable Logic) side mainly includes an FPGA module and a high-speed GTH interface, which are developed by the user. The PS side and PL side mainly communicate and interconnect through the AXI bus. The PSoC heterogeneous multi-core processor in this application adds an AI acceleration hard core (AI Engine) to the PL side, and also uses a 4-core high-performance CPU processor 2 (this is only an example and not a limitation of the present invention).

[0058] The network model receiving module 3 is used to receive configuration information sent by the host computer 1 and cache it in the first memory PS-DDR through the CPU processor. For example... Figure 1 As shown, the host computer 1 and the network model receiving module 3 communicate via an Ethernet interface.

[0059] In this embodiment, the configuration information is obtained by parsing the network structure description file and network parameter file of the trained convolutional neural network model using the Icraft compiler. The description file defines the network structure of the convolutional neural network model and includes the storage location of the network structure description file, the algorithm name of the convolutional neural network model, and the pixel format injected into the convolutional neural network model. The network parameter file contains the weight parameters of each layer in the convolutional neural network model.

[0060] The configuration information includes configuration files and data files:

[0061] Data files: instr.bin, weight.bin, const.bin;

[0062] Configuration files: icore.h, icore_pre.h.

[0063] `instr.bin` is the instruction file, used to instruct the function operation steps in the convolutional neural network model program code. `weight.bin` is the first data file, containing the weight parameters of each layer of the convolutional neural network model. `const.bin` is the second data file, containing the constant parameters in the program code. `icore.h` is the first configuration file, used to configure the AI-accelerated hard core 7. `icore_pre.h` is the second configuration file, defining the size, data format, and pixel format of each frame of image data.

[0064] Figure 2 The compilation process of the Icraft compiler is shown. This is prior art and will not be described in detail here. In one embodiment of the present invention, as... Figure 1 As shown, the Yolov2 algorithm model under the DarkNet framework, with RGB three-channel 8-bit pixel format, is used. The intermediate files Model_parsed.jason and Model_parsed.raw are generated through parsing. After operator optimization, INT8 quantization, and adaptation to the AI-accelerated hard-core 7 bare-metal mode, the final executable data files and configuration files for AI-accelerated hard-core 7 are generated: Model_instr.bin, Model_weight.bin, Model_const.bin, icore.h, and icore_pre.h. In another embodiment, the configuration information in this invention can also be obtained by compiling the prototxt and caffemodel files under the Caffe framework, or the pt files generated under the PyTorch framework.

[0065] Figure 2The .ini file of the Icraft compiler input mainly describes the basic information of the deep convolutional neural network model, such as: network model name yolov3, algorithm framework Darknet, the storage path of the file describing the network model on host computer 1: xxx / yolov3.cfg, and the storage path of the weight parameter file: xxx / yolov3.weights.

[0066] In this invention, the configuration information is also written to the off-chip memory 9 by the CPU processor. When the image recognition system of this invention is powered on, the configuration information in the off-chip memory 9 is loaded by the CPU processor 2 and cached in the first memory PS-DDR. Figure 3 The diagram illustrates the address space partitioning of the off-chip memory 9. The configuration file and data files occupy 16MB of the linearly accessible space and 96MB of the extended space in the off-chip memory 9. That is to say, as long as the configuration information remains unchanged, the image recognition system of this invention only needs to be injected with the configuration information once via the host computer 1.

[0067] The high-speed storage control module works in conjunction with the CPU processor 2 to read configuration information from the first memory PS-DDR and cache it in the second memory PL-DDR, and to read the recognized image data from the second memory PL-DDR and cache it in the first memory PS-DDR. How the recognized image data is generated and stored in the second memory PL-DDR will be described later. The following explains the data interaction principle between the first memory PS-DDR and the second memory PL-DDR.

[0068] like Figure 4 As shown, the control center 6 includes: a high-speed storage controller 63, an image preprocessing controller 61, and a status monitor 62.

[0069] CPU processor 2, through the high-speed storage controller 63, configures the high-speed storage control module 4 with instructions, addresses, and data lengths for reading data from the second memory PL-DDR, and instructions, addresses, and data lengths for writing data to the second memory PL-DDR. CPU processor 2 also configures the high-speed storage control module 4 with instructions, addresses, and data lengths for reading data from the first memory PS-DDR, and instructions, addresses, and data lengths for writing data to the first memory PS-DDR, through the high-speed storage controller 63. It should be noted that although CPU processor 2 issues read and write instructions, the timing of execution is determined (triggered) by control center 6.

[0070] like Figure 5As shown, the high-speed storage control module 4 includes: a first memory access module DDR_DMA, a second memory access module HPX_DMA, a data interaction module Change-DMA, and a second memory read / write controller PL-DDR-Ctrl.

[0071] The first memory PS-DDR access module is communicatively connected to the CPU processor and is used to obtain the instructions, address, and data length configured by the CPU processor to read data from the first memory PS-DDR, as well as the instructions, address, and data length to write data to the first memory PS-DDR.

[0072] The second memory access module HPX_DMA is connected to the CPU processor and is used to obtain the instructions, address, and data length configured by the CPU processor to read data from the second memory PL-DDR, as well as the instructions, address, and data length to write data to the second memory PL-DDR.

[0073] The Change-DMA data interaction module includes: a first memory data cache module h1, a second memory data cache module h2, and an interaction state machine hpx-ddr-itch. The first memory data cache module h1 and the second memory data cache module h2 are used to cache data read from the first memory PS-DDR and the second memory PL-DDR, respectively. The interaction state machine hpx-ddr-itch is used to write data from the first memory data cache module h1 to the second memory data cache module h2, or to write data from the second memory data cache module h2 to the first memory data cache module h1.

[0074] The second memory read / write controller PL-DDR-Ctrl is communicatively connected to the second memory access module HPX_DMA, the second memory PL-DDR, and the second memory data cache module h2. PL-DDR-Ctrl reads data from the second memory PL-DDR according to the CPU processor's configured instructions, address, and data length, and caches it in the second memory data cache module h2. PL-DDR-Ctrl also writes data from the second memory data cache module h2 into the second memory PL-DDR according to the CPU processor's configured instructions, address, and data length.

[0075] The first memory read / write controller PS-DDR-Ctrl communicates with the first memory access module DDR_DMA and the first memory data cache module h1 via the AXI4-HP interface on the high-speed storage control module 4.

[0076] The first memory read / write controller PS-DDR-Ctrl reads data from the first memory PS-DDR according to the instructions, address, and data length configured by the CPU processor, and caches the data in the first memory data cache module h1. The first memory read / write controller PS-DDR-Ctrl also writes data from the first memory data cache module h1 into the first memory PS-DDR according to the instructions, address, and data length configured by the CPU processor.

[0077] In this embodiment, when the second memory read / write controller PL-DDR-Ctrl reads 512 bits of data from the second memory PL-DDR and caches it in the second memory data cache module h2, it will automatically trigger the first memory PS-DDR read / write controller to write data to the first memory PS-DDR in 8 separate transactions via the AXI4-HP interface. After the first memory PS-DDR read / write controller has cumulatively read data from the first memory PS-DDR 8 times (64 bits each time) via the AXI4-HP interface, it will trigger the interactive state machine hpx-ddr-itch to write data to the second memory PL-DDR via the second memory read / write controller PL-DDR-Ctrl.

[0078] Figure 6 The middle part refers to the timing diagram of writing data in the second memory PL-DDR (with large bandwidth) to the AXI-HP interface (i.e., PS-DDR) in 8 parts; Figure 7 The timing diagram for reading data from the AXI-HP interface (i.e., PS-DDR) in the second memory PL-DDR is as follows: data in the first memory PS-DDR (with small bandwidth) is packaged into a frame by 8 consecutive clock cycles and sent to the second memory PL-DDR.

[0079] The second memory read / write controller PL-DDR-Ctrl is also used to realize data read / write between the AI ​​acceleration hard core 7 and the second memory PL-DDR, and between the image preprocessing module 5 and the second memory PL-DDR.

[0080] The AI ​​acceleration hard core 7 contains multiple status registers, and the status monitor 62 reads back multiple states of the AI ​​acceleration hard core 7 through these status registers. These multiple states include: startup state, configuration file to be injected state, image data to be injected state, and image data recognition completed state.

[0081] After power-on, if the status monitor 62 determines that the AI ​​acceleration hard core 7 has not started based on the read-back startup status, the control center 6 sends a startup command to the AI ​​acceleration hard core 7.

[0082] When the status monitor 62 reads back that the AI-accelerated hard core 7 has been activated, the image preprocessing controller 61 triggers the image preprocessing module 5 to receive image data acquired by the external optical sensor 8, and triggers the image preprocessing module 5 to retrieve the second configuration file icore_pre.h from the second memory PL-DDR (via the second memory read / write controller PL-DDR-Ctrl). The image preprocessing module 5 preprocesses the acquired image data based on the second configuration file and writes the preprocessed image data (via the first memory read / write controller PS-DDR-Ctrl) to the second memory PL-DDR. It should be noted that the optical sensor 8 continuously acquires image data, while the image preprocessing module 5 only receives a frame of image data when triggered.

[0083] In this embodiment, the preprocessing of image data mainly includes two steps: noise reduction and data rearrangement.

[0084] In this embodiment, the optical sensor 8 (e.g., a camera) acquires 10-bit single-channel image data. This data is padded with six zeros to form {6 bits 0, 10 bits pixel} 16-bit data, which is then sent to the image preprocessing module 5 at the PL end (using a high-speed serial port for reception) via the GTP interface. The 10-bit data (0-1023) is quantized into 8-bit data (0-255) through a linear transformation (÷4). Then, the single-channel 8-bit pixel is copied three times to form an RGB888 image format (8 bits for the R channel, 8 bits for the G channel, and 8 bits for the B channel), resulting in 24-bit image data (i.e., the RGB channels are the same), which is then used for computation by the AI ​​acceleration hard core 7. Using other optical lenses also requires conversion to RGB888 format before computation can be performed. The image preprocessing module 5 also performs zero-padding and other processing on the 24-bit RGB888 image data. These steps complete the data rearrangement to meet the requirements of the AI ​​acceleration hard core 7.

[0085] If the status monitor 62 reads back that the AI ​​acceleration hard core 7 has been started, and the status of the configuration file to be injected and the status of the image data to be injected are both "not injected", the control center 6 triggers the AI ​​acceleration hard core 7 to obtain the preprocessed image data and configuration information from the second memory PL-DDR through the first memory read / write controller PS-DDR-Ctrl. Based on the configuration information, the AI ​​acceleration hard core 7 uses a deep convolutional neural network algorithm to identify the image data.

[0086] If the status monitor 62 reads back that the AI ​​acceleration hard core 7 has completed image data recognition, it triggers the AI ​​acceleration hard core 7 to send the recognized image data to the second memory PL-DDR through the first memory read / write controller PS-DDR-Ctrl, and triggers the image preprocessing module 5 to receive the next frame of new image data through the image preprocessing controller 61.

[0087] When the status monitor 62 reads back that the AI-accelerated hard core 7 has completed image data recognition, it also triggers the high-speed storage control module 4 to send the recognition results in the second memory PL-DDR to the first memory PS-DDR.

[0088] In another embodiment, after the image recognition system of the present invention is powered on, the status monitor 62 also compares the configuration information in the first memory PS-DDR and the off-chip memory 9 to determine whether there is an error in loading the configuration information in the first memory PS-DDR.

[0089] In this embodiment, the CPU processor 2 further performs frame-adding processing on the identified image information in the image data, and converts the frame-adding image data into HDMI format and sends it to the display for visualization.

[0090] This invention also provides a real-time image recognition method, such as... Figure 8 , Figure 9 As shown, the real-time image recognition system based on a PSoC chip, as described in this invention, includes the following steps:

[0091] S1. The network model receiving module 3 receives the configuration information sent by the host computer 1 and caches it in the first memory PS-DDR through the CPU processor 2; the high-speed storage control module 4 obtains the configuration information in the first memory PS-DDR and caches it in the second memory PL-DDR.

[0092] In step S1, the CPU processor 2 configures the high-speed storage control module 4 with instructions, addresses, and data lengths for reading data from the first memory PS-DDR and writing data to the second memory PL-DDR through the control center 6; and triggers the high-speed storage control module 4 to read configuration information from the first memory PS-DDR and write it to the second memory PL-DDR through the control center 6.

[0093] S2. The image preprocessing module 5 receives externally acquired image data, preprocesses the acquired image data based on the configuration information, and caches the preprocessed image data in the second memory PL-DDR.

[0094] In step S2, when the control center 6 reads that the AI ​​acceleration hard core 7 has completed the recognition of one frame of image data, the new module in the control center triggers the image preprocessing module 5 to receive a new frame of image acquired from the outside.

[0095] S3, Control Center 6 starts AI Acceleration Hard Core 7, triggering AI Acceleration Hard Core 7 to obtain configuration information and image data from the second memory PL-DDR; Based on the obtained configuration information, AI Acceleration Hard Core 7 recognizes the image data through a deep convolutional neural network and stores the recognized image data in the second memory PL-DDR;

[0096] S4. The high-speed storage control module 4 sends the identified image data in the second memory PL-DDR to the first memory PS-DDR; the CPU processor 2 frames the image information in the identified image data in the first memory PS-DDR (e.g., ...). Figure 10 (as shown), and convert it to HDMI format for display on a monitor.

[0097] In another embodiment, prior to the bounding box operation, some non-linear post-processing operations are performed in CPU processor 2. These non-linear algorithms have low computational complexity but are not suitable for processing in FPGA logic. After performing these post-processing operations, the image data recognition is completed.

[0098] In data step S4, the control center 6 configures the high-speed storage control module 4 with instructions, addresses, and data lengths for reading data from the second memory PL-DDR and writing data to the first memory PS-DDR based on the CPU processor, thereby triggering the high-speed storage control module 4 to read the identified image data from the second memory PL-DDR and write it to the first memory PS-DDR.

[0099] The recognition system of this invention injects configuration information corresponding to the deep convolutional neural network algorithm and uses the AI-accelerated hard core 7 to recognize image data based on this configuration information, thus achieving a real-time embedded implementation of the deep convolutional neural network algorithm without being limited by the type of network algorithm. In this invention, the utilization rate of the AI-accelerated hard core 7 exceeds 80%, with an average computing power of 1.6 TOPS. Compared to the traditional method of recognizing image data using the CPU processor 2, this invention significantly improves the speed of intelligent image recognition and reduces computational costs.

[0100] The control logic of the recognition system of this invention is implemented by the control center 6 based on the state of the AI ​​acceleration hard core 7 in real-time readback. The control center 6 starts the AI ​​acceleration hard core 7, triggers the image preprocessing module 5 to receive a new frame of image data, triggers the AI ​​acceleration hard core 7 to read image data and configuration information from the second memory PL-DDR, triggers the high-speed storage control module 4 to read the configuration information of the first memory PS-DDR and cache it in the second memory PL-DDR, and triggers the high-speed storage control module 4 to read the recognized image data in the second memory PL-DDR and cache it in the first memory PS-DDR. The control logic of this invention is highly efficient. Since the control logic does not need to be implemented by the CPU processor 2, the burden on the CPU processor 2 is greatly reduced and the data processing speed is improved.

[0101] In this invention, the instructions, starting address, and data length for reading / writing the first and second storage modules are all configured by the CPU processor. The high-speed storage control module 4 communicates with the first and second PL-DDR memories through different interfaces and realizes data exchange between the first and second storage modules within the high-speed storage control module 4, which greatly improves the data transmission efficiency within the system.

[0102] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0103] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time image recognition system based on a PSoC chip and a neural network, characterized in that, It includes: a PSoC heterogeneous multi-core processor, a network model receiving module, a first memory, a second memory, and a control center; the PSoC heterogeneous multi-core processor includes: a CPU processor, an AI acceleration hard core, and an FPGA module; The network model receiving module is used to receive configuration information sent by the host computer and cache it in the first memory through the CPU processor; the configuration information includes: an instruction file, used to indicate the operation steps of functions in the convolutional neural network model program code; a first data file, which contains the weight parameters of each layer of the convolutional neural network model; a first configuration file, used to configure the AI ​​acceleration hard core; and a second configuration file, which defines the parameters for preprocessing image data. The FPGA module includes: an image preprocessing module and a high-speed storage control module; The image preprocessing module receives externally acquired image data and preprocesses the received image data based on the second configuration file; The high-speed storage control module caches the preprocessed image data in the second memory; the high-speed storage control module also obtains the configuration information in the first memory through the CPU processor and caches it in the second memory; The AI ​​acceleration hard core obtains configuration information and preprocessed image data from the second memory through the high-speed storage control module, identifies the image data based on the configuration information, and caches the identification results in the second memory; The high-speed storage control module sends the recognition results in the second memory to the first memory. The CPU processor retrieves the recognition results from the first memory and displays them visually. The control center includes: an image preprocessing controller and a status monitor; The status monitor is used to start the AI ​​acceleration hard core and read back multiple states of the AI ​​acceleration hard core; the multiple states include: startup state, configuration file to be injected state, image data to be injected state, and image data recognition completed state; the status monitor triggers the AI ​​acceleration hard core to obtain preprocessed image data and configuration information from the second memory based on the read-back startup state, configuration file to be injected state, and image data to be injected state, and triggers the AI ​​acceleration hard core to send the recognized image data to the second memory based on the read-back image data recognition completed state; The image preprocessing controller triggers the image preprocessing module to retrieve the second configuration file from the second memory and write the preprocessed image data into the second memory based on the startup state. Based on the startup state and the image data recognition completion state, the image preprocessing module is triggered to receive externally acquired image data.

2. The real-time image recognition system based on a PSoC chip using neural networks as described in claim 1, characterized in that, The control center also includes: a high-speed storage controller; The CPU processor configures the high-speed storage control module with instructions, addresses, and data lengths for reading data from the second memory, as well as instructions, addresses, and data lengths for writing data to the second memory, through the high-speed storage controller. The CPU processor configures the high-speed storage control module with instructions, addresses, and data lengths for reading data from the first memory, as well as instructions, addresses, and data lengths for writing data to the first memory, through the high-speed storage controller.

3. The real-time image recognition system based on a PSoC chip according to claim 2, characterized in that, The AI ​​acceleration hard core contains multiple status registers; the status monitor reads back the status corresponding to the AI ​​acceleration hard core through the status registers.

4. The real-time image recognition system based on a PSoC chip according to claim 2, characterized in that, It also includes off-chip memory, which is non-volatile memory; The CPU processor also writes the configuration information to the off-chip memory; The first and second memories are volatile memories; When the image recognition system is powered on, the CPU processor caches the configuration information in the off-chip memory into the first memory and the second memory.

5. The real-time image recognition system based on a PSoC chip according to claim 4, characterized in that, After the image recognition system is powered on, the status monitor also compares the configuration information in the first memory and the off-chip memory to determine whether there is an error in loading the configuration information into the first memory.

6. The real-time image recognition system based on a PSoC chip using neural networks as described in claim 1, characterized in that, The high-speed storage control module includes: The first memory access module is communicatively connected to the CPU processor and is used to obtain the instructions, address, and data length configured by the CPU processor to read data from the first memory, as well as the instructions, address, and data length to write data to the first memory. The second memory access module is communicatively connected to the CPU processor and is used to obtain the instructions, addresses, and data lengths configured by the CPU processor for reading data from the second memory, as well as the instructions, addresses, and data lengths for writing data to the second memory. The data interaction module includes: a first memory data cache module, a second memory data cache module, and an interaction state machine; the first memory data cache module and the second memory data cache module are respectively used to cache data read from the first memory and the second memory; the interaction state machine is used to exchange data read from the first memory data cache module and the second memory data cache module. The second memory read / write controller is used to read data from the second memory and cache it in the second memory data cache module according to the instructions, address and data length configured by the CPU processor to read data from the second memory. The second memory read / write controller also writes the exchanged data in the second memory data cache module into the second memory according to the instructions, address, and data length configured by the CPU processor to write data to the second memory.

7. The real-time image recognition system based on a PSoC chip according to claim 6, characterized in that, It also includes: a first memory read / write controller, which is communicatively connected to the first memory access module and the data interaction module; The first memory read / write controller is used to read data from the first memory and cache it in the first memory data cache module according to the instruction, address and data length configured by the CPU processor to read data from the first memory; the first memory read / write controller is also used to write the exchanged data in the first memory data cache module into the first memory according to the instruction, address and data length configured by the CPU processor to write data to the first memory.

8. The real-time image recognition system based on a PSoC chip using neural networks as described in claim 1, characterized in that, The CPU processor also frames the identified image information in the image data, converts the framed image data into HDMI format, and sends it to the display for visualization.

9. A real-time image recognition method, implemented using a PSoC chip-based neural network real-time image recognition system as described in any one of claims 1 to 8, characterized in that, Includes the following steps: S1. The network model receiving module receives the configuration information sent by the host computer and caches it in the first memory through the CPU processor; the high-speed storage control module reads the configuration information in the first memory and caches it in the second memory. S2. The image preprocessing module receives externally acquired image data, preprocesses the acquired image data based on configuration information, and caches the preprocessed image data in the second memory. S3. The control center activates the AI ​​acceleration hard core, triggering it to retrieve configuration information and image data from the second memory. Based on the retrieved configuration information, the AI ​​acceleration hard core uses a deep convolutional neural network to recognize the image data and stores the recognized image data in the second memory. S4. The recognized image data in the second memory is sent to the first memory through the high-speed storage control module; The CPU frames the image information in the recognized image data in the first memory, converts it into HDMI format, and provides it to the display for visualization.

10. The real-time image recognition method as described in claim 9, characterized in that, In step S1, the CPU processor configures the high-speed storage control module with instructions, addresses, and data lengths for reading data from the first memory and writing data to the second memory through the control center; the control center triggers the high-speed storage control module to read configuration information from the first memory and write it to the second memory. In step S2, when the control center reads that the AI ​​acceleration hard core has completed the recognition of one frame of image data, the new module in the control center triggers the image preprocessing module to receive a new frame of image data. In step S4, the control center configures the high-speed storage control module with instructions, addresses, and data lengths for reading data from the second memory and writing data to the first memory via the CPU processor, thereby triggering the high-speed storage control module to read the identified image data from the second memory and write it to the first memory.

Images