Multi-channel image processing system and processing method

By designing a multi-channel image processing system, using the CMOS acquisition module and core module combined with the model accelerator, the problem of large throughput data processing and transmission speed in multi-eye vision is solved, and the efficiency of image recognition, classification and three-dimensional reconstruction is achieved. support.

CN118941874APending Publication Date: 2024-11-12AEROSPACE NEWSKY TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411135348.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art is difficult to meet the processing speed and transmission speed requirements of large throughput data in multi-eye vision applications, and limits the application of multi-eye vision in three-dimensional reconstruction, virtual reality, traffic security and other fields.

Method used

A multi-channel image processing system was designed, using CMOS acquisition module to connect to CMOS image sensor, and image recognition, classification, matching and splicing are realized through the core module and the built-in model accelerator. Data transmission is carried out using Camera Link and Ethernet interfaces, combined with FPGA and ZYNQ chips to achieve efficient image processing.

Benefits of technology

It realizes stable and reliable multi-channel synchronous/asynchronous image acquisition and transmission, meets the processing speed of large throughput data, supports algorithms such as image recognition, classification, matching and three-dimensional reconstruction, and promotes multi-eye vision applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118941874A_ABST
    Figure CN118941874A_ABST
Patent Text Reader

Abstract

The invention relates to a multi-channel image processing system and a multi-channel image processing method. The system comprises a plurality of CMOS acquisition modules which are respectively connected with a plurality of CMOS image sensors and are used for acquiring target image data respectively corresponding to the plurality of CMOS image sensors; the core module is respectively connected with the plurality of CMOS acquisition modules and is used for receiving the target image data and obtaining a model prediction result according to the target image data and a built-in model accelerator; the model prediction result is used for indicating at least one of an image recognition result, an image classification result, an image splicing result or an image matching result for the target image data; and the upper computer is connected with the core module and is used for receiving the model prediction result transmitted by the core module. By adopting the system, the processing speed requirement and the transmission speed requirement of large-throughput data can be met.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multi-channel image processing system and a processing method. Background Art

[0002] As a common means of acquiring external scene information, cameras based on visual perception have been widely used in traffic supervision systems, real-time video conferencing, medical equipment, industrial control, aerospace and other fields. With the development of technology, multi-eye vision has been widely used in 3D reconstruction and modeling, virtual reality and augmented reality, transportation and security, human-computer interaction, intelligent robots and autonomous driving. These fields have increasingly higher requirements for the transmission rate and data throughput of image transmission systems, and there is an urgent need for an image processing system that can meet the processing speed and transmission speed of large throughput data. Summary of the invention

[0003] Based on this, it is necessary to provide a multi-channel image processing system and processing method that can meet the processing speed requirements and transmission speed requirements of large throughput data in response to the above technical problems.

[0004] In a first aspect, the present application provides a multi-channel image processing system. The system comprises:

[0005] A plurality of CMOS acquisition modules are respectively connected to the plurality of CMOS image sensors, and are used to acquire target image data respectively corresponding to the plurality of CMOS image sensors;

[0006] a core module, connected to the plurality of CMOS acquisition modules respectively, for receiving the target image data, and obtaining a model prediction result according to the target image data and a built-in model hardware accelerator, wherein the model prediction result is used to indicate at least one of an image recognition result, an image classification result, an image stitching result or an image matching result for the target image data;

[0007] The host computer is connected to the core module and is used to receive the model prediction result transmitted from the core module.

[0008] In one embodiment, the CMOS acquisition module includes:

[0009] An LVDS interface submodule, connected to the CMOS image sensor, and used to collect raw image data of the CMOS image sensor;

[0010] A field programmable gate array, connected to the LVDS interface circuit, for receiving the original image data from the LVDS interface circuit, and preprocessing the original image data to obtain initial image data;

[0011] The first Camera Link interface submodule is connected to the field programmable gate array and the core module, and is used for performing parallel-to-serial conversion on the initial image data to obtain target image data, and transmitting the target image data to the core module.

[0012] In one embodiment, when executing the step of preprocessing the raw image data to obtain the initial image data, the field programmable gate array is configured to:

[0013] Performing filtering processing on the original image data to obtain original image data after noise reduction;

[0014] The original image data after noise reduction is encoded according to the Camera Link protocol to obtain the initial image data.

[0015] In one of the embodiments, the CMOS acquisition module further includes a CMOS driving submodule;

[0016] The CMOS driving submodule is connected to the CMOS image sensor and is used to provide an operating voltage to the CMOS image sensor.

[0017] In one embodiment, the core module includes:

[0018] A second Camera Link interface submodule, connected to the CMOS acquisition module, is used to receive the target image data and perform serial-to-parallel conversion on the target image data to obtain target parallel image data;

[0019] A processing submodule is connected to the second Camera Link interface submodule, and is used to receive the target parallel image data from the second Camera Link interface submodule, and decode the target parallel image data according to the Camera Link protocol to obtain target decoded image data, and obtain the model prediction result according to the target decoded image data and the model accelerator.

[0020] In one embodiment, the core module further includes:

[0021] A Gigabit Ethernet interface communication module, connected to the processing submodule and the host computer, for receiving network model parameters and sending the model prediction results of the processing submodule to the host computer;

[0022] The 10 Gigabit Ethernet interface communication module is connected to the processing submodule and the host computer, and is used to send the target decoded image data of the processing submodule to the host computer.

[0023] In one of the embodiments, the 10 Gigabit Ethernet interface submodule includes a media access controller, a physical coding sublayer, a physical attachment sublayer, and a physical dependency sublayer;

[0024] The media access controller is used to implement UDP sending and receiving using a UDP protocol stack;

[0025] The physical coding sublayer and the physical attachment sublayer are used to implement data serialization and deserialization, CRC check and line coding;

[0026] The physical dependent sublayer is used to realize photoelectric / electro-optical conversion.

[0027] In one of the embodiments, the model accelerator includes a model parsing module, a process control module, an interrupt module, a parameter configuration module, a storage control module, a data loading module, a standard convolution module, a depth-separable convolution module, an average pooling module and a fully connected module; the interrupt module is respectively connected to the process control module, the standard convolution module, the depth-separable convolution module, the average pooling module and the fully connected module;

[0028] The model parsing module is used to read network model parameters;

[0029] The process control module is used to control the process and timing of the model accelerator;

[0030] The interrupt module is used to process interrupts generated by the process control module, the standard convolution module, the depthwise separable convolution module, the average pooling module and the fully connected module;

[0031] The parameter configuration module is used to receive the parameters of the network model read and transmitted by the model parsing module after model training, and configure and manage the parameters of the network model;

[0032] The storage control module is used to realize the caching and reading of feature map data and weights;

[0033] The data loading module is used to realize the sequential reading and writing of feature map data and weights / biases;

[0034] The standard convolution operation module is used to implement convolution operation and batch normalization operation;

[0035] The depthwise separable convolution module is used to implement depthwise convolution, pointwise convolution and batch normalization operations;

[0036] The average pooling module is used to implement the average pooling operation;

[0037] The fully connected module is used to implement classification.

[0038] In one implementation, the processing submodule uses a ZYNQ chip, and both the PS end and the PL end of the ZYNQ chip are mounted with DDR memory;

[0039] The DDR memory of the PS end is used to store configuration information and model prediction results;

[0040] The DDR memory at the PL end is used to store input feature data and weight parameters.

[0041] In a second aspect, the present application also provides a multi-channel image processing method. The method comprises:

[0042] Multiple CMOS acquisition modules acquire target image data corresponding to multiple CMOS image sensors respectively;

[0043] The core module receives the target image data, and obtains a model prediction result according to the target image data and a built-in model hardware accelerator, wherein the model prediction result is used to indicate at least one of an image recognition result, an image classification result, an image stitching result, or an image matching result for the target image data;

[0044] The host computer receives the model prediction result transmitted from the core module.

[0045] The above-mentioned multi-channel image processing system and processing method, the system includes multiple CMOS acquisition modules, which are respectively connected to multiple CMOS image sensors, and are used to obtain target image data corresponding to the multiple CMOS image sensors; core modules are respectively connected to the multiple CMOS acquisition modules, and are used to receive the target image data, and obtain model prediction results according to the target image data and the built-in model accelerator, and the model prediction results are used to indicate at least one of the image recognition results, image classification results, image stitching results or image matching results for the target image data; the host computer is connected to the core module, and is used to receive the model prediction results transmitted from the core module. The above-mentioned system can realize stable and reliable multi-channel synchronous / asynchronous image acquisition, caching and transmission, and a model hardware accelerator is built in the system to meet the processing speed of large throughput data, making it possible to implement algorithms such as image recognition, image classification, image matching and three-dimensional reconstruction at the front end, which is of great value for further promoting the application of multi-view vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a structural block diagram of a multi-channel image processing system in one embodiment;

[0047] Figure 2 is a structural block diagram of a multi-channel image processing system in another embodiment;

[0048] Figure 3 A circuit block diagram of a CMOS acquisition board in one embodiment;

[0049] Figure 4 A circuit block diagram of a core board in one embodiment;

[0050] Figure 5 A diagram of a MobileNet network model in one embodiment;

[0051] Figure 6 is a structural block diagram of a MobileNet accelerator in one embodiment;

[0052] Figure 7 It is a structural block diagram of a 3*3 convolution calculation unit in one embodiment;

[0053] Figure 8 is a structural block diagram of a ReLU6 computing unit in one embodiment;

[0054] Fig. 9 is a block diagram of a standard convolution module in one embodiment;

[0055] Fig.10 is a structural block diagram of a deep convolution operation unit in one embodiment;

[0056] Fig.11 is a structural block diagram of a point convolution operation unit in one embodiment;

[0057] Fig.12 Schematic diagram of a multi-channel image processing method in one embodiment. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0059] Current image processing systems often have only one image data channel, and each data acquisition can only be directed to a single sensor. When users need to use multiple cameras at the same time, there are many limitations and inconveniences. In addition, the transmission speed and processing speed of high-throughput data when applied to multiple cameras also face considerable challenges.

[0060] The present application provides a multi-channel image processing system based on a field programmable gate array (FPGA), which can acquire, process, cache and transmit multi-channel image data in real time, and has the potential to realize image recognition, image classification, image stitching, image matching and even three-dimensional reconstruction at the front end.

[0061] In one embodiment, Figure 1 As shown, a multi-channel image processing system is provided, which includes multiple CMOS acquisition modules, a core module and a host computer. Among them:

[0062] The multiple CMOS acquisition modules in the embodiment of the present application are respectively connected to the multiple CMOS image sensors, and the multiple CMOS acquisition modules are used to respectively acquire target image data corresponding to the multiple CMOS image sensors.

[0063] Among them, CMOS image sensor (Complementary Metal-Oxide-Semiconductor Image Sensor) is an image sensor that can be used in surveillance equipment such as video cameras, digital cameras, and cameras. CMOS image sensors can capture images by converting optical signals into electrical signals. CMOS acquisition modules are image data acquisition and image data transmission modules corresponding to CMOS image sensors. Multiple CMOS acquisition modules correspond to multiple CMOS image sensors one by one.

[0064] The core module of the embodiment of the present application is connected to multiple CMOS acquisition modules respectively. The core module is used to receive target image data from multiple CMOS acquisition modules and obtain model prediction results based on the target image data and a built-in model accelerator.

[0065] The core module is connected to multiple CMOS acquisition modules, and the core module can receive and process multiple channels of target image data in real time, and send the processed results and processed image data to the host computer. A model accelerator is built in the core module, which refers to a tool for accelerating the training and reasoning process of machine learning and deep learning models. Through the model accelerator, the deep learning network can be applied on the core module to implement at least one of image recognition processing, image classification processing, image matching processing or image splicing processing, and the corresponding model prediction results are obtained by combining the target image data and the built-in model accelerator.

[0066] Exemplarily, the model accelerator can be a MobileNet accelerator, which is a lightweight and efficient convolutional neural network architecture that uses depthwise separable convolution. This convolution method decomposes traditional convolution operations into depth convolution and point-by-point convolution, thereby greatly reducing the amount of calculation and the amount of model parameters, making the model more lightweight and efficient. By building a MobileNet accelerator on the core board, convolutional neural networks can be applied to implement algorithms such as image recognition, image classification, and image matching on the core board. For example, real-time identification of objects or scenes in target image data, classification of target objects in target image data, object detection and tracking in autonomous driving, or face recognition and identity authentication in security systems.

[0067] The host computer in the embodiment of the present application is connected to the core module, and the host computer is used to receive the model prediction results transmitted from the core module.

[0068] Among them, the host computer serves as the control center and data processing terminal of the multi-channel image processing system, which can realize the functions of data reception and display, system control and configuration, data processing and analysis, providing user interface, and data transmission and communication.

[0069] The above-mentioned multi-channel image processing system includes a plurality of CMOS acquisition modules, which are respectively connected to a plurality of CMOS image sensors, and are used to obtain target image data corresponding to the plurality of CMOS image sensors; a core module, which is respectively connected to the plurality of CMOS acquisition modules, and is used to receive the target image data, and obtain a model prediction result according to the target image data and a built-in model accelerator, and the model prediction result is used to indicate at least one of an image recognition result, an image classification result, an image stitching result or an image matching result for the target image data; and a host computer, which is connected to the core module, and is used to receive the model prediction result transmitted from the core module. The above-mentioned system can realize stable and reliable multi-channel synchronous / asynchronous image acquisition, caching and transmission, and a model hardware accelerator is built into the system to meet the processing speed of large throughput data, making it possible to realize algorithms such as image recognition, image classification, image matching and three-dimensional reconstruction at the front end, which is of great value for further promoting the application of multi-view vision.

[0070] In such Figure 1 In an optional implementation of the embodiment shown, as Figure 2As shown, the CMOS acquisition module can be a CMOS acquisition board, the core module can be a core board, and the CMOS acquisition board corresponds to the CMOS image sensor one by one. The CMOS acquisition board is used to acquire image data of the CMOS image sensor and perform filtering, noise reduction and other processing, encode the image after filtering, noise reduction and other processing according to the Camera Link protocol, and then send the data to the core board; the core board can receive data from multiple CMOS acquisition boards and decode to restore the image, and use the model accelerator to process the image, and send the model prediction results of the model accelerator and the image data restored after decoding to the host computer.

[0071] In one embodiment, the CMOS acquisition module may include a field programmable gate array, a CMOS driver submodule, an LVDS interface submodule and a first Camera Link interface submodule.

[0072] Among them, a Field-Programmable Gate Array (FPGA) is an integrated circuit (IC) whose design allows configuration and reconfiguration in the field (rather than during the production process).

[0073] Among them, the LVDS (Low-Voltage Differential Signaling) interface is a commonly used high-performance data transmission standard, which is widely used in high-speed data transmission between electronic devices. LVDS technology uses differential signals to achieve high-speed and low-power data transmission.

[0074] Among them, the Camera Link interface is a high-speed serial interface standard widely used in industrial cameras and image processing systems, and is used to connect cameras with image acquisition cards or processing systems.

[0075] The CMOS driving submodule of the embodiment of the present application is connected to the CMOS image sensor, and the CMOS driving submodule is used to provide an operating voltage to the CMOS image sensor.

[0076] The LVDS interface submodule of the embodiment of the present application is connected to the CMOS image sensor, and the LVDS interface submodule is used to collect raw image data of the CMOS image sensor.

[0077] The field programmable gate array of the embodiment of the present application is connected to the LVDS interface circuit, and is used to receive the original image data from the LVDS interface circuit, and pre-process the original image data to obtain initial image data.

[0078] Exemplarily, the field programmable gate array performs filtering processing on the original image data to obtain the original image data after noise reduction; and performs encoding processing on the original image data after noise reduction according to the Camera Link protocol to obtain the initial image data.

[0079] The first Camera Link interface submodule of the embodiment of the present application is connected to the field programmable gate array and the core module. The first Camera Link interface submodule is used to perform parallel-to-serial conversion on the initial image data to obtain target image data, and transmit the target image data to the core module.

[0080] Exemplarily, the first Camera Link interface submodule performs parallel-to-serial conversion on the initial image data, converts the multi-channel parallel low-speed data inside the FPGA into high-speed differential serial data, and sends it to the core module. The receiving port of the first Camera Link interface submodule is constructed through the IP core (Intellectual Property Core) primitive. First, the transmission clock is multiplied by 7 through PLL (Phase-Locked Loop) or MMCM (Mixed-Mode Clock Manager) to obtain the transmission clock, and then OSERDES (Output SERial DESerializer) is used for 7:1 conversion to obtain 4 7-fold differential data signals and 1 pair of 7-fold clock signals.

[0081] Please refer to Figure 3 The CMOS acquisition module is a CMOS acquisition board, and the circuit of the CMOS acquisition board includes FPGA, CMOS drive circuit, LVDS interface circuit, Camera Link interface circuit, Flash storage chip and DDR memory, etc. The CMOS image data is received in real time through the LVDS interface circuit, and the image is pre-processed, and the image data is sent to the core board through the CameraLink interface after parallel-to-serial conversion.

[0082] In one embodiment, the core module includes a second Camera Link interface submodule and a processing submodule.

[0083] The second Camera Link interface submodule is connected to the CMOS acquisition module, and is used to receive target image data and perform serial-to-parallel conversion on the target image data to obtain target parallel image data.

[0084] The processing submodule is connected to the second Camera Link interface submodule, and is used to receive the target parallel image data from the second Camera Link interface submodule, and decode the target parallel image data according to the CameraLink protocol to obtain target decoded image data, and obtain the model prediction result according to the target decoded image data and the model accelerator.

[0085] Please refer to Figure 4 The core module can be a core board, and the processing submodule specifically uses a ZYNQ chip. The ZYNQ chip combines the ARM processor and FPGA functions, and its architecture is divided into two parts: PS (Processing System) and PL (Programmable Logic). The PS part includes the processor and the system's memory and has a fixed architecture; the PL part is used to create customized peripherals.

[0086] The second Camera Link interface submodule adopts a Camera Link interface circuit, and the second Camera Link interface submodule is connected and communicated with an external multi-channel CMOS acquisition module. The second Camera Link interface submodule is used to receive the target image data transmitted by the first Camera Link interface submodule in the external multi-channel CMOS acquisition module, and the target image data is high-speed differential serial data, and the received high-speed differential serial data is serial-to-parallel converted, and the high-speed differential serial data is converted into multi-channel parallel low-speed data that can be received by the ZYNQ chip. The receiving port of the second Camera Link interface submodule is constructed by IP core primitives. First, the transmission clock is multiplied by 7 through PLL or MMCM to obtain a sampling clock, and the 4-channel LVDS data is sampled by 7 times, and then a 1:7 conversion is performed through a cascaded ISERDES (deserializer) to obtain a 28-bit parallel signal. When receiving LVDS data, in order to improve the transmission efficiency, the data of the receiving port of the second Camera Link interface submodule must be phase-aligned, and the input delay unit IODELAY is to delay the data sampling clock by a certain frequency, and use a dynamic clock and data phase adjustment unit to achieve the purpose of synchronous transmission of clock and data. The input clock signal is delayed by IODELAY, and the data realignment module Bit slip generated by the phase adjustment module reorders the parallel data in the ISERDES module, and finally obtains the field valid signal, line valid signal and 24-bit valid data according to the corresponding relationship of each bit. In particular, for the multi-channel image synchronous acquisition system, in order to ensure the synchronization of the image, the core board can send synchronous exposure instructions through the 4 pairs of control signals reserved in the Camera Link interface.

[0087] In an alternative implementation, please continue to refer to Figure 4 The core board also includes a Gigabit Ethernet interface communication module, a 10 Gigabit Ethernet interface communication module (including a 10Gigabit Ethernet interface circuit), a Flash storage chip and a DDR memory. The core board receives image data sent by multiple CMOS acquisition boards through the Camera Link interface circuit, realizes data and command interaction with the host computer through the Gigabit Ethernet interface, and sends the decoded image data through the 10 Gigabit Ethernet.

[0088] Specifically, the Gigabit Ethernet interface communication module is connected to the processing submodule and the host computer, and is used to receive network model parameters and send the model prediction results of the processing submodule to the host computer. The 10 Gigabit Ethernet interface communication module is connected to the processing submodule and the host computer, and is used to send the target decoded image data of the processing submodule to the host computer.

[0089] In the embodiment of the present application, the 10 Gigabit Ethernet interface communication module includes a media access controller (Media Access Controller, MAC), a physical coding sublayer (Physical Coding Sublayer, PCS), a physical media attachment sublayer (Physical Media Attach, PMA) and a physical media dependent sublayer (Physical Media Dependent, PMD). Specifically, the PMD sublayer can be replaced by an SFP+ optical module to achieve photoelectric / electro-optical conversion, PMA, PCS, and MAC can be constructed by a general IP core 10G Ethernet subsystem (10G Ethernet Subsystem), PMA and PCS are used to achieve data serialization and deserialization, CRC check and line coding, and the MAC layer uses a UDP protocol stack to achieve UDP transmission and reception.

[0090] The following will take the MobileNet accelerator as an example to explain in detail the model accelerator built into the core module.

[0091] In one embodiment, please refer to Figure 5 As an example, the MobileNet network model can include 1 layer of standard convolution, 11 layers of depth-separable convolution, an average pooling layer, a fully connected layer, and an output layer. MobileNet is a lightweight neural network. Its main feature is that while ensuring the accuracy of the model, it greatly reduces the number of model parameters and the amount of computation, making it possible to run deep learning network models on mobile devices with limited hardware resources and computing power. It is widely used in computer vision tasks in various mobile and embedded devices, such as image classification, target detection, and other fields.

[0092] In one embodiment, Figure 6 As shown, the MobileNet accelerator includes a model parsing module, a process control module, an interrupt module, a first parameter configuration module, a first storage control module, a second parameter configuration module, a second storage control module, a data loading module, a standard convolution module, a depth-separable convolution module, an average pooling module, a fully connected module and a Softmax module.

[0093] Among them, the MobileNet accelerator is deployed on the ZYNQ chip. Specifically, the model parsing module, process control module, first parameter configuration module, first storage control module, fully connected module and Softmax module are deployed on the PS end of the ZYNQ chip, which are implemented by the ARM processor; the interrupt module, second parameter configuration module, second storage control module, data loading module, standard convolution module, depthwise separable convolution module and average pooling module are deployed on the PL end of the ZYNQ chip, which are implemented by FPGA.

[0094] Exemplarily, the host computer sends the trained model parameters to the PS end of the ZYNQ chip through the Gigabit Ethernet interface communication module. Specifically, the PS end receives the trained network model parameters through the Gigabit Ethernet communication module, and transmits the trained network model parameters to the model parsing module. The model parsing module can read the network model parameters. The main function of the process control module is to realize the process and timing control of the accelerator. The parameter configuration modules of the PS end and the PL end are configurable IPs. The storage control modules of the PS end and the PL end can realize the caching and reading of feature maps, weights / biases and other data. Among them, the PL end receives parallel image data from CMOS through the Camera Link interface circuit, and the ZYNQ chip decodes the target parallel image data according to the Camera Link protocol to obtain target decoded image data. The image data is written into the DDR3 memory by the second storage control module, and the target decoded image data is processed by the model accelerator deployed on the ZYNQ chip to obtain the model prediction result. The data loading module realizes the sequential reading and writing of feature maps, weights / biases and other data under the control of the process control module. The standard convolution operation module can realize convolution operation and batch normalization operation. The depth separable convolution can realize depth convolution, point convolution and batch normalization operation. The average pooling module can realize average pooling operation. The fully connected module and the softmax module are used to realize classification. The softmax module sends the final model prediction result to the Gigabit Ethernet interface communication module, so that the Gigabit Ethernet interface communication module uploads the model prediction result to the host computer; the interrupt module is a global interrupt control module, which is responsible for receiving and processing interrupts generated by the process control module, the standard convolution module, the depth separable convolution module, the average pooling module, and the fully connected module. In particular, in order to improve the calculation speed, DDR memory can be mounted on the PS and PL ends of the ZYNQ chip respectively. The configuration information and prediction results are stored in the DDR on the PS end, and the input feature data and weight parameters and other information are stored in the DDR on the PL end.

[0095] Exemplarily, the first parameter configuration module on the PS side configures some parameters required by the PS side, such as model parameters of the fully connected module and the Softmax module, and receives parameters required by the PL side, such as model parameters of modules such as the standard convolution and depthwise separable convolution in the trained model parameters. The first parameter configuration module on the PS side writes these parameters into the BRAM on the FPGA on the PL side through the first storage control module on the PS side, the interrupt module on the PL side, the second parameter configuration module on the PL side, and the second storage control module on the PL side, so that the standard convolution module, depthwise separable convolution module, average pooling module, etc. on the PL side can call the corresponding parameters during calculation.

[0096] Taking 3×3 convolution as an example, the calculation formula implemented by the 3×3 convolution unit structure is:

[0097] X=A o w0+A1w1+…+A8w8; (1)

[0098] The formula for batch normalization is:

[0099] Y=aX+b; (2)

[0100] Substituting formula (1) into formula (2), we get:

[0101] Y=aA o w0+aA1w1+…+aA8w8+b;

[0102] In the formula, a represents the slope, b represents the intercept, A0 to A8 represent the 9 inputs of the 3*3 convolution, and w0 to w8 represent the 9 weights of the convolution.

[0103] It can be seen from the above formula that after obtaining the trained network model, the normalization parameters can be fused with the weights, and the batch normalization operation can be completed with only one adder.

[0104] See also Figure 7 In the figure, A and B of DSP Slice are multiplication inputs, C is addition input, and D is output. It can complete the operation of D=A×B+C. The convolution calculation unit consists of 9 DSP Slices and multiple D flip-flops. The C port of the first DSP is set to zero, and the remaining 8 DSPs use the corresponding delay of 0 to 8 clocks of the flip-flops to connect the output port of the current DSP to the addition input port of the next DSP. The output of the 9th DSP is the result of the 3×3 convolution unit operation.

[0105] See also Figure 8 , ReLU6 activation function is a variant of ReLU, and its function expression is f(x)=min(max(0,x),6). Similar to ReLU, when the input value x is greater than 0, the output of ReLU6 is equal to the input value x, but the difference is that when the input value x is greater than 6, the output of ReLU6 will be truncated to 6. ReLU6 can limit the output range of neurons, which helps to stabilize the training process. Using ReLU6 in low-precision environments such as mobile terminals can reduce precision loss. The computing unit consists of 1 comparator and 2 multiplexers. The input of the first multiplexer selection signal is the sign bit of the input data. When the sign bit is 1, it means that the data is negative and the output data is 0; when the sign bit is 0, the data enters the first multiplexer, and its selection signal input is the output of the comparator. When the input data is greater than 6, the output data is 6, otherwise the input data is output.

[0106] See also Fig. 9The standard convolution module consists of three 3×3 convolution operation units, multiple D flip-flops and one ReLU6 calculation unit. The MobileNet accelerator first uses the data loading module to read the image data, weights and normalized bias of the three channels, and then uses the three convolution operation units in the standard convolution module to perform parallel operations on the three input channels respectively, and then adds the results of the three 3×3 convolution operation units to the normalization parameter b respectively, and finally performs a ReLU6 operation on their sum to complete a standard convolution.

[0107] The basic idea of ​​the depthwise separable convolution in the depthwise separable convolution module is to decompose the standard convolution operation into two steps: depthwise convolution and pointwise convolution. This decomposition can greatly reduce the number of parameters and computational complexity while maintaining the expressiveness of the model. The depthwise separable convolution module in the embodiment of the present application includes a depthwise convolution operation unit and a pointwise convolution operation unit.

[0108] See also Fig.10 The deep convolution operation unit consists of a 3×3 convolution operation unit, a D flip-flop, and a ReLU6 calculation unit. The accelerator first uses the data loading module to read the weight and normalized bias b of the Nth layer of convolution, uses the convolution operation unit to perform convolution operations on the input feature map, and then adds it to the normalized parameter b. Finally, the ReLU6 operation is performed on their sum to complete a deep convolution.

[0109] See also Fig.11 The point convolution operation unit consists of multiple convolution operation units, a multiplexer, a BRAM cache unit and a ReLU module. The convolution operation unit is related to the network model. The BRAM cache unit is used to cache incomplete feature maps until the deep convolution outputs all feature maps to obtain complete feature maps, and then the feature maps are normalized and ReLU6 activated to output point convolution feature maps.

[0110] The beneficial effects of the embodiments of this application include:

[0111] 1. The embodiments of this application fully utilize the on-chip resources of FPGA, implement the Camera Link protocol and 10 Gigabit Ethernet communication protocol for image transmission, and design a multi-channel high-speed image acquisition system that can realize multi-channel image synchronous acquisition and long-distance real-time transmission functions.

[0112] 2. The embodiments of this application fully utilize the on-chip resources of ZYNQ to design a MobileNet accelerator, which is suitable for the MobileNetV3-small model. By loading different models, it can realize image classification, target detection and other functions in the embedded system, and realize real-time image processing.

[0113] Based on the same inventive concept, the embodiment of the present application also provides a multi-channel image processing method for implementing the multi-channel image processing system involved above. The implementation solution provided by the method to solve the problem is similar to the implementation solution recorded in the above system, so the specific limitations in one or more multi-channel image processing method embodiments provided below can refer to the limitations of the multi-channel image processing method system above, and will not be repeated here.

[0114] In one embodiment, Fig.11 As shown, a multi-channel image processing method is provided, the method comprising the following steps:

[0115] Step 1102: multiple CMOS acquisition modules acquire target image data corresponding to multiple CMOS image sensors respectively.

[0116] Step 1104, the core module receives the target image data, and obtains a model prediction result based on the target image data and a built-in model hardware accelerator, wherein the model prediction result is used to indicate at least one of an image recognition result, an image classification result, an image stitching result, or an image matching result for the target image data.

[0117] Step 1106: The host computer receives the model prediction result transmitted from the core module.

[0118] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0119] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0121] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0122] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A multi-channel image processing system, characterized in that: The system comprises: A plurality of CMOS acquisition modules are respectively connected to the plurality of CMOS image sensors, and are used to acquire target image data respectively corresponding to the plurality of CMOS image sensors; a core module, connected to the plurality of CMOS acquisition modules respectively, for receiving the target image data, and obtaining a model prediction result according to the target image data and a built-in model hardware accelerator, wherein the model prediction result is used to indicate at least one of an image recognition result, an image classification result, an image stitching result or an image matching result for the target image data; The host computer is connected to the core module and is used to receive the model prediction result transmitted from the core module.

2. The system according to claim 1, characterized in that The CMOS acquisition module comprises: An LVDS interface submodule, connected to the CMOS image sensor, and used to collect raw image data of the CMOS image sensor; A field programmable gate array, connected to the LVDS interface circuit, for receiving the original image data from the LVDS interface circuit, and preprocessing the original image data to obtain initial image data; The first Camera Link interface submodule is connected to the field programmable gate array and the core module, and is used for performing parallel-to-serial conversion on the initial image data to obtain target image data, and transmitting the target image data to the core module.

3. The system according to claim 2, characterized in that When executing the step of preprocessing the raw image data to obtain initial image data, the field programmable gate array is configured as follows: Performing filtering processing on the original image data to obtain original image data after noise reduction; The original image data after noise reduction is encoded according to the Camera Link protocol to obtain the initial image data.

4. The system according to claim 2, characterized in that The CMOS acquisition module also includes a CMOS driving submodule; The CMOS driving submodule is connected to the CMOS image sensor and is used to provide an operating voltage to the CMOS image sensor.

5. The system according to claim 1, characterized in that The core modules include: A second Camera Link interface submodule, connected to the CMOS acquisition module, is used to receive the target image data and perform serial-to-parallel conversion on the target image data to obtain target parallel image data; A processing submodule is connected to the second Camera Link interface submodule, and is used to receive the target parallel image data from the second Camera Link interface submodule, and decode the target parallel image data according to the Camera Link protocol to obtain target decoded image data, and obtain the model prediction result according to the target decoded image data and the model accelerator.

6. The system according to claim 5, characterized in that The core module also includes: A Gigabit Ethernet interface communication module, connected to the processing submodule and the host computer, for receiving network model parameters and sending the model prediction results of the processing submodule to the host computer; The 10 Gigabit Ethernet interface communication module is connected to the processing submodule and the host computer, and is used to send the target decoded image data of the processing submodule to the host computer.

7. The system according to claim 6, characterized in that The 10 Gigabit Ethernet interface submodule includes a media access controller, a physical coding sublayer, a physical attachment sublayer and a physical dependency sublayer; The media access controller is used to implement UDP sending and receiving using a UDP protocol stack; The physical coding sublayer and the physical attachment sublayer are used to implement data serialization and deserialization, CRC check and line coding; The physical dependent sublayer is used to realize photoelectric / electro-optical conversion.

8. The system according to claim 5, characterized in that The model accelerator includes a model parsing module, a process control module, an interrupt module, a parameter configuration module, a storage control module, a data loading module, a standard convolution module, a depth-separable convolution module, an average pooling module and a fully connected module; the interrupt module is respectively connected to the process control module, the standard convolution module, the depth-separable convolution module, the average pooling module and the fully connected module; The model parsing module is used to read network model parameters; The process control module is used to control the process and timing of the model accelerator; The interrupt module is used to process interrupts generated by the process control module, the standard convolution module, the depthwise separable convolution module, the average pooling module and the fully connected module; The parameter configuration module is used to receive the parameters of the network model read and transmitted by the model parsing module after model training, and configure and manage the parameters of the network model; The storage control module is used to realize the caching and reading of feature map data and weights; The data loading module is used to realize the sequential reading and writing of feature map data and weights / biases; The standard convolution operation module is used to implement convolution operation and batch normalization operation; The depthwise separable convolution module is used to implement depthwise convolution, pointwise convolution and batch normalization operations; The average pooling module is used to implement the average pooling operation; The fully connected module is used to implement classification.

9. The system according to claim 8, characterized in that The processing submodule adopts a ZYNQ chip, and both the PS end and the PL end of the ZYNQ chip are mounted with DDR memory; The DDR memory of the PS end is used to store configuration information and model prediction results; The DDR memory at the PL end is used to store input feature data and weight parameters.

10. A multi-channel image processing method, characterized in that: Applied to the system according to any one of claims 1 to 9, the method comprises: Multiple CMOS acquisition modules acquire target image data corresponding to multiple CMOS image sensors respectively; The core module receives the target image data, and obtains a model prediction result according to the target image data and a built-in model hardware accelerator, wherein the model prediction result is used to indicate at least one of an image recognition result, an image classification result, an image stitching result, or an image matching result for the target image data; The host computer receives the model prediction result transmitted from the core module.