Spectral image detection model, detection method and camera processor with on-chip computing

Abstract

A spectral image detection model, a detection method and a camera processor for on-chip computing, wherein the spectral image detection model includes N serially connected KAM blocks and a classifier block connected after the N KAM blocks; the KAM block includes a depthwise separable convolution layer, a batch normalization layer, an improved triple attention layer and ReLU The activation function layer; the depthwise separable convolutional layer is used to extract image features, and the improved triple attention layer is used to enhance the information interaction within and between feature maps; the classifier block includes the depthwise separable convolutional layer, batch normalization layer, ordinary convolutional layer and Sigmoid The spectral image detection model in the present invention has the characteristics of light weight and small computational complexity, and can be deployed in the camera processor in the camera to meet the needs of mobile and edge hyperspectral image processing.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a spectral image detection model, a detection method and an on-chip computing camera processor. Background Art

[0002] Hyperspectral imaging is an emerging imaging technology developed in recent years. Compared with ordinary RGB imaging solutions, hyperspectral imaging can not only obtain image spatial information, but also obtain additional spectral dimension information of each pixel in the image space. Since objects of different components have different reflection intensities in different spectral bands, spectral dimension information can better assist computer vision systems in classification and detection tasks.

[0003] The existing hyperspectral image detection network model has problems such as complex framework and large amount of computation, and an ultra-lightweight spectral image detection model is urgently needed.

[0004] In addition, the scale of three-dimensional hyperspectral images is larger than that of ordinary RGB images, and the models for analyzing and processing hyperspectral images are computationally intensive and storage intensive. Considering that the chips on ordinary industrial cameras have low computing power, hyperspectral cameras generally have to transmit the collected images to an external GPU server through a transmission interface, relying on a powerful external GPU server for image processing and analysis tasks. This will lead to an increase in cost, volume, and power consumption, and is also not conducive to the deployment of real-time application scenarios of hyperspectral images on edge and mobile terminals. Summary of the invention

[0005] The present invention provides a spectral image detection model, a detection method and a camera processor for on-chip computing to solve the technical problems mentioned in the background technology.

[0006] To achieve the above object, the technical solution of the present invention is achieved as follows:

[0007] The present invention provides a method for constructing an ultra-lightweight spectral image detection model based on an attention mechanism, comprising N serially connected KAM blocks and a classifier block connected after the N KAM blocks;

[0008] The KAM block is used for feature extraction of hyperspectral images. Each KAM block includes a depth-separable convolutional layer, a batch normalization layer, an improved triple attention layer, and ReLU Activation function layer; the depth-wise separable convolutional layer is used to extract image features while reducing the computational complexity and storage capacity of the spectral image detection model, and the improved triple attention layer is used to enhance the information interaction within and between feature maps;

[0009] The classifier block is used to output the final classification result, including the depth-wise separable convolutional layer, batch normalization layer, ordinary convolutional layer and Sigmoid Activation function layer.

[0010] Furthermore, the improved triple attention replaces the 3×3 convolutional layer of the existing triple attention with a 1×1 convolutional layer, thereby reducing the amount of computation and storage without affecting the overall performance; in addition, the existing triple attention Sigmoid The number of activation function layers is replaced by HSigmoid Activate the function layer to reduce hardware resource consumption;

[0011] The calculation formula of the improved triple attention layer is as follows:

[0012] ;

[0013] in, y represents the output of the improved triple attention layer; for The new tensor formed by rotating 90 degrees along the H axis is is the input three-dimensional tensor; express Hsigmoid Function layer; yes A new tensor formed by rotating 90 degrees along the W axis; , Respectively , go through ZPool The new tensor obtained after the operation; Represents a matrix-scalar multiplication operation; yes Directly pass without rotation Zpool The tensor generated after the operation;

[0014] In the above formula, The following relationship is satisfied:

[0015] ;

[0016] in, B The value of , and ; and for The decomposed parameters are H×C matrices; K 1 and K 2 is the convolution kernel K The parameters obtained by decomposition are all 1×1 scalar values;

[0017] also, The following relationship is satisfied:

[0018] ;

[0019] Among them, subscript 0 d It means that the input image dimension 0 is used for the maximum pooling operation and the average pooling operation; express Zpool operate; represents the average pooling operation; Represents a max pooling operation.

[0020] Furthermore, the HSigmoid The expression of the activation function layer is as follows:

[0021] .

[0022] The second aspect of the present invention proposes an image detection method based on deep learning, which uses the spectral image detection model constructed by the above construction method to perform detection, and specifically includes the following steps:

[0023] S1. Collect hyperspectral images and construct a dataset, then divide the dataset into training set, validation set and test set;

[0024] S2, select an image from the training set and transfer it to the spectral image detection model to obtain the predicted label;

[0025] S3, construct a loss function, calculate the loss between the predicted label and the real label of the image, and loop from S2 to S3 until the spectral image detection model converges, use the validation set for verification, select a set of weights with the best accuracy in the validation set as the weights of the spectral image detection model, and obtain the trained spectral image detection model;

[0026] S4. Use the test set to test the trained spectral image detection model to obtain the final classification result.

[0027] The third aspect of the present invention provides a highly integrated on-chip computing camera processor, which processes images using the spectral image detection model constructed by the above construction method, including:

[0028] PCB board 1, integrating an STM32 controller, a connector 1, a debugging module 1, a debugging module 2, a debugging module 3 and a power module 1, wherein the debugging module 1 and the debugging module 2 are connected to the connector 1, and the debugging module 3 and the power module 1 are both connected to the STM32 controller;

[0029] PCB board 2 is mounted on PCB board 1, and PCB board 2 integrates an FPGA processor, a buffer module, a memory module, a transmission interface module 2 and a power module 2, wherein PCB board 1 is connected to the FPGA processor through connector 1 to transmit a variety of control signals to the FPGA processor, and the buffer module, the memory module, the transmission interface module 2 and the power module 2 are all connected to the FPGA processor; the spectral image detection model is deployed in the FPGA processor;

[0030] PCB board three is mounted on the side of PCB board two. Power module three, connector two and CMOS sensor are integrated on PCB board three. Both power module three and CMOS sensor are connected to FPGA processor on PCB board two through connector two.

[0031] Further, the power module 1 is used to access the external power supply and output the main power supply through the overvoltage protection circuit. The main power supply is divided into two paths, one of which is transmitted to the power module 2 of the PCB board 2 through the connector 1, and the other is stepped down and input into the STM32 controller to power the STM32 controller;

[0032] The debugging module 1 is used to access the JTAG signal sent from the outside, and then transmit it to the FPGA processor in the PCB board 2 through the connector 1 to configure the control timing and image processing algorithm in the FPGA processor;

[0033] The debugging module 2 is used to send and receive UART signals, and transmit the signals to the FPGA processor in the PCB board 2 through the connector 1, so as to debug the control and image algorithms fixed in the FPGA processor;

[0034] The debugging module three is used for debugging the hardware of the STM32 controller and curing the power control timing of the STM32 controller;

[0035] The transmission interface module 1 is used to send and receive Ethernet signals, and transmit them to the FPGA processor through the connector 1, and is also used to receive test data sent from the outside;

[0036] The STM32 controller is used to schedule the power control timing, and transmits the power control signal to the power module 2 of the PCB board 2 through the connector 1 to meet the power-on requirements of the FPGA processor with different voltages.

[0037] Furthermore, the FPGA processor in the second PCB board is used for camera control, image acquisition, signal preprocessing and image detection;

[0038] The buffer module is used to perform a buffer operation on data;

[0039] The memory module is used to store the hyperspectral image and the spectral image detection model;

[0040] The transmission interface module 2 is used to transmit the hyperspectral image and the detection result of the hyperspectral image to the host computer for display;

[0041] The power supply module 2 is used to supply power to each IO port in the FPGA processor, the GTH transceiver and logic resources in the FPGA processor; and to supply power to the memory module and the buffer module.

[0042] Furthermore, the FPGA processor has a built-in signal preprocessing module, an image detection module, an image acquisition module, a camera control module, and an on-chip and off-chip cache interaction buffer;

[0043] The signal preprocessing module, the image detection module, the image acquisition module, the buffer module, and the memory module are all connected to the on-chip and off-chip buffer interaction buffers; the connector 2 in the PCB board 3 is connected to the image acquisition module through the LVDS interface; the camera control module is connected to the connector 2 in the PCB board 3 through the SPI interface; the connector 1 on the PCB board 1 and the transmission interface module 2 and the power module 2 on the PCB board 2 are all interconnected with the camera control module; the spectral image detection model is deployed in the image detection module.

[0044] Furthermore, the image detection module is integrated with an on-chip deployment module;

[0045] The on-chip deployment module includes a plurality of hardware computing modules 1 integrating a depth-separable convolution layer and a batch normalization layer and a plurality of hardware computing modules 2 integrating an improved triple attention layer and an activation function layer; the plurality of hardware computing modules 1 and the plurality of hardware computing modules 2 are arranged at intervals and are connected head to tail through a buffer in an FPGA processor;

[0046] The hardware computing module 1 reads the image pixels from the output buffer of the previous level, and obtains a total of nine image pixels X1~X9 through row buffer 1 and row buffer 2, and then multiplies the nine weights W1~W9 of the 3×3 convolution kernel pre-stored in the FPGA processor with X1~X9 in parallel respectively. On the other hand, the feedback control signal and the current position signal of the convolution window are sent to the non-blocking Padding pipeline control mechanism through the input control signal, row counter r and column counter c. The multiplication result obtained above is assigned by the Padding judgment of the non-blocking Padding pipeline control mechanism, and the result is input into the addition tree for accumulation to obtain the convolution result. The convolution result is stored in the result registers R1~R9, and the convolution result is also output to the buffer of the next level; the non-blocking Padding pipeline control mechanism is used to avoid pipeline confusion of the convolution window when the convolution window slides to the edge of the image and zero padding is performed.

[0047] Further, the CMOS sensor in the PCB board three is used to collect image data according to the control signal;

[0048] The power module three is used to supply power to the CMOS sensor;

[0049] The second connector is used to access the CMOS sensor control signal and the power supply signal from the second PCB board, and to transmit the differential image data signal from the CMOS sensor to the second PCB board.

[0050] Beneficial effects of the present invention:

[0051] 1. The present invention provides an ultra-lightweight spectral image detection model based on an attention mechanism, comprising N serially connected KAM blocks and a classifier block connected after the N KAM blocks, wherein the KAM block comprises a depth-separable convolutional layer, a batch normalization layer, an improved triple attention layer and ReLU Activation function layer;

[0052] Compared with ordinary convolutional layers, the depthwise separable convolutional layer can significantly reduce the amount of computation and storage of the model while maintaining the model detection performance, which is conducive to the deployment of the convolution algorithm on the camera processor;

[0053] The improved triple attention layer replaces the existing 3×3 convolution layer of the triple attention with a 1×1 convolution layer, which simplifies the convolution calculation to the sum operation of matrix point multiplication, thereby reducing the amount of calculation and storage without affecting the overall performance; in addition, the existing triple attention Sigmoid The number of activation function layers is replaced by HSigmoid Activate function layers to reduce hardware resource consumption.

[0054] 2. The present invention provides a camera processor with on-chip computing capabilities. By constructing an FPGA processor, a memory module and other components, the camera processor realizes a full on-chip computing function integrating acquisition, processing and detection. The integrated FPGA processor can control the camera's acquisition, perform image preprocessing calculations, and replace the external GPU server to perform classification and detection tasks with specific requirements.

[0055] The proposed camera processor has a high degree of integration and has the characteristics of miniaturization and low power consumption, and can meet the needs of hyperspectral image processing on mobile and edge terminals. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a structural block diagram of the spectral image detection model in the present invention;

[0057] Figure 2It is a connection diagram of various components in the camera processor of the present invention;

[0058] Figure 3 is a three-dimensional structural diagram of the camera processor in the present invention;

[0059] Figure 4 It is a structural block diagram of an on-chip deployment module in an embodiment of the present invention;

[0060] Figure 5 is an operating principle diagram of a hardware computing module 1 in an embodiment of the present invention;

[0061] Figure 6 This is a schematic diagram of the operation principle of the hardware computing module 2 in an embodiment of the present invention.

[0062] Description of reference numerals:

[0063] 1. PCB board one; 2. PCB board two; 3. PCB board three; 4. Connector one; 5. Connector two; 6. Copper pillar. DETAILED DESCRIPTION

[0064] In order to facilitate the understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings. Preferred embodiments of the present invention are provided in the drawings. However, the present invention can be implemented in many other different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive.

[0065] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0066] It should be understood that the orientation or position relationship indicated by terms such as "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside" and "outside" are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0067] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0068] It should also be noted that, in the embodiments of the present application, the same figure mark is used to represent the same component or the same part. For the same parts in the embodiments of the present application, the figure may only mark one of the parts or components as an example. It should be understood that the figure mark also applies to other identical parts or components.

[0069] Reference Figure 1 , the embodiment of the present application provides a method for constructing an ultra-lightweight spectral image detection model (KAM-Net) based on an attention mechanism, including N series-connected KAM blocks and a classifier block connected after the N KAM blocks; KAM-Net is composed of several KAM blocks and is scalable, and KAM blocks can be added or reduced for specific tasks, thereby achieving a balance between model scale and detection performance;

[0070] The KAM block is used for feature extraction of hyperspectral images. Each KAM block includes a depth-separable convolutional layer, a batch normalization layer, an improved triple attention layer, and ReLU The activation function layer; the depthwise separable convolution layer is used to extract image features while reducing the computational and storage requirements of the spectral image detection model. The batch normalization layer (or batch normalization layer) follows the depthwise separable convolution layer to enhance the model fitting performance. The improved triple attention layer is used to enhance the information interaction within and between feature maps. ReLU Activation function layer, used to increase the generalization performance of the model;

[0071] The classifier block is used to output the final classification result, including the depth-wise separable convolutional layer, batch normalization layer, ordinary convolutional layer and Sigmoid Activation function layer.

[0072] The reason for using the depthwise separable convolutional layer is that, compared with the ordinary convolutional layer, the depthwise separable convolutional layer can greatly reduce the computational and storage amount of the model while maintaining the model detection performance, which is conducive to the deployment of the convolution algorithm on the camera processor.

[0073] In some embodiments, the improved triple attention replaces the existing 3×3 convolution layer of the triple attention with a 1×1 convolution layer, and the convolution kernel size is [2,1,1], which can simplify the convolution calculation to the sum operation of matrix point multiplication, thereby reducing the amount of calculation and storage without affecting the overall performance; in addition, the existing triple attention Sigmoid The number of activation function layers is replaced by HSigmoid Activate the function layer to reduce hardware resource consumption;

[0074] The calculation formula of the improved triple attention layer is as follows:

[0075] ;

[0076] in, y represents the output of the improved triple attention layer; for The new tensor formed by rotating 90 degrees along the H axis has a shape of (W×H×C), where C is the number of channels, H is the height, and W is the width. is the input three-dimensional tensor; express Hsigmoid Function layer; yes A new tensor formed by rotating 90 degrees along the W axis; , Respectively , go through ZPool The new tensor obtained after the operation; is a tensor of shape (2×H×C), is a tensor of shape (2×C×W); Represents a matrix-scalar multiplication operation; yes Directly pass without rotation Zpool The resulting shape after the operation is a tensor of (2×H×W);

[0077] In the above formula, The following relationship is satisfied:

[0078] ;

[0079] in, B The value of , and ; and for The decomposed parameters are H×C matrices; K 1 and K 2 is the convolution kernel K The parameters obtained by decomposition are all 1×1 scalar values;

[0080] also, The following relationship is satisfied:

[0081] ;

[0082] Among them, subscript 0 d It means that the input image dimension 0 is used for the maximum pooling operation and the average pooling operation; express Zpool operate; represents the average pooling operation; Represents a max pooling operation.

[0083] In some embodiments, the HSigmoid The expression of the activation function layer is as follows:

[0084] .

[0085] Hsigmoid The activation function layer removes the original Sigmoid The exponential calculation in the function can convert the fixed division calculation of 1 / 6 into a multiplication by 0.16667. In addition, the operations of obtaining the maximum and minimum values ​​can directly use comparators in digital logic without special calculations. These improvements can reduce hardware resource consumption.

[0086] The second aspect of the present invention further provides an image detection method based on deep learning, which uses the spectral image detection model constructed by the above construction method for detection, and specifically includes the following steps:

[0087] S1. Collect hyperspectral images and construct a dataset, then divide the dataset into training set, validation set and test set;

[0088] S2, select an image from the training set and transfer it to the spectral image detection model to obtain the predicted label;

[0089] S3, construct a loss function, calculate the loss between the predicted label and the real label of the image, and loop from S2 to S3 until the spectral image detection model converges, use the validation set for verification, select a set of weights with the best accuracy in the validation set as the weights of the spectral image detection model, and obtain the trained spectral image detection model;

[0090] S4. Use the test set to test the trained spectral image detection model to obtain the final classification result.

[0091] Reference Figure 2 and Figure 3The third aspect of the present invention further provides a highly integrated on-chip computing camera processor, which processes images using the spectral image detection model constructed by the above construction method, including:

[0092] PCB board 1 integrates an STM32 controller, a connector 4, a debugging module 1, a debugging module 2, a debugging module 3 and a power module 1, wherein the debugging module 1 and the debugging module 2 are connected to the connector 4, and the debugging module 3 and the power module 1 are both connected to the STM32 controller; PCB board 1 is mainly used to place physical interfaces such as the STM32 controller, the camera main power supply / programming / debugging, etc., and has a size of 130 mm×100 mm;

[0093] PCB board 2 2 is mounted on PCB board 1 1. PCB board 2 2 is mainly used to place FPGA processor, cache module, memory module and its peripheral circuits, and has a size of 130 mm×100 mm. PCB board 2 2 integrates FPGA processor, cache module, memory module, transmission interface module 2 and power module 2, wherein PCB board 1 is connected to FPGA processor through connector 1 4 to transmit various control signals to FPGA processor, and cache module, memory module, transmission interface module 2 and power module 2 are all connected to FPGA processor; spectral image detection model is deployed in FPGA processor;

[0094] PCB board three 3 is mounted on the side of PCB board two 2. PCB board three 3 is mainly used to place CMOS image sensor and its peripheral circuits, and has a size of 58 mm×58 mm. PCB board three 3 integrates power module three, connector two 5 and CMOS sensor. Both power module three and CMOS sensor are connected to FPGA processor on PCB board two 2 through connector two 5. The CMOS sensor is designed separately on PCB board three 3, which is convenient for replacing different CMOS sensors in different usage scenarios, and is easy for transplantation and upgrade iteration.

[0095] The camera processor in the present invention has a high degree of spatial integration, with a spatial volume not exceeding 150 mm×120 mm×10 mm, and can be easily assembled in a camera housing.

[0096] The "full on-chip computing" described in the present invention is reflected in the fact that the camera processor can complete all processes of the three aspects of hyperspectral camera image acquisition, signal preprocessing, and image detection and analysis without the participation of other computing carriers outside the camera.

[0097] In some embodiments, the power module 1 is used to access an external power supply and output a +12.0V main power supply through an overvoltage protection circuit. The main power supply is divided into two paths, one of which is transmitted to the power module 2 of the PCB board 2 2 through the connector 1 4, and the other path steps down the +12.0V main power supply to +3.3V and inputs it into the STM32 controller to power the STM32 controller;

[0098] The debugging module 1 is used to access the JTAG signal sent from the outside, and then transmit it to the FPGA processor in the PCB board 2 2 through the connector 1 4 to configure the control timing and spectral image detection model in the FPGA processor;

[0099] The debugging module 2 is used to send and receive UART signals, and transmit them to the FPGA processor in the PCB board 2 through the connector 1 4, so as to debug the control and spectral image detection model solidified by the FPGA processor;

[0100] The debugging module three is used for debugging the hardware of the STM32 controller and curing the power control timing of the STM32 controller;

[0101] The transmission interface module 1 is used to send and receive Ethernet signals, and transmit them to the FPGA processor through the connector 1 4, and is also used to receive test data sent from the outside;

[0102] The STM32 controller is used to schedule the power control timing and transmit the power control signal to the power module 2 of the PCB board 2 through the connector 1 4 to meet the power-on requirements of different voltages of the FPGA processor.

[0103] In some embodiments, the FPGA processor in the PCB board 2 is used for camera control, image acquisition, signal preprocessing and image detection; in addition, the FPGA processor sends a control stream based on the SPI protocol through the HP (High Performance) IO port, and the control stream enters the XCE, SDO, SCK, and SDI ports corresponding to the CMOS sensor of the PCB board 1 to configure the registers inside the CMOS sensor, including setting the CMOS sensor to a trigger mode, the number of output channels to 8, the AD converter to 8 bits, etc.;

[0104] The buffer module is used to cache data; specifically, the buffer module is used for cache operations such as image segmentation and temporary storage of intermediate values ​​during hyperspectral image processing;

[0105] The memory module is used to store the hyperspectral image and the spectral image detection model;

[0106] The transmission interface module 2 uses two Gigabit SFP+ optical ports to transmit large-scale hyperspectral images and the detection results of hyperspectral images to the host computer for display;

[0107] The power module 2 is used to supply power to each IO port in the FPGA processor, the GTH transceiver in the FPGA processor, and the logic resources; and to supply power to the memory module, the buffer module, etc. Specifically, the power module 2 in the PCB board 2 is used to step down the +12.0V main power supplied from the connector 1 4 to +3.3V, +5.0V, +1.8V, +1.2V, +2.3V, +0.85V, etc. Among them, +3.3V and +1.8V are mainly used to supply power to each IO port in the FPGA processor, and the memory module; +1.2V is mainly used to supply power to the buffer module; +2.3V is mainly used to supply power to the GTH transceiver in the FPGA processor; +0.85V is mainly used to supply power to the logic resources in the FPGA processor.

[0108] In some embodiments, the FPGA processor has a built-in signal preprocessing module, an image detection module, an image acquisition module, a camera control module, and an on-chip and off-chip cache interaction buffer;

[0109] The signal preprocessing module, image detection module, image acquisition module, cache module, and memory module are all connected to the on-chip and off-chip cache interaction buffers. The connector two 5 in the PCB board three 3 is connected to the image acquisition module through the LVDS interface, and the camera control module is connected to the connector two 5 in the PCB board three 3 through the SPI interface; the connector one 4 on the PCB board one 1 and the transmission interface module two and the power supply module two on the PCB board two 2 are all interconnected with the camera control module; the spectral image detection model is deployed in the image detection module.

[0110] Reference Figure 4 , in some embodiments, the image detection module is integrated with an on-chip deployment module;

[0111] The on-chip deployment module includes a plurality of hardware computing modules 1 integrating a depth-separable convolution layer and a batch normalization layer and a plurality of hardware computing modules 2 integrating an improved triple attention layer and an activation function layer; the plurality of hardware computing modules 1 and the plurality of hardware computing modules 2 are arranged at intervals and are connected head to tail through a buffer in an FPGA processor;

[0112] The hardware computing module 1 reads the image pixels from the output buffer of the previous level, and obtains a total of nine image pixels X1~X9 through row buffer 1 and row buffer 2, and then multiplies the nine weights W1~W9 of the 3×3 convolution kernel pre-stored in the FPGA processor with X1~X9 in parallel respectively. On the other hand, the feedback control signal and the current position signal of the convolution window are sent to the non-blocking Padding pipeline control mechanism through the input control signal, row counter r and column counter c. The multiplication result obtained above is assigned by the Padding judgment of the non-blocking Padding pipeline control mechanism, and the result is input into the addition tree for accumulation to obtain the convolution result. The convolution result is stored in the result registers R1~R9, and the convolution result is also output to the buffer of the next level; the non-blocking Padding pipeline control mechanism is used to avoid pipeline confusion of the convolution window when the convolution window slides to the edge of the image and zero padding is performed.

[0113] Specifically, Figure 5 As shown in the figure, FIFO1 and FIFO2 are row buffer 1 and row buffer 2 respectively; W1~W9 are the weights of the 3×3 convolution kernel pre-stored in the FPGA processor, and X1~X9 are the incoming image pixels; W1~W9 are multiplied in parallel with X1~X9, and the results are input into the addition tree for accumulation to obtain the convolution results, which are stored in the result registers R1~R9; the convolution results flow into the output buffer; due to the zero-padding operation in the 3×3 convolution layer, when the convolution window slides to the edge of the image, it is necessary to additionally fill in pixels with a value of 0; this operation disrupts the pipeline of the convolution window.

[0114] To this end, an innovative non-blocking Padding pipeline control mechanism with Padding judgment assignment operation for 3×3 convolution is proposed. By setting the row and column counters of the image additionally, according to the row and column counters, when the convolution window moves to the edge of the image, the results of the R1~R9 registers are set to 0. The judgment rules are shown in Table 1. First, the length H and width W of the input image are given. When the input image flows into the calculation unit, the row counter and column counter are accumulated to calculate the coordinates (r, c) of the current incoming pixel in the original image. According to the conditions of (r, c) in Table 1, specific operations are performed on the result registers R1~R9.

[0115] Table 1: Judgment rules of non-blocking Padding flow control mechanism;

[0116] ;

[0117] In addition, the hardware computing module 2 implements the hardware computing module of the triple attention algorithm (i.e., the improved triple attention layer) on the FPGA processor for the first time. Figure 6 As shown; according to the triple attention algorithm, attention needs to be calculated in three directions: (C, W), (H, C) and (H, W), and applied independently in each direction Zpool Operation, calculate the maximum and average values ​​of pixels in three directions. The present invention designs a triple attention calculation unit for each direction, namely, row attention calculation module PE CW , column attention calculation module PE HC , channel attention calculation module PE HW , the three submodules are calculated in parallel, which shortens the on-chip calculation time of the triple attention algorithm and improves the real-time performance of the overall calculation; the row counter, column counter or pixel counter in the three submodules counts according to the input data flow, and makes boundary judgments according to the size of the input image to control the operation of each device in each submodule; since the present invention changes the 3×3 convolution in the triple attention to 1×1 convolution, compared with the 3×3 convolution, the 1×1 convolution does not require the design of an additional convolution layer module, but directly multiplies and accumulates the maximum value and average value obtained with two parameters respectively. The result is sent to Hsigmoid In the activation function layer; compared to Sigmoid Hardware is required to implement extremely resource-intensive exponential and division calculations. HSigmoid The activation function layer only requires a few comparators and multipliers in hardware;

[0118] In addition, when the hardware computing module 2 performs triple attention algorithm (i.e., improved triple attention layer) calculation, a large number of intermediate values ​​generated during the calculation process are sent to the cache module for caching by using the cache module installed in the PCB board 2 2; compared with caching to the internal memory or register of the FPGA processor, the use of an external cache can greatly reduce the resource consumption of the FPGA processor in the PCB board 2 2.

[0119] In some embodiments, the CMOS sensor in the PCB board 3 is used to collect image data according to the control signal;

[0120] The power module three is used to supply power to the CMOS sensor;

[0121] The second connector 5 is used to access the CMOS sensor control signal and the power supply signal from the second PCB board 2 , and to transmit the differential image data signal from the CMOS sensor to the second PCB board 2 .

[0122] In some embodiments, the PCB board 1 is connected to the FPGA processor via a connector 4 to transmit ETH, UART, JTAG and power control signals to the FPGA processor.

[0123] In some embodiments, reference Figure 3 The PCB board 1 is mounted directly below the PCB board 2 through multiple copper columns 6 and the connector 1 4; the PCB board 3 is mounted on the side of the PCB board 2 2 through the connector 2 5.

[0124] In some embodiments, the CMOS sensor is a Sony IMX273 black and white sensor.

[0125] In some embodiments, the FPGA processor uses a Xilinx ZU19EG SoC chip; the buffer module uses eight MT40A1G16RC chips connected in series; and the memory module uses an IS21ES08G chip.

[0126] When the camera processor performs signal preprocessing calculations, the steps are as follows:

[0127] The FPGA processor reads the original hyperspectral image cached in the village buffer module (DDR), performs an interpolation algorithm on the original hyperspectral image, and corrects the damaged pixels;

[0128] The FPGA processor performs black level correction to eliminate the effects of the original hyperspectral image caused by the difference in the response of each pixel of the CMOS sensor;

[0129] The FPGA processor performs band merging to fuse the adjacent hyperspectral band images, and after fusion, a continuous 32-band hyperspectral image between 400nm and 800nm ​​is obtained;

[0130] The FPGA processor performs image normalization, mapping the pixel values ​​of the hyperspectral image from 0 to 255 to between -1 and 1, thereby facilitating the subsequent image detection steps; at this point, the hyperspectral image signal preprocessing is completed and the image is stored in the buffer module.

[0131] Taking the classification of the Chinese medicinal material Achyranthes bidentata-Chuanxiong as an example, the implementation process of the camera processor performing classification calculations is introduced:

[0132] The FPGA processor reads the hyperspectral image in the buffer module after signal preprocessing, and sends it to the KAM-Net network that has been trained and solidified in the FPGA in 12×12×32 blocks. In this embodiment, KAM-Net includes three KAM blocks; Figure 4 The hyperspectral image is calculated by multiple rounds of hardware computing module 1 and hardware computing module 2, and finally passes through the classifier block. sigmoidAfter the activation function layer, the predicted category of the pixel in the block is obtained as 0 or 1, where 0 represents that the pixel is Achyranthes bidentata and 1 represents that the pixel is Achyranthes bidentata;

[0133] The above process is repeated until all blocks are calculated; the calculation results of all blocks are stored in the buffer module.

[0134] The above signal preprocessing tasks and image classification tasks are all calculated within the FPGA processor without the participation of external computing carriers.

[0135] The present invention discloses a camera processor with on-chip computing capability. By constructing an FPGA processor, a memory module and other components, the camera processor realizes a full on-chip computing function integrating acquisition, processing and detection. The integrated FPGA processor can control the camera's acquisition, perform image preprocessing calculations, and replace an external GPU server to perform classification and detection tasks with specific requirements.

[0136] The proposed camera processor has a high degree of integration and has the characteristics of miniaturization and low power consumption, and can meet the needs of hyperspectral image processing on mobile and edge terminals.

[0137] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered within the protection scope of the present invention. In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the ability of ordinary technicians in the field to implement. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for constructing an ultra-lightweight spectral image detection model based on an attention mechanism, characterized in that: It includes N serially connected KAM blocks and a classifier block connected after the N KAM blocks; The KAM block is used for feature extraction of hyperspectral images. Each KAM block includes a depth-separable convolutional layer, a batch normalization layer, an improved triple attention layer, and ReLU Activation function layer; the depth-wise separable convolutional layer is used to extract image features while reducing the computational complexity and storage capacity of the spectral image detection model, and the improved triple attention layer is used to enhance the information interaction within and between feature maps; The classifier block is used to output the final classification result, including the depth-wise separable convolutional layer, batch normalization layer, ordinary convolutional layer and Sigmoid Activation function layer; The improved triple attention replaces the existing 3×3 convolutional layer of the triple attention with a 1×1 convolutional layer, thereby reducing the amount of computation and storage without affecting the overall performance; in addition, the existing triple attention Sigmoid The number of activation function layers is replaced by HSigmoid Activate the function layer to reduce hardware resource consumption; The calculation formula of the improved triple attention layer is as follows: ； in, y represents the output of the improved triple attention layer; for The new tensor formed by rotating 90 degrees along the H axis is is the input three-dimensional tensor; express Hsigmoid Function layer; yes The new tensor is formed by rotating 90 degrees along the W axis; , Respectively , go through ZPool The new tensor obtained after the operation; Represents a matrix-scalar multiplication operation; yes Directly pass without rotation Zpool The tensor generated after the operation; In the above formula, The following relationship is satisfied: ； in, B The value of , and ; and for Decomposed parameters; K 1 and K 2 is the convolution kernel K The parameters obtained by decomposition are all 1×1 scalar values; also, The following relationship is satisfied: ； Among them, subscript 0 d It means that the input image dimension 0 is used for the maximum pooling operation and the average pooling operation; express Zpool operate; represents the average pooling operation; Represents a max pooling operation.

2. The construction method according to claim 1, characterized in that: Said HSigmoid The expression of the activation function layer is as follows: 。 3. A deep learning-based image detection method, characterized in that: The detection is performed using the spectral image detection model constructed by the construction method according to any one of claims 1 to 2, specifically comprising the following steps: S1. Collect hyperspectral images and construct a dataset, then divide the dataset into training set, validation set and test set; S2, select an image from the training set and transfer it to the spectral image detection model to obtain the predicted label; S3, construct a loss function, calculate the loss between the predicted label and the real label of the image, and loop from S2 to S3 until the spectral image detection model converges, use the validation set for verification, select a set of weights with the best accuracy in the validation set as the weights of the spectral image detection model, and obtain the trained spectral image detection model; S4. Use the test set to test the trained spectral image detection model to obtain the final classification result.

4. A highly integrated on-chip computing camera processor, characterized in that: The spectral image detection model constructed by the construction method according to any one of claims 1 to 2 is used to process the image, comprising: PCB board 1, integrating an STM32 controller, a connector 1, a debugging module 1, a debugging module 2, a debugging module 3 and a power module 1, wherein the debugging module 1 and the debugging module 2 are connected to the connector 1, and the debugging module 3 and the power module 1 are both connected to the STM32 controller; PCB board 2 is mounted on PCB board 1, and PCB board 2 integrates an FPGA processor, a buffer module, a memory module, a transmission interface module 2 and a power module 2, wherein PCB board 1 is connected to the FPGA processor through connector 1 to transmit a variety of control signals to the FPGA processor, and the buffer module, the memory module, the transmission interface module 2 and the power module 2 are all connected to the FPGA processor; the spectral image detection model is deployed in the FPGA processor; PCB board three is mounted on the side of PCB board two. Power module three, connector two and CMOS sensor are integrated on PCB board three. Both power module three and CMOS sensor are connected to FPGA processor on PCB board two through connector two.

5. The camera processor according to claim 4, characterized in that: The power module 1 is used to access the external power supply and output the main power supply through the overvoltage protection circuit. The main power supply is divided into two paths, one of which is transmitted to the power module 2 of the PCB board 2 through the connector 1, and the other is input into the STM32 controller after stepping down to power the STM32 controller; The debugging module 1 is used to access the JTAG signal sent from the outside, and then transmit it to the FPGA processor in the PCB board 2 through the connector 1 to configure the control timing and image processing algorithm in the FPGA processor; The debugging module 2 is used to send and receive UART signals, and transmit the signals to the FPGA processor in the PCB board 2 through the connector 1, so as to debug the control and image algorithms fixed in the FPGA processor; The debugging module three is used for debugging the hardware of the STM32 controller and curing the power control timing of the STM32 controller; The transmission interface module 1 is used to send and receive Ethernet signals, and transmit them to the FPGA processor through the connector 1, and is also used to receive test data sent from the outside; The STM32 controller is used to schedule the power control timing, and transmits the power control signal to the power module 2 of the PCB board 2 through the connector 1 to meet the power-on requirements of the FPGA processor with different voltages.

6. The camera processor according to claim 4, characterized in that: The FPGA processor in the PCB board 2 is used for camera control, image acquisition, signal preprocessing and image detection; The buffer module is used to perform a buffer operation on data; The memory module is used to store the hyperspectral image and the spectral image detection model; The transmission interface module 2 is used to transmit the hyperspectral image and the detection result of the hyperspectral image to the host computer for display; The power supply module 2 is used to supply power to each IO port in the FPGA processor, the GTH transceiver and logic resources in the FPGA processor; and to supply power to the memory module and the buffer module.

7. The camera processor according to claim 4, characterized in that: The FPGA processor has a built-in signal preprocessing module, an image detection module, an image acquisition module, a camera control module, and an on-chip and off-chip cache interaction buffer; The signal preprocessing module, the image detection module, the image acquisition module, the buffer module, and the memory module are all connected to the on-chip and off-chip buffer interaction buffers; the connector 2 in the PCB board 3 is connected to the image acquisition module through the LVDS interface; the camera control module is connected to the connector 2 in the PCB board 3 through the SPI interface; the connector 1 on the PCB board 1 and the transmission interface module 2 and the power module 2 on the PCB board 2 are all interconnected with the camera control module; the spectral image detection model is deployed in the image detection module.

8. The camera processor according to claim 7, characterized in that: The image detection module is integrated with an on-chip deployment module; The on-chip deployment modules include multiple hardware computing modules 1 integrating a depth-separable convolution layer and a batch normalization layer, and multiple hardware computing modules 2 integrating an improved triple attention layer and an activation function layer; The plurality of hardware computing modules 1 and the plurality of hardware computing modules 2 are arranged at intervals and connected head to tail through a buffer in the FPGA processor; The hardware computing module 1 reads the image pixels from the output buffer of the previous level, and obtains a total of nine image pixels X1~X9 through row buffer 1 and row buffer 2, and then multiplies the nine weights W1~W9 of the 3×3 convolution kernel pre-stored in the FPGA processor with X1~X9 in parallel respectively. On the other hand, the feedback control signal and the current position signal of the convolution window are sent to the non-blocking Padding pipeline control mechanism through the input control signal, row counter r and column counter c. The multiplication result obtained above is assigned by the Padding judgment of the non-blocking Padding pipeline control mechanism, and the result is input into the addition tree for accumulation to obtain the convolution result. The convolution result is stored in the result registers R1~R9, and the convolution result is also output to the buffer of the next level; the non-blocking Padding pipeline control mechanism is used to avoid pipeline confusion of the convolution window when the convolution window slides to the edge of the image and zero padding is performed.

9. The camera processor according to claim 4, characterized in that: The CMOS sensor in the PCB board three is used to collect image data according to the control signal; The power module three is used to supply power to the CMOS sensor; The second connector is used to access the CMOS sensor control signal and the power supply signal from the second PCB board, and to transmit the differential image data signal from the CMOS sensor to the second PCB board.

Images