FPGA (Field Programmable Gate Array) implementation architecture of light-weight preprocessing network of lens-free imaging system

Abstract

The invention provides an FPGA (Field Programmable Gate Array) implementation architecture of a light-weight preprocessing network of a lens-free imaging system. The light-weight preprocessing network comprises seven Conv + ReLU layers and a Conv layer, the FPGA implementation architecture comprises a control module, a BRAM storage module, a fifo cache module, a register block module, a windowing module, a convolution module and a ReLU module. According to the design of the ISP lightweight convolutional neural network (CNN-ISP-net), a corresponding hardware framework is realized on the FPGA, the calculation speed of the network can be accelerated, and the picture with better performance can be reconstructed after preprocessing of the method.

Description

Technical Field

[0001] The present invention relates to the technical field of lens-free imaging, and in particular to an FPGA implementation architecture of a lightweight preprocessing network of a lens-free imaging system. Background Art

[0002] Lens-free imaging is an emerging imaging technology that replaces lenses with computing. Compared with traditional imaging systems, it is small and convenient.

[0003] The patent application with patent number CN114677766A proposes a sign language recognition method, system and interactive device based on lens-free imaging technology; by using lens-free imaging technology, the optical lens part of the sign language recognition system is removed, and the light modulation function of the optical lens is replaced by back-end computing and optical modulation devices, which overcomes the shortcomings of traditional sign language recognition systems such as large size and low concealment, improves the communication efficiency between ordinary people and deaf-mute people, and enables deaf-mute people to better integrate into social life.

[0004] The patent application with patent number CN112153296A proposes an automatic exposure control method, device and camera with FPGA; target detection is performed on the current image frame captured by the camera, and the area where the target exists in the current image frame is determined as the target image area; the brightness of the target image area is counted to obtain a reference brightness; according to the reference brightness, the exposure strategy of the camera is configured to improve the user experience of the user viewing the image.

[0005] Since the lensless imaging system is an embedded imaging system, it has high requirements on volume and imaging speed. Therefore, it is necessary to realize lensless imaging at the hardware level.

[0006] In view of this, the present invention implements a corresponding hardware framework on FPGA based on the designed ISP lightweight convolutional neural network (CNN-ISP-net) to speed up the calculation speed of the network. Summary of the invention

[0007] The purpose of the present invention is to propose an FPGA implementation architecture of a lightweight preprocessing network for a lensless imaging system; according to the designed ISP lightweight convolutional neural network (CNN-ISP-net), a corresponding hardware framework is implemented on the FPGA to speed up the calculation speed of the network.

[0008] To achieve the above purpose, the technical solution of the present invention is: an FPGA implementation architecture of a lightweight preprocessing network for a lensless imaging system; the lightweight preprocessing network includes 7 Conv+ReLU layers and one Conv layer; the FPGA implementation architecture includes a control module, a BRAM storage module, a fifo cache module, a register group module, a windowing module, a convolution module, and a ReLU module;

[0009] The control module is responsible for scheduling the input and output of image data and weight data, and controlling the start and end of tasks of each module;

[0010] The BRAM storage module is used to store original image data, weight data and pre-processed output data;

[0011] The fifo buffer module is used to buffer the intermediate calculation data of the convolution module and the ReLU module;

[0012] The register group module includes a weight register group and a data register group, which are respectively used to cache weight data and image data involved in convolution calculation; the weight data comes from the BRAM storage module; the image data comes from the BRAM storage module and the fifo cache module;

[0013] The windowing module is used to realize the stepping operation of convolution together with the data register group;

[0014] The convolution module and the ReLU module are responsible for the computational tasks of the lightweight preprocessing network.

[0015] Preferably, the windowing module is connected to the BRAM storage module and the fifo cache module respectively through a multiplexer MUX to achieve scheduling of two data sources.

[0016] Preferably, the first layer of the lightweight preprocessing network uses 8 filters of size 3×3×1 for convolution to generate 8 feature maps; the 2nd to 7th Conv+ReLU layers use 8 filters of size 3×3×8; and the last Conv layer uses a convolution kernel of size 3×3×8 to complete the reconstruction output of the image.

[0017] Preferably, the windowing process of the windowing module is implemented through a row caching process of a shift register; the weight register group and the data register group each include 9 registers, wherein the 9 registers of the data register group are used to cache data within a convolution window, and the 9 registers of the weight register group are used to cache corresponding convolution weights.

[0018] Preferably, the fifo cache module uses 16 fifos, which are divided into two groups, group 1 and group 2, with 8 fifos in each group;

[0019] Group1 and group2 are used to cache the output and input of the convolution operation. When the fifo in group1 is used as the convolution operation feature map output cache, the fifo in group2 is used as the convolution operation feature map input cache. When group1 is used as the convolution operation feature map input cache, group2 is used as the convolution operation feature map output cache.

[0020] The fifo supports simultaneous reading and writing and is controlled by two enable signals r_en and w_en; when the fifo is used to cache the feature map data input by the convolution operation, the r_en enable signal is pulled high, and when the fifo is used to cache the feature map data output by the convolution operation, the w_en enable signal is pulled high.

[0021] Preferably, the convolution module is composed of 8 multiplication-addition units, and the multiplication-addition units form a multiplication-accumulation array through nine multipliers and eight adders, and the multiplication-accumulation array performs the following operations in a single clock cycle:

[0022]

[0023] Where X is the input pixel value, y ij Represents the convolution result of the convolution window. The subscripts i and j represent the pixel in the i-th row and j-th column of the image. m and n are used to traverse the rows and columns of the convolution window, respectively. w is the corresponding weight value in the convolution window of size K×K.

[0024] Preferably, the ReLU module is composed of 8 ReLU activation function units, and the ReLU activation function unit adopts a multiplexer to set the highest bit of the data as a sign bit as a selection signal of the multiplexer. When the highest bit of the signal is 1, it means that the input is less than 0, and the ReLU activation function outputs 0. When the highest bit of the signal is 0, it means that the input is greater than 0, and the ReLU activation function outputs the signal itself.

[0025] Preferably, the control module is composed of a state machine, including the following four states:

[0026] IDLE: Standby state. When the data in the BRAM storage module starts to be output, the Start signal is output and the state enters State1 to start network calculation.

[0027] State0: In this state, the calculation of the first layer Conv+ReLU layer of the convolutional network is performed. After the calculation is completed, the S0_done signal is output and the state enters State1;

[0028] State1: In this state, the Conv+ReLU layer calculations of layers 2-7 are performed, and after the calculations are completed, the S1_done signal is output and State2 is entered;

[0029] State2: In this state, the last convolutional layer is calculated and the calculated data is stored in the BRAM storage module. After completion, the S2_done signal is output and the IDLE state is entered.

[0030] Preferably, the State0 stage uses 1 convolution calculation module, 8 weight register groups and 1 data register group for parallel calculation to realize the calculation of the first layer Conv+ReLU layer of the convolutional network; wherein the convolution calculation module includes a convolution module and a ReLU module; the State0 stage specifically performs the following operations:

[0031] The BRAM storage module outputs the original image data and the first layer weight data of the network to the windowing module and the weight register group respectively. The windowing module receives the original image data and sends it to the data register group. When one data register and eight weight register groups are filled, the data register group and the weight register group output nine image data and seventy-two weight data in parallel, and output a juanji1_start signal. Eight 3×3 convolution kernel operation units simultaneously calculate the nine image data, and the calculation results are cached in the fifo module. The convolution kernel operation unit includes one multiplication and addition unit and one ReLU activation function unit, and the convolution calculation module includes eight convolution kernel operation units.

[0032] When the first layer of convolution calculation is completed, the S0_done signal is output, and the fifo cache module sends the 8 feature maps output by the first layer of convolution calculation to the register group module for the second layer of convolution calculation.

[0033] Preferably, the State1 stage uses 1 convolution calculation module, 8 weight register groups and 8 data register groups to perform Conv+ReLU layer calculations of the 2nd to 7th layers; the State1 stage specifically performs the following operations:

[0034] After receiving the signal S0_done of State0 completion, enter the State1_1 state in State1, which is used to fill the data register group and the weight register group. When the 8 data register groups and the 8 weight register groups are filled with the image data output by the fifo buffer module and the first group of weight data of the second layer of the network output by the BRAM storage module, the counter count2 is increased by 1, indicating that the first convolution layer calculation of the State1 state is performed, and the juanji2_start signal is output to enter the State1_2 state;

[0035] In the State1_2 state, the eight channels of a 3×3 convolution kernel are calculated at the same time, and the results of the eight convolution kernel operation units are added to an output value of a 3×3×8 convolution kernel, which is stored in the fifo cache module; after completing the calculation of a 3×3×8 convolution kernel, the counter count1 is increased by 1;

[0036] When count1 is not equal to 8, enter the State1_3 state to replace the convolution kernel weight data, use the next set of convolution weight data of the second layer of the network in the BRAM storage module to fill the weight register group, and after filling, enter State1_2 to calculate the next convolution kernel;

[0037] If count1 is equal to 8, it means that the calculation of the 8 convolution kernels of the convolution layer is completed, and the state of State1_1 is re-entered. At the same time, the counter count1 is cleared and the calculation is restarted. The feature map data of the previous layer cached in the fifo module and the first group of weight data of the next layer of the network in the BRAM storage module are filled in the data register group and the weight register group. The counter count2 is increased by 1, and then the state of State1_2 is returned to calculate the current convolution layer;

[0038] The Conv+ReLU layer calculations of layers 2-7 are performed through state machines State1_1, State1_2, and State1_3; when count2 is equal to 6, it means that the calculations of the 6 convolutional layers in this stage are all completed, and the S1_done signal is output to enter State2;

[0039] The State2 stage uses the convolution module, 8 weight register groups, and 8 data register groups to perform the 8th layer Conv layer calculation; the State2 stage specifically performs the following operations:

[0040] After receiving the signal S1_done of State1 completion, enter the State2_1 state in State2, which is used to fill the data register group and the weight register group. When the 8 data register groups and the 8 weight register groups are filled with the image data output by the fifo buffer module and the network layer 8 weight data output by the BRAM storage module, respectively, enter the State2_2 state;

[0041] In the State2_2 state, a 3×3×8 convolution kernel is used to perform convolution to restore the two-dimensional image, and the restored image data is output to the BRAM storage module. After completion, the State2_done signal is output.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention preprocesses the data to be reconstructed in the lensless imaging system through a designed lightweight preprocessing network (CNN-ISP-net) to improve the effect of reconstructing images in the lensless imaging system; in addition, the present invention implements a corresponding hardware framework on FPGA based on the designed ISP lightweight convolutional neural network, which can speed up the calculation speed of the network by using fewer computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 This is the framework diagram of the lightweight CNN-ISP-net of the present invention;

[0045] Figure 2 This is the hardware architecture diagram of CNN-ISP-net of the present invention;

[0046] Figure 3 This is a schematic diagram of the State1 state machine of the present invention;

[0047] Figure 4 This is a schematic diagram of the window opening process of the present invention;

[0048] Figure 5 This is the RTL diagram of the windowing module of the present invention;

[0049] Figure 6 It is a schematic diagram of register group data movement of the present invention;

[0050] Figure 7 Figure 2 is a diagram of the register set RLT of the present invention;

[0051] Figure 8 It is a waveform diagram of the register group of the present invention;

[0052] Fig. 9 This is a schematic diagram of a fifo buffer module of the present invention;

[0053] Fig.10 This is a schematic diagram of ReLU of the present invention;

[0054] Fig.11 Schematic diagram of the convolution kernel computing unit of the present invention;

[0055] Fig.12 This is the RTL diagram of a single convolution kernel operation unit of the present invention;

[0056] Fig.13 Schematic diagram of the convolution module of the present invention;

[0057] Fig.14 This is a comparison diagram of preprocessing in one embodiment of the present invention;

[0058] Fig.15 A comparison diagram of preprocessing software and hardware results in one embodiment of the present invention;

[0059] Fig.16 A resource report diagram in one embodiment of the present invention;

[0060] Fig.17 This is a timing report diagram in one embodiment of the present invention. DETAILED DESCRIPTION

[0061] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0062] The present invention proposes an FPGA implementation architecture for a lightweight preprocessing network of a lens-free imaging system; Figure 1 As shown in the figure, the lightweight preprocessing network (CNN-ISP-net) includes 7 Conv+ReLU layers and one Conv layer; the first layer uses 8 filters of size 3×3×1 for convolution to generate 8 feature maps, the 2nd to 7th Conv+ReLU layers use 8 filters of size 3×3×8, and the last Conv layer uses a convolution kernel of size 3×3×8 to complete the reconstruction output of the image.

[0063] According to the lightweight CNN-ISP-net network, the CNN-ISP-net hardware architecture designed by the present invention is as follows Figure 2 The FPGA implementation architecture includes a control module, a BRAM storage module, a fifo cache module, a register group module, a windowing module, a convolution module, and a ReLU module;

[0064] The control module is responsible for scheduling the input and output of image data and weight data, and controlling the start and end of tasks of each module;

[0065] The BRAM storage module is used to store original image data, weight data and pre-processed output data;

[0066] The fifo buffer module is used to buffer the intermediate calculation data of the convolution module and the ReLU module;

[0067] The register group module includes a weight register group and a data register group, which are respectively used to cache weight data and image data involved in convolution calculation; the weight data comes from the BRAM storage module; the image data comes from the BRAM storage module and the fifo cache module;

[0068] The windowing module is used to realize the stepping operation of convolution together with the data register group;

[0069] The convolution module and the ReLU module are responsible for the computational tasks of the lightweight preprocessing network.

[0070] The windowing module is connected to the BRAM storage module and the fifo buffer module respectively through a multiplexer MUX to realize the scheduling of two data sources.

[0071] The following is a detailed description of each module.

[0072] 1. Control module

[0073] The control module of the present invention is composed of a state machine, which is responsible for scheduling the calculation process and storage actions of the network. The state machine is shown in Table 1 and includes four states.

[0074] Table 1 Control module status

[0075]

[0076] In the State0 stage, the BRAM storage module outputs the original image data and weight data to the windowing module and weight register group respectively. The windowing module receives the original image data and sends it to the data register group. The convolution module and the ReLU module complete the convolution calculation of the input image with 8 convolution kernels. In this stage, 1 convolution calculation module, 8 weight register groups and 1 data register group are used for parallel calculation, where the convolution calculation module includes the convolution module and the ReLU module.

[0077] When the system enable signal Start is pulled high, 35 clocks later, 1 data register group is filled and starts waiting for the weight register group to be filled. After 72 clocks, 8 weight register groups are filled. At this time, the data register group and the weight register group output 9 image data and 72 weight data in parallel, and output the juanji1_start signal. 8 3×3 convolution kernel operation units calculate the 9 image data at the same time, and each clock generates a value of a feature map. The calculation result is cached in the fifo module. When a layer of convolution calculation is completed, the S0_done signal is output, and the fifo cache module sends the 8 feature maps output by the first layer of convolution calculation to the register group module for the second layer of convolution calculation.

[0078] Then enter the State1 stage, a total of 6 convolution layers need to be completed, and each convolution layer is convolutionally calculated by 8 3×3×8 filters. Therefore, this stage requires the use of 8 convolution kernel operation units, 8 weight register groups and 8 data register groups, among which the data register and window module are used to complete the convolution step task of 8 feature maps. The present invention designs the State1 stage to be completed by a three-state state machine, such as Figure 3 .

[0079] After receiving the signal S0_done of State0 completion, the state 1_1 in State1 is entered. This state is used to fill the data register group and the weight register group. After 72 clocks, the register group is filled. At this time, the counter count2 is increased by 1 (the initial value of the counter is 0), indicating that the first convolution layer calculation of the State1 state is performed, and the juanji2_start signal is output to enter the State1_2 state. In the State1_2 state, the 8 channels of a 3×3 convolution kernel are calculated at the same time. The results of the 8 convolution kernel operation units are added to an output value of a 3×3×8 convolution kernel, which is stored in the fifo cache module. After 256 clocks, the calculation of a feature map is completed, and the counter count1 is increased by 1, indicating that the calculation of a 3×3×8 convolution kernel is completed. There are 8 convolution kernels in each layer, so when count1 is not equal to 8, it enters the State1_3 state to replace the convolution kernel weight, that is, use the new convolution weight to fill the weight register group. After the filling is completed, it enters the State1_2 to calculate the next convolution kernel. If count1 is equal to 8, it means that the calculation of the 8 convolution kernels of this convolution layer is completed, and then it re-enters the State1_1 state, and the counter count1 is cleared, and the calculation is restarted to fill the feature map data and new weight data cached in the fifo module of the previous layer. When count2 is equal to 6, it means that the calculation of the 6 convolution layers of this stage is completed, and the S1_done signal is output to enter State2.

[0080] In the State2 stage, a 3×3×8 convolution layer calculation is completed to restore the image to two dimensions. This stage is similar to the State1_1 state and State1_2 state in the State1 stage. First, the data register and weight register are filled. When the register is filled, a 3×3×8 convolution kernel is used to convolve and restore it to a two-dimensional image. The restored image data is directly output to the BRAM storage module, and the State2_done signal is output after completion.

[0081] 2. Storage unit

[0082] The storage unit of the present invention is composed of a BRAM storage module, a register group module and a fifo cache module. The BRAM storage module stores trained network parameters and original image data, and the fifo cache module is used to cache the feature map output by each layer of the network for calculation of the next layer of the network.

[0083] In the BRAM storage module, the BRAM weight parameters use eight-bit fixed-point numbers to store 256 original image data and 3600 convolution weight parameters. The image data is stored in the BRAM with a data width of 8 bits and a depth of 256. The BRAM that stores the convolution kernel parameters has an address width of 8 bits and a depth of 3600.

[0084] In order to process the local part of the image, the window module and the register group module are designed to facilitate the application of the convolution filter to each area and increase the parallelism of the circuit. The window module and the register group module jointly complete the convolution step operation and output the data involved in the convolution calculation in parallel.

[0085] The windowing process of size 3×3 is also the row buffering process of the shift register, such as Figure 4 As shown in the figure, the depth of the buffer area Buffer is the number of pixel values ​​in the image row minus 1. The BRAM outputs the image data serially. After the row buffer, the first column X in the window can be 00 , X 10 , X 20 Output simultaneously. Similarly, each column in the convolution window can be output in parallel, which is convenient for subsequent parallel operations.

[0086] Based on the above windowing operation, two additional register groups and logic controllers (included in the control module) are opened. Each register group includes nine registers, one of which is used to cache data in a convolution window, and the other register group is used to cache convolution weights, such as Figure 6 At the first clock, the logic controller sets X 00 , X 10 , X 20 Stored in the first column of the data register group, at the second clock, the logic controller will X 01 , X 11 , X 21 Stored in the first column of the data register bank, and X 00 , X 10 , X 20 Move to the second column, and the data register group completes storing the nine numbers in the window at the third clock. Since there is no windowing module before the weight register group, the weight register is filled in serially.

[0087] The data in the data register group is shifted after three clocks, while the weight in the weight register remains unchanged, which means that the convolution window slides in the image with a step size of 1. When the convolution filter completes the convolution of the image, the weight parameters of the next filter are reloaded for convolution.

[0088] exist Figure 8In the figure, the red box represents the movement of data in different columns in the register group, and the data register outputs the nine cached data at the same time.

[0089] The fifo in the fifo cache module supports simultaneous reading and writing, and is controlled by two enable signals r_en and w_en. Fig. 9 As shown. In this cache module, 16 fifos are used, which are divided into group1 and group2, each with 8 fifos, and the depth of each fifo is 256. Group1 and group2 are used to cache the output and input of the convolution layer. When the fifo in group1 is used as the feature map output cache, the fifo in group2 is used as the feature map input cache. Similarly, when group1 is used as the feature map input cache, group2 is used as the feature map output cache. When the fifo group is used to cache the output feature map data, the r_en enable signal is pulled high. When the fifo group is used to cache the feature map data output by the convolution layer, the w_en enable signal is pulled high.

[0090] 3 Computing Module

[0091] The present invention needs to perform a large number of 2D convolutions in realizing the lightweight ISP convolutional network, so the efficiency of the system depends largely on the efficiency of the convolution hardware. The convolution calculation module of the present invention includes two parts: a convolution module and a ReLU module.

[0092] The convolution module consists of 8 multiplication and addition units. The multiplication and accumulation array in the multiplication and addition unit reads data from the data register group for convolution calculation. The multiplication and accumulation array performs the following basic operations in a single clock cycle:

[0093]

[0094] Where X is the input pixel value, y ij It represents the convolution result of the convolution window. The subscripts i and j represent the pixel points in the i-th row and j-th column of the image. m and n are used to traverse the rows and columns of the convolution window respectively. w is the corresponding weight value in the K×K window. The multiplication-accumulation array circuit includes nine multipliers. The adder is used to accumulate the outputs of the nine multipliers. The final output result is the convolution result of the convolution window.

[0095] The ReLU activation function expression is as follows:

[0096] f(x)=max(0,x) (2)

[0097] This means that for an input x, if x is positive, the output is x; if x is negative, the output is 0.

[0098] The ReLU module consists of 8 ReLU activation function units. Only one multiplexer is needed to implement the ReLU activation function unit. The highest bit of the data is set as the sign bit as the selection signal of the multiplexer. When the highest bit of the signal is 1, it means that the input is less than 0, and the ReLU activation function outputs 0. When the highest bit of the signal is 0, it means that the input is greater than 0, and the ReLU activation function outputs the signal itself. The schematic diagram of this unit is as follows Fig.10 shown.

[0099] Based on the design of the multiplication-addition unit and the ReLU activation function unit, the circuit of a convolution kernel operation unit is as follows: Fig.11 As shown in the figure, each convolution kernel operation unit consists of a multiplication and addition unit (nine multipliers and eight adders) plus a ReLU activation function unit. The data and weight at the input end are the data and weight output in parallel by the register group. In State0, the data source is the BRAM storage module, and in other states, the data source is the 8 fifos in the fifo cache module. The convolution calculation module consists of 8 convolution kernel operation units in total to improve the parallelism of the architecture.

[0100] 4 Results analysis and conclusion

[0101] In the present invention, the FPGA outputs the image data of the point spread function PSF saved in binary format, and after storing the data in the PC, it is converted into decimal data for display, such as Fig.14 As shown in the figure, the original PSF is blurred, underexposed, and lacks details, while the PSF after network correction has clear texture. Then, the corrected PSF is sent to the Le-ADMM algorithm on the PC for image reconstruction. The reconstruction result is shown in the figure. Fig.15 As shown in the figure, it can be seen that the correction effect achieved by the hardware side and the software side is very close, but because the software side is a floating point calculation, the peak signal-to-noise ratio (PSNR) is 2.19db higher. This experiment verifies the correctness of the CNN-ISP-net hardware system.

[0102] The resource report of the present invention is as follows Fig.16 As shown, the report explains that the present invention uses less computing resources, and the resource utilization is not high because the network architecture is relatively simple and the parallelism of the convolution calculation is not fully increased in the hardware circuit design.

[0103] Fig.17This is the timing report of the system. Under the condition that the circuit working clock period is constrained to 5ns, the circuit's worst negative timing margin (WNS) is 0.420ns, the worst hold timing margin (WHS) is 0.087ns, and the worst pulse width margin (WPWS) is 6.370ns. All three indexes are positive, indicating that the circuit still maintains a positive timing margin in the worst case, that is, it meets the clock constraint requirements.

[0104] In general, this circuit architecture uses fewer computing resources and can meet the operating clock frequency of 200M. The PSF corrected by this circuit can be sent to the Le-ADMM reconstruction algorithm to reconstruct a better-performing image.

[0105] The above are preferred embodiments of the present invention. Any changes made according to the technical solution of the present invention, as long as the resulting functions do not exceed the scope of the technical solution of the present invention, belong to the protection scope of the present invention.

Claims

1. FPGA implementation architecture of lightweight preprocessing network for lens-free imaging system, characterized by: The lightweight preprocessing network includes 7 Conv+ReLU layers and one Conv layer; the FPGA implementation architecture includes a control module, a BRAM storage module, a fifo cache module, a register group module, a windowing module, a convolution module, and a ReLU module; The control module is responsible for scheduling the input and output of image data and weight data, and controlling the start and end of tasks of each module; The BRAM storage module is used to store original image data, weight data and pre-processed output data; The fifo buffer module is used to buffer the intermediate calculation data of the convolution module and the ReLU module; The register group module includes a weight register group and a data register group, which are respectively used to cache weight data and image data involved in convolution calculation; the weight data comes from the BRAM storage module; the image data comes from the BRAM storage module and the fifo cache module; The windowing module is used to realize the stepping operation of convolution together with the data register group; The convolution module and the ReLU module are responsible for the computational tasks of the lightweight preprocessing network.

2. The FPGA implementation architecture of the lightweight preprocessing network for the lens-free imaging system according to claim 1, characterized in that: The windowing module is connected to the BRAM storage module and the fifo buffer module respectively through a multiplexer MUX to realize the scheduling of two data sources.

3. The FPGA implementation architecture of the lightweight preprocessing network for the lens-free imaging system according to claim 1, characterized in that: The first layer of the lightweight preprocessing network uses 8 filters of size 3×3×1 for convolution to generate 8 feature maps; the 2nd to 7th Conv+ReLU layers use 8 filters of size 3×3×8; and the last Conv layer uses a convolution kernel of size 3×3×8 to complete the reconstruction output of the image.

4. The FPGA implementation architecture of the lightweight preprocessing network for the lens-free imaging system according to claim 3, characterized in that: The windowing process of the windowing module is implemented through the row caching process of the shift register; the weight register group and the data register group each include 9 registers, wherein the 9 registers of the data register group are used to cache data in a convolution window, and the 9 registers of the weight register group are used to cache corresponding convolution weights.

5. The FPGA implementation architecture of the lightweight preprocessing network for the lens-free imaging system according to claim 3, characterized in that: The fifo cache module uses 16 fifos, which are divided into two groups, group 1 and group 2, with 8 fifos in each group; Group1 and group2 are used to cache the output and input of the convolution operation. When the fifo in group1 is used as the convolution operation feature map output cache, the fifo in group2 is used as the convolution operation feature map input cache. When group1 is used as the convolution operation feature map input cache, group2 is used as the convolution operation feature map output cache. The fifo supports simultaneous reading and writing and is controlled by two enable signals r_en and w_en; when the fifo is used to cache the feature map data input by the convolution operation, the r_en enable signal is pulled high, and when the fifo is used to cache the feature map data output by the convolution operation, the w_en enable signal is pulled high.

6. The FPGA implementation architecture of the lightweight preprocessing network for the lens-free imaging system according to claim 3, characterized in that: The convolution module is composed of 8 multiplication-addition units, which form a multiplication-accumulation array through nine multipliers and eight adders. The multiplication-accumulation array performs the following operations in a single clock cycle: Where X is the input pixel value, y ij Represents the convolution result of the convolution window. The subscripts i and j represent the pixel in the i-th row and j-th column of the image. m and n are used to traverse the rows and columns of the convolution window, respectively. w is the corresponding weight value in the convolution window of size K×K.

7. The FPGA implementation architecture of the lightweight preprocessing network for the lens-free imaging system according to claim 3, characterized in that: The ReLU module is composed of 8 ReLU activation function units. The ReLU activation function unit adopts a multiplexer to set the highest bit of the data as the sign bit as the selection signal of the multiplexer. When the highest bit of the signal is 1, it means that the input is less than 0, and the ReLU activation function outputs 0. When the highest bit of the signal is 0, it means that the input is greater than 0, and the ReLU activation function outputs the signal itself.

8. The FPGA implementation architecture of the lightweight preprocessing network for the lens-free imaging system according to claim 3, characterized in that: The control module is composed of a state machine, including the following four states: IDLE: Standby state. When the data in the BRAM storage module starts to be output, the Start signal is output and the state enters State1 to start network calculation. State0: In this state, the calculation of the first layer Conv+ReLU layer of the convolutional network is performed. After the calculation is completed, the S0_done signal is output and the state enters State1; State1: In this state, the Conv+ReLU layer calculations of layers 2-7 are performed, and after the calculations are completed, the S1_done signal is output and State2 is entered; State2: In this state, the last convolutional layer is calculated and the calculated data is stored in the BRAM storage module. After completion, the S2_done signal is output and the IDLE state is entered.

9. The FPGA implementation architecture of the lightweight preprocessing network for the lens-free imaging system according to claim 8, characterized in that: The State0 stage uses 1 convolution calculation module, 8 weight register groups and 1 data register group for parallel calculation to realize the calculation of the first layer Conv+ReLU layer of the convolutional network; the convolution calculation module includes the convolution module and the ReLU module; the State0 stage specifically performs the following operations: The BRAM storage module outputs the original image data and the first layer weight data of the network to the windowing module and the weight register group respectively. The windowing module receives the original image data and sends it to the data register group. When one data register and eight weight register groups are filled, the data register group and the weight register group output nine image data and seventy-two weight data in parallel, and output a juanji1_start signal. Eight 3×3 convolution kernel operation units simultaneously calculate the nine image data, and the calculation results are cached in the fifo module. The convolution kernel operation unit includes one multiplication and addition unit and one ReLU activation function unit, and the convolution calculation module includes eight convolution kernel operation units. When the first layer of convolution calculation is completed, the S0_done signal is output, and the fifo cache module sends the 8 feature maps output by the first layer of convolution calculation to the register group module for the second layer of convolution calculation.

10. The FPGA implementation architecture of the lightweight pre-processing network for the lens-free imaging system according to claim 8, characterized in that: The State1 stage uses 1 convolution calculation module, 8 weight register groups, and 8 data register groups to perform Conv+ReLU layer calculations for layers 2-7. The State1 stage specifically performs the following operations: After receiving the signal S0_done of State0 completion, enter the State1_1 state in State1, which is used to fill the data register group and the weight register group. When the 8 data register groups and the 8 weight register groups are filled with the image data output by the fifo buffer module and the first group of weight data of the second layer of the network output by the BRAM storage module, the counter count2 is increased by 1, indicating that the first convolution layer calculation of the State1 state is performed, and the juanji2_start signal is output to enter the State1_2 state; In the State1_2 state, the eight channels of a 3×3 convolution kernel are calculated simultaneously. The results of the eight convolution kernel operation units are added together to form an output value of the 3×3×8 convolution kernel, which is stored in the fifo cache module. After completing the calculation of a 3×3×8 convolution kernel, the counter count1 is incremented by 1. When count1 is not equal to 8, enter the State1_3 state to replace the convolution kernel weight data, use the next set of convolution weight data of the second layer of the network in the BRAM storage module to fill the weight register group, and after filling, enter State1_2 to calculate the next convolution kernel; If count1 is equal to 8, it means that the calculation of the 8 convolution kernels of the convolution layer is completed, and the state of State1_1 is re-entered. At the same time, the counter count1 is cleared and the calculation is restarted. The feature map data of the previous layer cached in the fifo module and the first group of weight data of the next layer of the network in the BRAM storage module are filled in the data register group and the weight register group. The counter count2 is increased by 1, and then the state of State1_2 is returned to calculate the current convolution layer; The Conv+ReLU layer calculations of layers 2-7 are performed through state machines State1_1, State1_2, and State1_3; when count2 is equal to 6, it means that the calculations of the 6 convolutional layers in this stage are all completed, and the S1_done signal is output to enter State2; The State2 stage uses the convolution module, 8 weight register groups, and 8 data register groups to perform the 8th layer Conv layer calculation; the State2 stage specifically performs the following operations: After receiving the signal S1_done of State1 completion, enter the State2_1 state in State2, which is used to fill the data register group and the weight register group. When the 8 data register groups and the 8 weight register groups are filled with the image data output by the fifo buffer module and the network layer 8 weight data output by the BRAM storage module, respectively, enter the State2_2 state; In the State2_2 state, a 3×3×8 convolution kernel is used to perform convolution to restore the image into a two-dimensional image, and the restored image data is output to the BRAM storage module. After completion, the State2_done signal is output.

Images