Infrared image super-resolution system and device based on ZYNQ

Abstract

The invention relates to the technical field of image processing and computer vision, and discloses a ZYNQ-based infrared image super-resolution system and device, and the system comprises an infrared image collection module, a ZYNQ infrared image processing module which comprises an ARM processor and an FPGA hardware acceleration module, a high-resolution infrared image output module, and a high-resolution infrared image display module. The infrared image acquisition module acquires a low-resolution infrared image through an infrared camera, image data acquired by the infrared camera is transmitted to the ZYNQ infrared image processing module through a digital interface, and an image processing algorithm in the ZYNQ infrared image processing module begins to perform super-resolution processing on the image data; and the processed high-resolution image data is sent to the high-resolution infrared image output module through the digital interface, is converted into HDMI through the VGA interface, and is output to the high-resolution infrared image display module. Through the optimized super-resolution algorithm, image details are effectively recovered, and the image resolution is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing and computer vision, and in particular relates to a ZYNQ-based infrared image super-resolution system and device. Background Art

[0002] Infrared image super-resolution technology is a technology that uses infrared detection equipment to measure the infrared radiation difference of objects and obtains high-quality infrared images through algorithm processing. Infrared images are widely used in military, medical, remote sensing, hydrology and video surveillance fields due to their unique imaging advantages. However, infrared images generally have problems such as lack of details and low resolution, which affects their application in industry.

[0003] The existing infrared image super-resolution methods mainly include the following:

[0004] Interpolation-based super-resolution methods: such as nearest neighbor interpolation, bicubic interpolation, and bilinear interpolation. These methods are simple to use and computationally efficient, but they cannot effectively restore image details and are prone to artifacts at the edges of images. Interpolation methods cannot predict the lost high-frequency information based on the local structure and content of infrared images, resulting in discontinuous edges and incomplete contours.

[0005] Super-resolution method based on degradation model: Use image prior knowledge and global reconstruction constraints to construct a cost function to keep the image edge clear, but the calculation is large and requires a lot of prior knowledge, which is not suitable for terminal devices. The degradation model method has high computational complexity and poor real-time performance, and is not suitable for systems with high real-time requirements.

[0006] Super-resolution methods based on deep learning: such as SRCNN, GAN, ESRGAN and other algorithms. These methods can capture complex structures and details of images, but the network structure is redundant and lacks lightweight design, making it difficult to deploy on platforms with limited resources. The network structure of deep learning algorithms is redundant and has not been lightweighted. The platform development cycle is long, the algorithm portability is poor, and the subjective visual effect is subject to noise interference.

[0007] The reasons for these shortcomings are: the interpolation method simply fills pixels without considering the image structure and content; the degradation model method involves complex optimization problems and the solution process is time-consuming; the deep learning algorithm does not consider the platform characteristics, does not perform lightweight processing, and the loss function does not consider the characteristics of human vision.

[0008] In order to solve these problems, the present invention designs an infrared image super-resolution system and device based on the ZYNQ platform. By improving the ESRGAN network structure, introducing the TV-LOSS loss function and performing model lightweight processing, the infrared image super-resolution performance is effectively improved, and the practicability and portability of the system are improved. Summary of the invention

[0009] The present invention aims to solve the problems that the infrared image super-resolution methods in the above-mentioned prior art often have low computational efficiency, poor restored image quality, and cannot effectively restore detailed features; the existing deep learning super-resolution algorithms usually consume large computing resources and are difficult to deploy on resource-constrained embedded platforms; the existing algorithm models are often difficult to port between different platforms and meet real-time processing requirements due to hardware compatibility and real-time processing capability limitations; and the infrared image display quality in the prior art is often poor, with technical problems such as noise and unevenness.

[0010] To achieve the above object, the present invention provides the following technical solutions:

[0011] An infrared image super-resolution system based on ZYNQ includes the following components:

[0012] Infrared image acquisition module, responsible for acquiring low-resolution infrared images;

[0013] The ZYNQ infrared image processing module includes an ARM processor and an FPGA hardware acceleration module for executing image processing algorithms. The image processing algorithm is responsible for super-resolution reconstruction of the acquired low-resolution infrared images to obtain high-resolution images.

[0014] High-resolution infrared image output module, responsible for outputting the reconstructed high-resolution infrared image;

[0015] A high-resolution infrared image display module, used for displaying high-resolution infrared images;

[0016] The infrared image acquisition module collects low-resolution infrared images through an infrared camera. The image data collected by the infrared camera is transmitted to the ZYNQ infrared image processing module through a digital interface. The image processing algorithm in the ZYNQ infrared image processing module starts to perform super-resolution processing on the image data. The processed high-resolution image data is sent to the high-resolution infrared image output module through a digital interface and output to the high-resolution infrared image display module through HDMI.

[0017] Preferably, the ARM processor communicates with the FPGA hardware acceleration module via an internal AXI4 bus.

[0018] Preferably, the FPGA hardware acceleration module includes an I2C control module, an infrared image super-resolution module, a YUV2RGB color gamut conversion module, an infrared image cache module, a RAW conversion RGB module, a deserialization module and a CMOS control module.

[0019] Preferably, the ARM processor controls the start, stop and data reading of the infrared image acquisition module through I2C, the I2C control module is used to communicate with the infrared image acquisition module, the image data of the infrared image acquisition module is processed by the deserialization module, and then communicates with the FPGA hardware acceleration module through the I2C control module; the deserialization module is used to process the serial data received from the infrared image acquisition module.

[0020] Preferably, the image collected by the infrared image acquisition module is converted into RGB format through the YUV2RGB color gamut conversion module and sent to the infrared image super-resolution module. The infrared image super-resolution module, as the core of the FPGA hardware acceleration module, is responsible for executing the super-resolution algorithm. Its input is a low-resolution infrared image and its output is a high-resolution infrared image. The infrared image cache module is used to store and process infrared image data for processing by the infrared image super-resolution module. The RAW conversion RGB module converts the original RAW format image data into RGB format. The CMOS control module is used to control the working status of the infrared image acquisition module.

[0021] Preferably, the ARM processor is connected to the PL DOR3 of the FPGA hardware acceleration module via PSDOR3 for data transmission and control.

[0022] Preferably, the high-resolution image data in RGB format processed by the infrared image super-resolution module is sent to the high-resolution infrared image output module through a digital interface.

[0023] Preferably, the high-resolution infrared image output module sends the image data to the high-resolution infrared image display module via HDMI or other display interfaces to display the image.

[0024] A ZYNQ-based infrared image super-resolution device, applied to the above-mentioned ZYNQ-based infrared image super-resolution system, comprising:

[0025] ZYNQ development board, used to store algorithms, intermediate data and output images;

[0026] Digital interface, used for data transmission between the infrared camera and the ZYNQ development board, and between the ZYNQ development board and the output device;

[0027] Power management module, providing stable power for the entire device;

[0028] Memory, used for temporary storage of data and algorithm models;

[0029] The infrared camera is connected to the ZYNQ development board through a digital interface. The image data captured by the infrared camera is transmitted to the ZYNQ development board for processing. The ARM processor and FPGA hardware acceleration module inside the ZYNQ development board are interconnected through an internal bus. The ARM processor is responsible for the main processing logic of the algorithm, and the FPGA hardware acceleration module realizes hardware acceleration.

[0030] A ZYNQ-based infrared image super-resolution design method, which is applied to the above-mentioned ZYNQ-based infrared image super-resolution device, comprises the following steps:

[0031] S1. System selection and construction: Select the appropriate ZYNQ development board and display device, use the black gold ZYNQ-7020 development board as the ZYNQ development board, and select the lightweight OV7725 module as the infrared image acquisition module;

[0032] S2. Dataset preparation: Use the public CVC09 and CVC14 image sets to create a new training set train-CVC containing 1000 infrared images, and preprocess the images, including resizing, shuffling, and degrading.

[0033] S3. Model parameter setting: Set the model parameters according to the device conditions, including the size of the low-resolution image, GPU path, number of iterations, and learning rate, and import the dataset into the model.

[0034] S4, model training and optimization: Use the CVC09 dataset to train the parameter model, initialize the model parameters, perform generalization training, introduce the attention mechanism and the IAT-ESRGAN generative network with multi-scale feature fusion, use the tripping activation function to optimize the network performance, set the loss function, including TV-LOSS and SSIM-Loss, perform iterative training, and save the training model;

[0035] S5, Model migration and deployment: Load the trained model and set it to inference mode, convert the model to ONNX format, and then convert it to wk format adapted to the front-end device, configure the development environment on the ZYNQ development board, install the tool chain and Python image processing library, copy the model file to the ZYNQ development board, download the bitstream through JTAG, connect to the OV7725 infrared image camera, configure the CMOS infrared image sensor, initialize and read image data through the SCCB bus; perform data processing and preprocessing on the ZYNQ-7020 development board, and use FPGA for hardware acceleration;

[0036] S6. System integration and demonstration: The processed high-resolution infrared image is converted into an HDMI image output interface through the VGA output interface and output to the display device, completing the packaging and demonstration of the system and showing the actual operation effect of the infrared image super-resolution system.

[0037] Compared with the prior art, the technical effects and advantages of the present invention are:

[0038] The ZYNQ-based infrared image super-resolution system and device, by introducing TV-LOSS and SSIM-LOSS loss functions, can more effectively control the recovery of image details and structures in the super-resolution reconstruction process, thereby improving the quality and objective evaluation indicators of the reconstructed image; through the improved loss function, the generated image is closer to the real scene in visual effect, meets the needs of human vision, and improves the user's viewing experience; the application of SSIM-LOSS and TV-LOSS in the field of infrared image super-resolution reconstruction and their impact on image quality should be included in the protection scope of the present invention.

[0039] By closely combining the attention mechanism (AM) with the newly proposed activation function triping, the present invention can more effectively extract the high-frequency information of the image and deeply mine the feature information at different levels; the improved design of this discriminator architecture and its ability to deeply mine high-frequency information and feature information should be included in the protection scope of the present invention.

[0040] Using ELU instead of LeakyReLU as the activation function of the discriminator can provide a more efficient and faster image processing process, thereby improving the overall efficiency of the system. The optimization of this activation function and its improvement of image processing efficiency should be included in the protection scope of the present invention.

[0041] The present invention performs lightweight processing on the image super-resolution algorithm so that it can run on the ZYNQ platform, which reduces the memory requirements of the algorithm and improves the efficiency and adaptability of the system. The functional division of the infrared image acquisition module and the high-resolution image output module enables the system to process data more efficiently and improves the overall work efficiency. The lightweight processing of the algorithm, the adaptability of the ZYNQ platform, and the optimization of the functional division should be included in the protection scope of the present invention.

[0042] In summary, the present invention not only improves the image quality and visual effects through the infrared image super-resolution system implemented on the ZYNQ platform, but also improves the efficiency and processing capacity of the system through technical optimization. These technical effects and advantages are not only reflected in the quality of image reconstruction, but also in the lightweight, real-time performance of the algorithm and the working efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1This is a working diagram of the infrared image super-resolution system and device of the present invention;

[0044] Figure 2 It is a flow chart of the infrared super-resolution algorithm of the present invention;

[0045] Figure 3 It is a schematic diagram of the flow of the generative adversarial network algorithm of the present invention;

[0046] Figure 4 Schematic diagram of the discriminator architecture of the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] The following is combined with Figure 1-4 To further explain this application in detail,

[0049] The present application embodiment discloses a ZYNQ-based infrared image super-resolution system, comprising the following components:

[0050] Infrared image acquisition module, responsible for acquiring low-resolution infrared images; the infrared image acquisition module adopts OV7725 module;

[0051] The ZYNQ infrared image processing module includes an ARM processor and an FPGA hardware acceleration module. The ARM processor communicates with the FPGA hardware acceleration module through the internal AXI4 bus to execute the image processing algorithm. The image processing algorithm is responsible for super-resolution reconstruction of the collected low-resolution infrared images to obtain high-resolution images.

[0052] High-resolution infrared image output module, responsible for outputting the reconstructed high-resolution infrared image;

[0053] A high-resolution infrared image display module, used for displaying high-resolution infrared images;

[0054] The infrared image acquisition module collects low-resolution infrared images through an infrared camera. The image data collected by the infrared camera is transmitted to the ZYNQ infrared image processing module through a digital interface. The image processing algorithm in the ZYNQ infrared image processing module starts to perform super-resolution processing on the image data. The processed high-resolution image data is sent to the high-resolution infrared image output module through a digital interface and output to the high-resolution infrared image display module through HDMI.

[0055] The FPGA hardware acceleration module includes an I2C control module, an infrared image super-resolution module, a YUV2RGB color gamut conversion module, an infrared image buffer module, a RAW conversion RGB module, a deserialization module and a CMOS control module.

[0056] The ARM processor controls the start, stop and data reading of the infrared image acquisition module through I2C. The I2C control module is used to communicate with the infrared image acquisition module. After the image data of the infrared image acquisition module is processed by the deserialization module, it communicates with the FPGA hardware acceleration module through the I2C control module; the deserialization module is used to process the serial data received from the infrared image acquisition module.

[0057] The image collected by the infrared image acquisition module is converted into RGB format through the YUV2RGB color gamut conversion module and sent to the infrared image super-resolution module. As the core of the FPGA hardware acceleration module, the infrared image super-resolution module is responsible for executing the super-resolution algorithm. Its input is a low-resolution infrared image and its output is a high-resolution infrared image. The infrared image cache module is used to store and process infrared image data for processing by the infrared image super-resolution module. The RAW conversion RGB module converts the original RAW format image data into RGB format. The CMOS control module is used to control the working status of the infrared image acquisition module.

[0058] The ARM processor is connected to the PL DOR3 of the FPGA hardware acceleration module through PSDOR3 for data transmission and control. The high-resolution image data in RGB format processed by the infrared image super-resolution module is sent to the high-resolution infrared image output module through a digital interface. The high-resolution infrared image output module sends the image data to the high-resolution infrared image display module through HDMI or other display interfaces to display the image.

[0059] A ZYNQ-based infrared image super-resolution device, applied to the above-mentioned ZYNQ-based infrared image super-resolution system, comprising:

[0060] ZYNQ development board, used to store algorithms, intermediate data and output images; ZYNQ development board uses the black gold ZYNQ-7020 development board;

[0061] Digital interface, used for data transmission between the infrared camera and the ZYNQ development board, and between the ZYNQ development board and the output device;

[0062] Power management module, providing stable power for the entire device;

[0063] Memory, used for temporary storage of data and algorithm models;

[0064] The infrared camera is connected to the ZYNQ development board through a digital interface. The image data captured by the infrared camera is transmitted to the ZYNQ development board for processing. The ARM processor and FPGA hardware acceleration module inside the ZYNQ development board are interconnected through an internal bus. The ARM processor is responsible for the main processing logic of the algorithm, and the FPGA hardware acceleration module realizes hardware acceleration.

[0065] The design of a ZYNQ-based infrared image super-resolution system includes the following steps:

[0066] (1) Select a suitable ZYNQ development board and display device. The present invention selects the black gold ZYNQ-7020 as the system development board and selects the lighter OV7725 module as the infrared camera module of the present device;

[0067] (2) Prepare a dataset suitable for the model. Use the public image sets CVC09 and CVC14 and shuffle their order to form a new training set train-CVC containing 1000 infrared images.

[0068] (3) Set model parameters according to equipment conditions;

[0069] (4) Train the model based on the CVC09 and CVC14 image sets and save the training parameters;

[0070] (5) Transplant the above infrared image super-resolution algorithm to the selected ZYNQ platform;

[0071] (6) Package and demonstrate infrared image super-resolution systems and devices;

[0072] The overall implementation process of the present invention is as follows Figure 1 shown.

[0073] First, each low-resolution infrared image in the data set is input into a trained system, which uses a convolutional neural network to extract features from the entire image. Then, the attention mechanism is introduced to quickly obtain the corresponding feature map. Finally, the tripping activation function proposed in the present invention is used to map the positive values ​​in the feature map to the interval (0.5, 1). The flowchart of the infrared super-resolution algorithm of the present invention is as follows: Figure 2 The generative adversarial network algorithm process used in the present invention is shown in the attached Figure 3 As shown in the figure, the generator generates part of the picture, and then the real picture and the picture obtained by the generator are given to the discriminator for discrimination, and then upsampled through the convolution layer and output a real and effective high-resolution infrared image.

[0074] Step (2) specifically includes:

[0075] Step 1: Use the street view infrared image database as a dataset sample.

[0076] Step 2: Use the nearest neighbor interpolation method to scale the sample low-resolution image to a uniform size of 100*100. The formula is as follows:

[0077] f(x new ,y new )=f(x old ,y old )

[0078] Among them, the pixel coordinates before scaling are respectively represented by x old and old Indicates that the height of the image is represented by h old Indicates the width with w old Indicates that the scaled pixel coordinates are represented by x new and new Indicates that the height of the image is represented by h new Indicates the width with w new f(x old ,y old ),f(x new ,y new ) represent the pixel value of the point before and after scaling.

[0079] Step 3: Shuffle the 500 training images in the dataset, downsample them four times using bicubic downsampling as low-resolution infrared images for training, and form a new dataset of 1,000 infrared images as the training set train-CVC for this experiment. All images in the datasets train-CVC, test-CVC09, and test-CVC14 are processed into 3-channel images using a program.

[0080] Step 4: Store all images in a folder and change the folder name to Image. Store all labeled files in the newly created test folder and ensure that the absolute path of the training image is correct.

[0081] Step (3) specifically includes:

[0082] Step 1: Set the low-resolution image size to 100*100 to make it the same size as the image size of the dataset.

[0083] Step 2: According to the device conditions, set the GPU path package and install the corresponding version of cuda and pytroch network, and set the number of iterations and learning rate.

[0084] Step 3: Import the dataset from step 4 in step (2) into the model.

[0085] Step (4) specifically includes:

[0086] Step 1: Use the public dataset CVC09 released by FIR to train the parameter model, generalize the model, initialize the model parameters, and reduce the training time.

[0087] Step 2: Input the processed data set into the system.

[0088] Step 3: The attention mechanism works. The low-resolution image is input into the IAT-ESRGAN generation network of the multi-scale feature fusion of the present invention. The (Attentional mechanisms) are used to help the convolutional neural network to quickly extract features of the entire image. First, the channel attention (CA) performs average adaptive pooling and maximum adaptive pooling operations on the input feature map T (H×W×C) along the spatial axis, and then passes through the multi-layer perceptron MLP to obtain the average set feature and the maximum set feature. The feature details on the channel are extracted as a whole. The focus is on extracting the high-frequency feature details on the channel. The matrix is ​​added to obtain the feature C(1×C). C(1×C) is nonlinearly mapped into a weight coefficient W with a value range of (0, 1) after the sigmoid activation function. c (1×C), W c (1×C) performs matrix dot multiplication on the input feature map T(H×W×C) and weights each corresponding channel of T(H×W×C). The mathematical formula is as follows:

[0089] Ca(T)=W c (T)·T

[0090] W c (T)=σ(MLP(AvgPool(T))+MLP(MaxPool(T)))

[0091] σ is the sigmoid function, T is the feature input map with height H, width W and number of channels C. MLP generally consists of a convolutional layer and a Relu activation layer.

[0092] Sigmoid is a commonly used hidden layer activation function. The present invention proposes a tripping activation function to replace the traditional sigmoid function to speed up the network. The formula is as follows:

[0093]

[0094] The discriminator architecture of the generative adversarial network of the present invention is as follows Figure 4 As shown;

[0095] Step 4:

[0096] The image features extracted by upsampling are enlarged, and one convolution kernel is replaced with two convolution kernels of different sizes (1*1, 3*3,) in each basic residual block. The features are then added and compressed into a 1*1 convolution kernel output for multi-channel fusion. In the present invention, ELU replaces LeakyReLu as the activation function, and the result after the activation function is input to the next layer. The ELU formula is as follows:

[0097]

[0098] Step 5: Set the loss function for each image generated by the generator

[0099] L G =L percep +λL Ra G +ξL ssim +αL TV ;

[0100] Wherein, α is the coefficient of TV-LOSS, ξ is the coefficient of SSIM-Loss, and TV-LOSS is a regularization term introduced by the present invention that is more in line with the reality of human vision. The TV-LOSS formula is as follows:

[0101]

[0102] SSIM-Loss is an important parameter for comparing the similarity between two images. It comprehensively evaluates brightness l, contrast c and structure s. The SSIM-Loss formula is as follows:

[0103] SSIM(x,y)=[l(x,y) α *c(x,y) β *s(x,y) γ ]

[0104]

[0105]

[0106] L percep To reduce the perceptual loss, a clearer edge and a more beautiful effect are provided. -8 , λ is The coefficient of is 8×10 in the invented model. -2 , ξ is taken as 1.

[0107] Step 5: Perform iterative training according to the parameter values ​​set in step 4, set batch-size to 32, epochs to 1000, validation batch-size to 1, initial learning rate to 0.0004, and use Adam as optimizer. Obtain a better detection model and save the training model.

[0108] Step (5) specifically includes:

[0109] The first step is to load the model that finally saves the training parameters in the above step (4) through a script file and set it to inference mode. Subsequently, the model is exported through the simulated input tensor preset by the script file, and the script file converts the pt format model of the training parameters into a trained ONNX format model. The caffe format model is converted into a wk format model adapted by the front-end device because in the process of converting the ONNX format model to the Caffe format, the Mul operator involved in the matrix dot multiplication operation needs to be converted accordingly. This process usually includes mapping the Mul operator in ONNX to an equivalent operator supported by Caffe, such as Eltwise or Scale, to ensure that the converted model can correctly perform the matrix dot multiplication operation in the Caffe framework.

[0110] The second step is to configure the required development environment on the image processing platform of the present invention, install the Linaro Toolchain suitable for ZYNQ, and configure the operating system of ZYNQ to PetaLinux so that it can mirror the configuration algorithm. Then install python3.9 on the platform through PetaLinux, and install the Python image processing library (such as Pillow, OpenCV, Matplotlib, NumPy, etc.) required by the algorithm on ZYNQ to process the low-resolution infrared image.

[0111] The third step is to connect ZYNQ to the host storing the WK format model, use USB as the communication interface, and run the Linux system configured on ZYNQ. Execute the 'df -h' command in the Linux terminal to identify the mount point of the USB device, and use the 'mount' command to mount it to the appropriate location in the file system. When the USB device is successfully mounted,

[0112] Use the 'cp' command to copy the WK format model file from the host device to the target directory of the ZYNQ development board (for example, the ` / home / root / models / ` directory). After the copy is complete, unmount the USB device to avoid data corruption or loss. In this process, Digilent JTAG-HS3 is connected to ZYNQ as a JTAG interface, which is mainly used to download FPGA chip configuration files or firmware. After the file transfer is completed, JTAG continues to download the bitstream required for this infrared image super-resolution device through JTAG.

[0113] The fourth step is to connect the OV7725 infrared image camera as the low-resolution infrared image acquisition module of this device. The present invention selects a CMOS infrared image sensor to collect images. It is small in size, highly integrated, and has low working voltage and power consumption. The chip is configured through the SCCB bus. The infrared camera is initialized first. After ZYNQ is powered on, it waits for 3ms for the level to stabilize, and then sends the initialization start flag. The register is configured through the SCCB bus protocol. After the configuration is completed, the initialization end flag signal is sent, and it also serves as the beginning of ZYNQ to start reading image data.

[0114] In the fifth step, after the low-resolution infrared image is read into the ZYNQ-7020 development board through the fourth step, the processing system (PS) will store the data in the external DDR3 memory. This provides enough space to store a large amount of image data. The data is transferred from the PS to the FPGA logic through the lightweight PS-to-FPGA AXI bridge. This lightweight bridge is suitable for processing smaller data blocks and control information. The data is pre-processed in the FPGA, and the pre-processed data is stored in the BRAM of the FPGA. Due to the limited capacity of BRAM, it is only used to store data in the pre-processing stage. The pre-processed data is stored in the external SDRAM. SDRAM is mainly used to store pre-processed data and provides sufficient memory space. The processed data is transferred from the FPGA back to the PS through the FPGA-to-PS AXI bridge. This bridge supports large-scale data transfer and is suitable for processing large batches of data streams. DDR3 memory is used as the storage of convolution weights for the IAT-ESRGAN algorithm. The weight data required for the convolution multiplication and addition operation is stored in DDR3, which supports the efficient execution of the convolution operation. In this way, the ZYNQ-7020 SoC is able to efficiently manage and process the large amounts of data required by the IAT-ESRGAN algorithm, optimizing low-resolution infrared image processing and convolution operations.

[0115] Step (6) specifically includes: outputting the output obtained by the ZYNQ infrared image processing module through the VGA output interface. Converting the signal from a digital signal to an RGB analog signal output. Through the image decoding chip ADV7123, the VGA output interface is selected to be matched with various types of displays as high-resolution infrared image output. The main signals displayed by the VGA interface of the FPGA module in Zynq include line synchronization signals, field synchronization signals, and red, green, and blue (RGB) primary color signals. The high-resolution infrared image signal is decomposed to obtain independent RGB three-color signals. These signals are converted into analog signals by the ADV7123 video digital-to-analog converter to generate digital image data of the FPGA, and are output to the display device through the VGA interface. With the cooperation of the line synchronization and field synchronization signals, the signals of the three RGB channels are transmitted in sequence to complete the display of the high-resolution infrared image.

[0116] In summary, the present invention accelerates the feature extraction speed of the infrared image super-resolution algorithm of the device through AM (attention mechanism), improves the ability of the network to extract effective high-frequency information details, and makes the infrared super-resolution model equipped have stronger information feature detail extraction capability.

[0117] The present invention introduces the ELU activation function to replace the Relu function in the discriminator structure, thereby increasing the nonlinearity of the discriminator network and improving the learning ability of the discriminator. ELU provides faster classification speed and higher accuracy, thereby improving the performance of the infrared image super-resolution algorithm.

[0118] In the generator loss function, the regularization loss function (TV loss) is introduced to make the high-frequency information of the image not too abrupt, SSIM Loss is used to replace L1 Loss to make the generated results closer to human subjective visual effects, and each loss term is fine-tuned to achieve the best network SR performance.

[0119] The infrared image super-resolution device integrates a low-resolution infrared image acquisition module, an infrared image super-resolution module, and a high-resolution infrared image output module. It has a high level of integration and configures the infrared image super-resolution algorithm on a ZYNQ with an ARM chip and an FPGA hardware acceleration module.

[0120] The present invention improves the existing infrared image super-resolution system and device, optimizes the network structure redundancy of the algorithm installed in the existing infrared image super-resolution system and device, and the lightweight algorithm can be configured on the ZYNQ platform. In addition, the present invention also improves the image feature extraction link, accelerating the convergence speed of the network.

[0121] Based on the existing infrared image super-resolution reconstruction technology, the present invention introduces the TV-LOSS regularization term into the loss function, making the generated high-resolution infrared image more consistent with human visual perception. In addition, combined with the use of the attention mechanism, the present invention realizes the rapid extraction of high-frequency information of the image, and has achieved significant improvement in speed compared with the existing algorithms.

[0122] The present invention proposes an effective solution to the problem that the activation function of the discriminator is not optimized in the prior art. By using the ELU function to improve the activation function of the discriminator, not only the accuracy of the discriminator is improved, but also the performance of the entire infrared image super-resolution reconstruction system is further enhanced.

[0123] In terms of the configuration of the infrared image super-resolution system and device, the present invention performs lightweight processing on the infrared image super-resolution algorithm so that the algorithm can be efficiently deployed on the ZYNQ platform with a simple operating system and rich logical computing resources. This improvement greatly improves the working efficiency of the infrared image super-resolution system and device, and provides a more efficient and convenient solution for practical applications.

[0124] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A ZYNQ-based infrared image super-resolution system, characterized in that: The system consists of the following components: Infrared image acquisition module, responsible for acquiring low-resolution infrared images; The ZYNQ infrared image processing module includes an ARM processor and an FPGA hardware acceleration module for executing image processing algorithms. The image processing algorithm is responsible for super-resolution reconstruction of the acquired low-resolution infrared images to obtain high-resolution images. High-resolution infrared image output module, responsible for outputting the reconstructed high-resolution infrared image; A high-resolution infrared image display module, used for displaying high-resolution infrared images; The infrared image acquisition module collects low-resolution infrared images through an infrared camera. The image data collected by the infrared camera is transmitted to the ZYNQ infrared image processing module through a digital interface. The image processing algorithm in the ZYNQ infrared image processing module starts to perform super-resolution processing on the image data. The processed high-resolution image data is sent to the high-resolution infrared image output module through a digital interface and output to the high-resolution infrared image display module through HDMI.

2. The infrared image super-resolution system based on ZYNQ according to claim 1, characterized in that: The ARM processor communicates with the FPGA hardware acceleration module through the internal AXI4 bus.

3. The infrared image super-resolution system based on ZYNQ according to claim 2, characterized in that: The FPGA hardware acceleration module includes an I2C control module, an infrared image super-resolution module, a YUV2RGB color gamut conversion module, an infrared image buffer module, a RAW conversion RGB module, a deserialization module and a CMOS control module.

4. The infrared image super-resolution system based on ZYNQ according to claim 3, characterized in that: The ARM processor controls the start, stop and data reading of the infrared image acquisition module through I2C. The I2C control module is used to communicate with the infrared image acquisition module. After the image data of the infrared image acquisition module is processed by the deserialization module, it communicates with the FPGA hardware acceleration module through the I2C control module; the deserialization module is used to process the serial data received from the infrared image acquisition module.

5. The infrared image super-resolution system based on ZYNQ according to claim 3, characterized in that: The image collected by the infrared image acquisition module is converted into RGB format through the YUV2RGB color gamut conversion module and sent to the infrared image super-resolution module. As the core of the FPGA hardware acceleration module, the infrared image super-resolution module is responsible for executing the super-resolution algorithm. Its input is a low-resolution infrared image and its output is a high-resolution infrared image. The infrared image cache module is used to store and process infrared image data for processing by the infrared image super-resolution module. The RAW conversion RGB module converts the original RAW format image data into RGB format. The CMOS control module is used to control the working status of the infrared image acquisition module.

6. The ZYNQ-based infrared image super-resolution system according to claim 3, characterized in that: The ARM processor is connected to the PL DOR3 of the FPGA hardware acceleration module through PSDOR3 for data transmission and control.

7. The ZYNQ-based infrared image super-resolution system according to claim 3, characterized in that: The high-resolution image data in RGB format processed by the infrared image super-resolution module is sent to the high-resolution infrared image output module through a digital interface.

8. The ZYNQ-based infrared image super-resolution system according to claim 7, characterized in that: The high-resolution infrared image output module transmits the image data to the high-resolution infrared image display module via the VGA image output interface to HDMI to realize image display.

9. A ZYNQ-based infrared image super-resolution device, applied to a ZYNQ-based infrared image super-resolution system as claimed in any one of claims 1 to 4, characterized in that: include: ZYNQ development board, used to store algorithms, intermediate data and output images; Digital interface, used for data transmission between the infrared camera and the ZYNQ development board, and between the ZYNQ development board and the output device; Power management module, providing stable power for the entire device; Memory, used for temporary storage of data and algorithm models; The infrared camera is connected to the ZYNQ development board through a digital interface. The image data captured by the infrared camera is transmitted to the ZYNQ development board for processing. The ARM processor and FPGA hardware acceleration module inside the ZYNQ development board are interconnected through an internal bus. The ARM processor is responsible for the processing logic of the algorithm, and the FPGA hardware acceleration module realizes hardware acceleration.

10. A ZYNQ-based infrared image super-resolution design method, characterized in that: The method is applied to the ZYNQ-based infrared image super-resolution device described in claim 9, comprising the following steps: S1. System selection and construction: Select the ZYNQ development board and display device, use the black gold ZYNQ-7020 development board as the ZYNQ development board, and select the lightweight OV7725 module as the infrared image acquisition module; S2. Dataset preparation: Use the public CVC09 and CVC14 image sets to create a new training set train-CVC containing 1000 infrared images, and preprocess the images, including resizing, shuffling, and degrading. S3, model parameter setting: set the model parameters according to the device conditions, including the size of the low-resolution image, GPU path, number of iterations and learning rate, and import the data set into the model; S4, model training and optimization: Use the CVC09 dataset to train the parameter model, initialize the model parameters, perform generalization training, introduce the attention mechanism and the IAT-ESRGAN generative network with multi-scale feature fusion, use the tripping activation function to optimize the network performance, set the loss function, including TV-LOSS and SSIM-Loss, perform iterative training, and save the training model; S5, Model migration and deployment: Load the trained model and set it to inference mode, convert the model to ONNX format, and then convert it to wk format adapted to the front-end device, configure the development environment on the ZYNQ development board, install the tool chain and Python image processing library, copy the model file to the ZYNQ development board, download the bitstream through JTAG, connect to the OV7725 infrared image camera, configure the CMOS infrared image sensor, initialize and read image data through the SCCB bus; perform data processing and preprocessing on the ZYNQ-7020 development board, and use FPGA for hardware acceleration; S6. System integration and demonstration: The processed high-resolution infrared image is converted into an HDMI image output interface through the VGA output interface and output to the display device, completing the packaging and demonstration of the system and showing the actual operation effect of the infrared image super-resolution system.

Images