Infrared image super-resolution system and device based on ZYNQ

Abstract

The present application relates to the technical field of image processing and computer vision, and discloses an infrared image super-resolution system and device based on ZYNQ, which comprises an infrared image acquisition module, a ZYNQ infrared image processing module comprising an ARM processor and a FPGA hardware acceleration module, a high-resolution infrared image output module and a high-resolution infrared image display module; the infrared image acquisition module acquires low-resolution infrared images through an infrared camera, the image data collected by the infrared camera is transmitted to the ZYNQ infrared image processing module through a digital interface, the image processing algorithm in the ZYNQ infrared image processing module starts to perform super-resolution processing on the image data, the processed high-resolution image data is sent to the high-resolution infrared image output module through a digital interface, and is converted into HDMI through a VGA interface and output to the high-resolution infrared image display module. Through the optimized super-resolution algorithm, the image details are effectively restored, and the image resolution is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of image processing and computer vision, and particularly relates to an infrared image super-resolution system and device based on ZYNQ. BACKGROUND

[0002] The infrared image super-resolution technology is a technology for obtaining a high-quality infrared image by using an infrared detection device to measure the infrared radiation difference of an object and through algorithm processing. The infrared image has a wide application in the fields of military, medicine, remote sensing, hydrology and video monitoring due to its unique imaging advantages. However, the infrared image generally has problems of detail loss and low resolution, which affects its application in the industry.

[0003] The existing infrared image super-resolution methods mainly include the following methods:

[0004] The interpolation-based super-resolution method: such as the nearest neighbor interpolation, bicubic interpolation and bilinear interpolation method, these methods are simple and easy to use, and have high calculation efficiency, but cannot effectively restore the image details, and are easy to produce artifacts at the image edge. The interpolation method cannot predict the lost high-frequency information according to the local structure and content of the infrared image, resulting in discontinuous edges and incomplete contours.

[0005] The super-resolution method based on the degradation model: the cost function is constructed by using the image priori knowledge and the global reconstruction constraint to maintain the clear image edge, but the calculation amount is large, a large amount of priori knowledge is required, and it is not suitable for terminal equipment. The degradation model method has high calculation complexity and poor real-time performance, and is not suitable for systems with high real-time requirements.

[0006] The super-resolution method based on deep learning: such as the SRCNN, GAN, ESRGAN and other algorithms, these methods can capture the complex structure and details of the image, but the network structure is redundant, lacks lightweight design, and is difficult to deploy on a platform with limited resources. The network structure of the deep learning algorithm is redundant, and the lightweight processing is not done, the platform development cycle is long, the algorithm portability is poor, and the subjective visual effect has noise interference.

[0007] The reasons for these shortcomings are: the interpolation method simply fills the pixels without considering the image structure and content; the degradation model method involves a complex optimization problem, and the solving process is time-consuming; the deep learning algorithm does not consider the platform characteristics, and does not perform lightweight processing, and the loss function does not consider the human visual characteristics.

[0008] In order to solve these problems, the application designs an infrared image super-resolution system and device based on the ZYNQ platform, improves the ESRGAN network structure, introduces the TV-LOSS loss function and performs model lightweight processing, effectively improves the infrared image super-resolution performance, and improves the practicability and portability of the system. SUMMARY

[0009] The present invention addresses the following issues in existing infrared image super-resolution methods: low computational efficiency, poor image quality, and inability to effectively recover detailed features; high computational resource consumption in existing deep learning super-resolution algorithms, making them difficult to deploy on resource-constrained embedded platforms; limitations in hardware compatibility and real-time processing capabilities, making it difficult to port these algorithms across different platforms and meet real-time processing requirements; and the poor display quality of infrared images in existing technologies, characterized by noise and unevenness.

[0010] To achieve the above objectives, the present invention provides the following technical solution:

[0011] An infrared image super-resolution system based on ZYNQ includes the following components:

[0012] The infrared image acquisition module is responsible for acquiring low-resolution infrared images;

[0013] The ZYNQ infrared image processing module includes an ARM processor and an FPGA hardware acceleration module, which are used to execute image processing algorithms. The image processing algorithms are responsible for super-resolution reconstruction of the acquired low-resolution infrared images to obtain high-resolution images.

[0014] The high-resolution infrared image output module is responsible for outputting the reconstructed high-resolution infrared image;

[0015] A high-resolution infrared image display module for displaying high-resolution infrared images;

[0016] The infrared image acquisition module acquires low-resolution infrared images through an infrared camera. The image data acquired by the infrared camera is transmitted to the ZYNQ infrared image processing module through a digital interface. The image processing algorithm in the ZYNQ infrared image processing module begins to perform super-resolution processing on the image data. The processed high-resolution image data is sent to the high-resolution infrared image output module through a digital interface and then output to the high-resolution infrared image display module via HDMI.

[0017] Preferably, the ARM processor communicates with the FPGA hardware acceleration module via the internal AXI4 bus.

[0018] Preferably, the FPGA hardware acceleration module includes an I2C control module, an infrared image super-resolution module, a YUV2RGB color gamut conversion module, an infrared image buffer module, a RAW to RGB conversion module, a deserialization module, and a CMOS control module.

[0019] As preferred, the ARM processor controls the start, stop and data reading of the infrared image acquisition module through I2C, the I2C control module is used for communication with the infrared image acquisition module, the image data of the infrared image acquisition module is communicated with the FPGA hardware acceleration module through the I2C control module after being processed by the deserializing module; the deserializing module is used for processing the serial data received from the infrared image acquisition module.

[0020] As preferred, the image collected by the infrared image acquisition module is converted from YUV format to RGB format by the YUV2RGB color gamut conversion module and sent to the infrared image super-resolution module, the infrared image super-resolution module is the core of the FPGA hardware acceleration module, responsible for executing the super-resolution algorithm, its input is a low-resolution infrared image, and its output is a high-resolution infrared image, the infrared image buffer module is used for storing the infrared image data in the process of storage and processing, for the infrared image super-resolution module, the RAW to RGB module converts the original RAW format image data to RGB format, and the CMOS control module is used for controlling the working state of the infrared image acquisition module.

[0021] As preferred, the ARM processor is connected with the PL DOR3 of the FPGA hardware acceleration module through the PSDOR3, for data transmission and control.

[0022] As preferred, the high-resolution image data in RGB format processed by the infrared image super-resolution module is sent to the high-resolution infrared image output module through a digital interface.

[0023] As preferred, the high-resolution infrared image output module sends the image data to the high-resolution infrared image display module through HDMI or other display interfaces, to realize the display of the image.

[0024] An infrared image super-resolution device based on ZYNQ, applied to the infrared image super-resolution system based on ZYNQ, comprising:

[0025] A ZYNQ development board, used for storing algorithms, intermediate data and output images;

[0026] A digital interface, used for data transmission between the infrared camera and the ZYNQ development board, and between the ZYNQ development board and the output device;

[0027] A power management module, used for providing stable power supply for the whole device;

[0028] A memory, used for temporarily storing data and algorithm models;

[0029] The infrared camera is connected with the ZYNQ development board through a digital interface, image data captured by the infrared camera is transmitted to the ZYNQ development board for processing, the ARM processor and the FPGA hardware acceleration module in the ZYNQ development board are interconnected through an internal bus, the ARM processor is responsible for the main processing logic of the algorithm, and the FPGA hardware acceleration module realizes hardware acceleration.

[0030] A ZYNQ-based infrared image super-resolution design method, which is applied to the ZYNQ-based infrared image super-resolution device and comprises the following steps:

[0031] S1, system selection and construction: selecting a suitable ZYNQ development board and display device, taking a black gold ZYNQ-7020 development board as the ZYNQ development board, and selecting a lightweight OV7725 module as an infrared image acquisition module;

[0032] S2, data set preparation: using the public CVC09 and CVC14 image sets, creating a new training set train-CVC containing 1000 infrared images, and pre-processing the images, including uniform size, random order and degradation processing;

[0033] S3, model parameter setting: setting model parameters according to device conditions, including the size of low-resolution images, GPU path, iteration number and learning rate, and importing the data set into the model.

[0034] S4, model training and optimization: training the parameter model using the CVC09 data set, initializing the model parameters, performing generalization training, introducing the attention mechanism and the multi-scale feature fusion IAT-ESRGAN generation network, optimizing the network performance using the triping activation function, setting the loss function including TV-LOSS and SSIM-Loss, performing iterative training, and saving the trained model;

[0035] S5, model transplantation and deployment: loading the trained model and setting it to inference mode, converting the model to ONNX format, then to wk format suitable for front-end devices, configuring the development environment on the ZYNQ development board, installing the tool chain and Python image processing library, copying the model file to the ZYNQ development board, downloading the bit stream through JTAG, accessing the OV7725 infrared image camera, configuring the CMOS infrared image sensor, initializing and reading image data through the SCCB bus; performing data processing and preprocessing on the ZYNQ-7020 development board, and using FPGA for hardware acceleration;

[0036] S6, system integration and demonstration: the processed high-resolution infrared image is converted into an HDMI image output interface through a VGA output interface and output to a display device, completing the packaging and demonstration of the system, and showing the actual operation effect of the infrared image super-resolution system.

[0037] Compared with the prior art, the technical effects and advantages of the present application are:

[0038] The infrared image super-resolution system and device based on ZYNQ can more effectively control the recovery of image details and structures in the super-resolution reconstruction process by introducing TV-LOSS and SSIM-LOSS loss functions, thereby improving the quality and objective evaluation index of the reconstructed image; through the improved loss function, the generated image is closer to the real scene in visual effect, meets the needs of human vision, and improves the user's viewing experience; the application of SSIM-LOSS and TV-LOSS in the field of infrared image super-resolution reconstruction and their influence on image quality should be included in the protection scope of the present application.

[0039] By closely combining the attention mechanism (AM) with the newly proposed activation function triping, the present application can more effectively extract high-frequency information of the image and deeply mine feature information at different levels; the improved design of this discriminator architecture and its deep mining ability for high-frequency information and feature information should be included in the protection scope of the present application.

[0040] Using ELU instead of LeakyReLU as the activation function of the discriminator can provide a more efficient and faster image processing process, thereby improving the overall efficiency of the system. The optimization of this activation function and its improvement of image processing efficiency should be included in the protection scope of the present application.

[0041] The present application performs lightweight processing on the image super-resolution algorithm, so that it can run on the ZYNQ platform, which reduces the memory requirement of the algorithm and improves the efficiency and adaptability of the system. The functional division of the infrared image acquisition module and the high-resolution image output module enables the system to more efficiently process data and improve the overall work efficiency. The lightweight processing of the algorithm, the adaptability of the ZYNQ platform and the optimization of the functional division should be included in the protection scope of the present application.

[0042] In summary, the infrared image super-resolution system implemented on the ZYNQ platform not only improves the image quality and visual effect, but also improves the efficiency and processing capacity of the system through technical optimization. These technical effects and advantages not only reflect in the quality of image reconstruction, but also reflect in the lightweight, real-time and work efficiency of the system. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1The workflow diagram of the infrared image super-resolution system and device of the present application;

[0044] Figure 2 The algorithm flowchart of the infrared super-resolution algorithm of the present application;

[0045] Figure 3 The algorithm flowchart of the generative adversarial network of the present application;

[0046] Figure 4 The schematic diagram of the discriminator architecture of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0048] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application. Figures 1-4 The present application is further described in detail,

[0049] The embodiment of the present application discloses an infrared image super-resolution system based on ZYNQ, which comprises the following components:

[0050] The infrared image acquisition module is responsible for acquiring low-resolution infrared images. The infrared image acquisition module adopts an OV7725 module.

[0051] The ZYNQ infrared image processing module contains an ARM processor and an FPGA hardware acceleration module. The ARM processor communicates with the FPGA hardware acceleration module through an internal AXI4 bus, and is used to execute an image processing algorithm. The image processing algorithm is responsible for super-resolution reconstruction of the acquired low-resolution infrared images to obtain high-resolution images.

[0052] The high-resolution infrared image output module is responsible for outputting the reconstructed high-resolution infrared images.

[0053] The high-resolution infrared image display module is used to display the high-resolution infrared images.

[0054] The infrared image acquisition module acquires low-resolution infrared images through an infrared camera. The image data collected by the infrared camera is transmitted to the ZYNQ infrared image processing module through a digital interface. The image processing algorithm in the ZYNQ infrared image processing module starts to perform super-resolution processing on the image data. The processed high-resolution image data is sent to the high-resolution infrared image output module through a digital interface, and is output to the high-resolution infrared image display module through HDMI.

[0055] The FPGA hardware acceleration module includes a 12C control module, an infrared image super-resolution module, a YUV2RGB color gamut conversion module, an infrared image cache module, a RAW conversion RGB module, a deserializing module, and a CMOS control module.

[0056] The ARM processor controls the start, stop, and data reading of the infrared image acquisition module through I2C, the 12C control module is used for communication with the infrared image acquisition module, the image data of the infrared image acquisition module is processed through the deserializing module, and then communicated with the FPGA hardware acceleration module through the 12C control module; the deserializing module is used for processing serial data received from the infrared image acquisition module.

[0057] The image collected by the infrared image acquisition module is converted from YUV format to RGB format by the YUV2RGB color gamut conversion module and sent to the infrared image super-resolution module, the infrared image super-resolution module is the core of the FPGA hardware acceleration module and is responsible for executing the super-resolution algorithm, the input of which is a low-resolution infrared image, and the output is a high-resolution infrared image, the infrared image cache module is used for storing and processing infrared image data during the process for the infrared image super-resolution module, the RAW conversion RGB module converts the original RAW format image data into RGB format, and the CMOS control module is used for controlling the working state of the infrared image acquisition module.

[0058] The ARM processor is connected with the PL DOR3 of the FPGA hardware acceleration module through the PSDOR3, for data transmission and control. The high-resolution image data in RGB format processed by the infrared image super-resolution module is sent to the high-resolution infrared image output module through a digital interface. The high-resolution infrared image output module sends the image data to the high-resolution infrared image display module through an HDMI or other display interface, to realize the display of the image.

[0059] An infrared image super-resolution device based on ZYNQ, applied to the infrared image super-resolution system based on ZYNQ, comprising:

[0060] A ZYNQ development board is used for storing algorithms, intermediate data, and output images; the ZYNQ development board adopts a black gold ZYNQ-7020 development board;

[0061] A digital interface is used for data transmission between the infrared camera and the ZYNQ development board, and between the ZYNQ development board and the output device;

[0062] A power management module is used for providing stable power supply for the whole device;

[0063] A memory is used for temporarily storing data and algorithm models;

[0064] The infrared camera is connected with the ZYNQ development board through a digital interface, image data captured by the infrared camera is transmitted to the ZYNQ development board for processing, the ARM processor and the FPGA hardware acceleration module in the ZYNQ development board are interconnected through an internal bus, the ARM processor is responsible for the main processing logic of the algorithm, and the FPGA hardware acceleration module realizes hardware acceleration.

[0065] The design of the infrared image super-resolution system based on ZYNQ comprises the following steps:

[0066] (1) selecting a suitable ZYNQ development board and a display device, in the application, a black gold ZYNQ-7020 is selected as the system development board, and an OV7725 module is selected as the infrared camera module of the device;

[0067] (2) preparing a data set suitable for the model, using the public image sets CVC09 and CVC14, and through shuffling to form a new training set train-CVC containing 1000 infrared images;

[0068] (3) setting the model parameters according to the equipment conditions;

[0069] (4) training the model based on the CVC09 and CVC14 image sets, and saving the training parameters;

[0070] (5) transplanting the above-mentioned infrared image super-resolution algorithm to the selected ZYNQ platform;

[0071] (6) packaging and demonstrating the infrared image super-resolution system and device;

[0072] The overall implementation process of the application is as shown in Figure 1 .

[0073] Firstly, each low-resolution infrared image in the data set is input into the trained system, the system extracts features from the whole image by using a convolutional neural network. Subsequently, the corresponding feature map is quickly obtained by introducing an attention mechanism. Finally, the positive values in the feature map are mapped to the interval (0.5, 1) by using the tripling activation function proposed in the application. The infrared super-resolution algorithm flow chart of the application is as shown in Figure 2 . The generative adversarial network algorithm process used in the application is as shown in the attached Figure 3 , the generator generates part of the picture, then the real picture and the picture obtained by the generator are submitted to the discriminator for discrimination, and then the real and effective high-resolution infrared image is output through the convolution layer upsampling.

[0074] Step (2) specifically comprises:

[0075] First step: Use Street View infrared image database as dataset samples.

[0076] Second step: Use nearest neighbor interpolation method to scale the sample low resolution image to a uniform size of 100*100, as follows:

[0077] f(x new ,y new )=f(x old ,y old )

[0078] wherein the pixel point coordinates before scaling are represented by x old and y old , the height of the image is represented by h old , the width is represented by w old , the pixel point coordinates after scaling are represented by x new and y new , the height of the image is represented by h new , the width is represented by w new . f(x old ,y old ), f(x new ,y new ) represent the pixel values of the point before and after scaling, respectively.

[0079] Third step: Shuffle the order of the 500 training images in the dataset, and use the bicubic down-sampling four times to degrade the low resolution infrared image during training, to form a new dataset with 1000 infrared images as the training set train-CVC of this experiment. Process all images in the datasets train-CVC, test-CVC09 and test-CVC14 into 3-channel images using the program.

[0080] Fourth step: Store all images in a folder and modify the folder name to Image, and store all annotation files in the newly created test folder, and ensure that the absolute path of the training image is correct.

[0081] Step (3) specifically includes:

[0082] First step: Set the low resolution image specification size to 100*100, so that it is the same size as the image size of the dataset.

[0083] Second step: According to the device conditions, set the GPU path package and install the corresponding version of cuda and pytroch network, set the number of iterations and learning rate.

[0084] Third step: Import the dataset of step (2) fourth step into the model.

[0085] Step (4) specifically comprises:

[0086] First step: use the public data set CVC09 published by FIR company to train the parameter model, perform model generalization, initialize the model parameters and reduce the training time length.

[0087] Second step: input the processed data set into the system.

[0088] Third step: the attention mechanism works to input the low-resolution image into the IAT-ESRGAN generation network of the multi-scale feature fusion of the application, and first uses the attention mechanism to help the convolutional neural network to quickly extract the features of the whole image. First, the channel attention (CA) simultaneously performs average adaptive pooling and maximum adaptive pooling operations on the input feature map T (H x W x C) along the spatial axis, and then passes through a multilayer perceptron (MLP) to obtain the average set feature and the maximum set feature The feature details information on the channel is extracted as a whole, The high-frequency feature details information on the channel is extracted as a focus. The and are added to obtain the feature C (1 x C). C (1 x C) is nonlinearly mapped into a weight coefficient W c (1 x C) by a sigmoid activation function c W c (1 x C) is multiplied by the input feature map T (H x W x C) to weight each corresponding channel of T (H x W x C). The mathematical formula is as follows:

[0089] Ca(T)=W c (T)·T

[0090] W c (T)=σ(MLP(AvgPool(T))+MLP(MaxPool(T)))

[0091] σ is a sigmoid function, T is a feature input image with a height of H, a width of W and a channel number of C, and the MLP generally consists of a convolutional layer and a Relu activation layer.

[0092] Sigmoid is a commonly used hidden layer activation function. The application proposes a tripling activation function to replace the traditional sigmoid function to accelerate the network. The formula is as follows:

[0093]

[0094] The discriminator architecture of the generative adversarial network of the application is as Figure 4 shown;

[0095] Step 4:

[0096] The image features extracted after upsampling are amplified. In each basic residual block, one convolutional kernel is replaced with two convolutional kernels of different sizes (1*1, 3*3). The features are then summed and compressed into a single 1*1 convolutional kernel for multi-channel fusion. In this invention, ELU replaces LeakyReLU as the activation function, and the result after activation is input to the next layer. The ELU formula is as follows:

[0097]

[0098] Step 5: Set the loss function for each image generated by the generator.

[0099] L G =L percep +λL Ra G +ξL ssim +αL TV ;

[0100] Where α is the coefficient of TV-LOSS, ξ is the coefficient of SSIM-Loss, and TV-LOSS is a regularization term introduced in this invention that better reflects the realism of human vision. The TV-LOSS formula is as follows:

[0101]

[0102] SSIM-Loss is an important parameter for comparing the similarity between two images. It comprehensively evaluates three aspects: brightness (l), contrast (c), and structure (s). The SSIM-Loss formula is as follows:

[0103] SSIM(x,y)=[l(x,y) α *c(x,y) β *s(x,y) γ ]

[0104]

[0105]

[0106] L percep To mitigate perceived loss, this invention provides sharper edges and a more aesthetically pleasing result. In this invention, α is 2 × 10⁻⁶. -8 λ is The coefficient is taken as 8 × 10 in the invention model. -2 ξ is set to 1.

[0107] Fifth step: iterative training according to the parameter value set in the fourth step, set batch-size to 32, epochs to 1000, validation batch-size to 1, initial learning rate to 0.0004, and optimizer to Adam. Get a better detection model, save the trained model.

[0108] Step (5) specifically includes:

[0109] First, load the model saved in the final step (4) through a script file and set it to inference mode. Then, export the model through the script file preset simulation input tensor, and the script file converts the pt format model with training parameters into an ONNX format model. The caffe format model is converted into the wk format model suitable for the front-end device because the Mul operator involved in the matrix point multiplication operation needs to be converted during the conversion of the ONNX format model into the Caffe format model. This process usually includes mapping the Mul operator in ONNX to the equivalent operator supported by Caffe, such as Eltwise or Scale, to ensure that the converted model can correctly perform the matrix point multiplication operation in the Caffe framework.

[0110] Second, configure the required development environment on the image processing platform of the application, install the tool chain Linaro Toolchain suitable for ZYNQ, and configure the operating system of ZYNQ as PetaLinux so that it can be imaged to configure the algorithm. Then install python3.9 on the platform through PetaLinux, and install the Python image processing library (such as Pillow, OpenCV, Matplotlib, NumPy, etc.) required by the algorithm on ZYNQ to process low-resolution infrared images.

[0111] Third, connect ZYNQ to the host storing the WK format model, use USB as the communication interface, and run the Linux system configured on ZYNQ. Execute the 'df-h' command in the Linux terminal to identify the mounting point of the USB device, and use the'mount' command to mount it to the appropriate location of the file system. When the USB device is successfully mounted,

[0112] Copy the model file in WK format from the host device to the target directory of the ZYNQ development board (for example, the ` / home / root / models / ` directory) using the `cp` command. After the copy is complete, unplug the USB device to avoid data corruption or loss. During this process, the Digilent JTAG-HS3 is connected to the ZYNQ as a JTAG interface, mainly used to download the FPGA chip configuration file or firmware. After the file transfer is complete, the JTAG continues to download the bitstream required for this infrared image super-resolution device through the JTAG.

[0113] Fourth step, access the OV7725 infrared image camera as the low-resolution infrared image acquisition module of the device, the application selects the CMOS infrared image sensor to collect the image, which is small in size, highly integrated, low in working voltage and power consumption, and is configured through the SCCB bus. The infrared camera is first initialized, and after the ZYNQ is powered on, it waits for 3ms to stabilize the level, and then sends a flag to start initialization. The registers are configured through the SCCB bus protocol, and after the configuration is completed, a flag signal indicating the end of initialization is sent, which also serves as the start of reading image data by the ZYNQ.

[0114] Fifth step, when the low-resolution infrared image is read into the ZYNQ-7020 development board through the fourth step, the processing system (PS) will store the data into the external DDR3 memory. This provides enough space to store a large amount of image data. The data is transmitted from the PS to the FPGA logic through a lightweight PS-to-FPGA AXI bridge. This lightweight bridge is suitable for handling small data blocks and control information. Preprocessing of data is performed in the FPGA, and the preprocessed data is stored in the BRAM of the FPGA. Due to the limited capacity of the BRAM, it is only used to store data during the preprocessing stage. The preprocessed data is stored in the external SDRAM. The SDRAM is mainly used to store preprocessed data, providing sufficient memory space. The processed data is transmitted back to the PS from the FPGA through a FPGA-to-PS AXI bridge. This bridge supports large-scale data transmission and is suitable for handling large data streams. The DDR3 memory is used to store the convolution weights of the IAT-ESRGAN algorithm. The weight data required for convolution multiplication and addition operations is stored in the DDR3, supporting efficient execution of convolution operations. In this way, the ZYNQ-7020 SoC can efficiently manage and process the large amount of data required by the IAT-ESRGAN algorithm, optimizing low-resolution infrared image processing and convolution operations.

[0115] The step (6) specifically comprises: outputting the output obtained by the ZYNQ infrared image processing module through a VGA output interface. The signal is converted from a digital signal to an RGB analog signal output. A VGA output interface is selected through an image decoding chip ADV7123, and various types of display devices are selected as high-resolution infrared image outputs. The main signals displayed by the FPGA module VGA interface in the Zynq include a horizontal synchronization signal, a field synchronization signal and red, green and blue (RGB) three primary color signals. The high-resolution infrared image signal is decomposed to obtain independent RGB three-color signals. The signals are converted into analog signals by the ADV7123 video digital-to-analog converter, and are output to the display device through the VGA interface. Under the cooperation of the horizontal synchronization signal and the field synchronization signal, the signals of the RGB three channels are transmitted in turn, and the display of the high-resolution infrared image is completed.

[0116] In conclusion, the application accelerates the feature extraction speed of the infrared image super-resolution algorithm of the device through AM (attention mechanism), improves the ability of the network to extract effective high-frequency information details, and enables the carried infrared super-resolution model to have stronger information feature detail extraction capability.

[0117] The application introduces an ELU activation function instead of a Relu function on the discriminator structure, increases the nonlinearity of the discriminator network, and improves the learning ability of the discriminator. ELU provides faster classification speed and higher accuracy, so that the performance of the infrared image super-resolution algorithm is improved.

[0118] On the generator loss function, a regular term loss function (TV loss) is introduced to make the high-frequency information of the image not too conspicuous, an SSIM Loss is used instead of an L1 Loss to make the generated result closer to the human subjective visual effect, and each loss term is fine-tuned to make the network SR performance optimal.

[0119] On the infrared image super-resolution device, a low-resolution infrared image acquisition module, an infrared image super-resolution module and a high-resolution infrared image output module are integrated, and the infrared image super-resolution algorithm is configured on the ZYNQ with an ARM chip and an FPGA hardware acceleration module.

[0120] The application improves the existing infrared image super-resolution system and device, optimizes the algorithm network structure redundancy of the existing infrared image super-resolution system and device, and configures the lightweight algorithm on the ZYNQ platform. In addition, the application improves the image feature extraction link and accelerates the convergence speed of the network.

[0121] On the basis of the existing infrared image super-resolution reconstruction technology, the TV-LOSS regular term is introduced into the loss function, so that the generated high-resolution infrared image is more consistent with human visual perception. In addition, by using the attention mechanism, the invention realizes the rapid extraction of high-frequency information of the image, and compared with the existing algorithm, the speed is significantly improved.

[0122] In view of the problem that the activation function of the discriminator is not optimized in the prior art, the application provides an effective solution. By using the ELU function to improve the activation function of the discriminator, the accuracy of the discriminator is improved, and the performance of the entire infrared image super-resolution reconstruction system is further enhanced.

[0123] In terms of the configuration of the infrared image super-resolution system and device, the application performs lightweight processing on the infrared image super-resolution algorithm, so that the algorithm can be efficiently deployed on the ZYNQ platform with simple operating system and rich logical computing resources. This improvement greatly improves the working efficiency of the infrared image super-resolution system and device, and provides a more efficient and convenient solution for practical applications.

[0124] Finally, it should be noted that: the above only for the preferred embodiments of the application, and not for limiting the application, although the application has been described in detail with reference to the foregoing embodiments, for those skilled in the art, the technical solution recorded in the foregoing embodiments can be modified, or some technical features can be replaced, any modification, equivalent replacement, improvement, etc. within the spirit and principles of the application, should be included in the protection scope of the application.

Claims

1. A ZYNQ-based infrared image super-resolution system, characterized in that, The system comprises the following components: An infrared image acquisition module, responsible for acquiring low-resolution infrared images; A ZYNQ infrared image processing module, comprising an ARM processor and an FPGA hardware acceleration module, used to execute an image processing algorithm, which is responsible for super-resolution reconstruction of the acquired low-resolution infrared images to obtain high-resolution images; A high-resolution infrared image output module, responsible for outputting the reconstructed high-resolution infrared images; A high-resolution infrared image display module, used to display the high-resolution infrared images; The infrared image acquisition module acquires low-resolution infrared images through an infrared camera, the image data collected by the infrared camera is transmitted to the ZYNQ infrared image processing module through a digital interface, the image processing algorithm in the ZYNQ infrared image processing module starts to perform super-resolution processing on the image data, the processed high-resolution image data is sent to the high-resolution infrared image output module through a digital interface, and is output to the high-resolution infrared image display module through HDMI; The ZYNQ-based infrared image super-resolution system is used to implement the following steps: S1, system selection and construction: selecting a ZYNQ development board and a display device, taking a black gold ZYNQ-7020 development board as the ZYNQ development board, and selecting a lightweight OV7725 module as the infrared image acquisition module; S2, data set preparation: using the public CVC09 and CVC14 image sets, creating a new training set train-CVC containing 1000 infrared images, and pre-processing the images, including uniform size, random order and degradation processing; S3, model parameter setting: setting model parameters according to device conditions, including low-resolution image size, GPU path, iteration number and learning rate, importing the data set into the model; S4, model training and optimization: training the parameter model using the CVC09 data set, initializing the model parameters, performing generalization training, introducing the IAT-ESRGAN generation network with attention mechanism and multi-scale feature fusion, optimizing the network performance using the triping activation function, setting the loss function including TV-LOSS and SSIM-Loss, performing iterative training, and saving the trained model; In the formula, the tripling activation function is as follows: ; S5, model transplantation and deployment: loading the trained model and setting it to inference mode, converting the model to ONNX format, then to wk format suitable for front-end devices, configuring the development environment on the ZYNQ development board, installing the tool chain and Python image processing library, copying the model file to the ZYNQ development board, downloading the bit stream through JTAG, connecting the OV7725 infrared image camera, configuring the CMOS infrared image sensor, initializing and reading image data through the SCCB bus; performing data processing and preprocessing on the ZYNQ-7020 development board, and using FPGA for hardware acceleration; S6, system integration and demonstration: the processed high-resolution infrared image is converted into HDMI image output interface output to the display device through VGA output interface, completing the packaging and demonstration of the system, showing the actual operation effect of the infrared image super-resolution system.

2. The ZYNQ-based infrared image super-resolution system according to claim 1, characterized in that: The ARM processor communicates with the FPGA hardware acceleration module through the internal AXI4 bus.

3. The ZYNQ-based infrared image super-resolution system according to claim 2, characterized in that: The FPGA hardware acceleration module includes a 12C control module, an infrared image super-resolution module, a YUV2RGB color gamut conversion module, an infrared image cache module, a RAW conversion RGB module, a deserializing module, and a CMOS control module.

4. The infrared image super-resolution system based on ZYNQ according to claim 3, characterized in that: The ARM processor controls the start, stop and data reading of the infrared image acquisition module through I2C, the 12C control module is used for communication with the infrared image acquisition module, and the image data of the infrared image acquisition module is communicated with the FPGA hardware acceleration module through the I2C control module after being processed by the deserializing module; the deserializing module is used for processing the serial data received from the infrared image acquisition module.

5. The ZYNQ-based infrared image super-resolution system according to claim 3, characterized in that: The image collected by the infrared image acquisition module is converted into RGB format by the YUV2RGB color gamut conversion module, and then sent to the infrared image super-resolution module; the infrared image super-resolution module is the core of the FPGA hardware acceleration module, responsible for executing the super-resolution algorithm, its input is a low-resolution infrared image, and its output is a high-resolution infrared image; the infrared image cache module is used for storing and processing the infrared image data during the process, so as to be processed by the infrared image super-resolution module; the RAW conversion RGB module converts the original RAW format image data into RGB format; the CMOS control module is used for controlling the working state of the infrared image acquisition module.

6. The ZYNQ-based infrared image super-resolution system according to claim 3, characterized in that: The ARM processor is connected with the PL DOR3 of the FPGA hardware acceleration module through the PS DOR3, which is used for data transmission and control.

7. The ZYNQ-based infrared image super-resolution system according to claim 3, characterized in that: The high-resolution image data in RGB format processed by the infrared image super-resolution module is sent to the high-resolution infrared image output module through a digital interface.

8. The ZYNQ-based infrared image super-resolution system according to claim 7, characterized in that: The high-resolution infrared image output module sends the image data to the high-resolution infrared image display module through the VGA image output interface to HDMl, realizing the display of the image.

9. The ZYNQ-based infrared image super-resolution device is applied to the ZYNQ-based infrared image super-resolution system of any one of claims 1-4, wherein It includes: A ZYNQ development board is used to store algorithms, intermediate data and output images; A digital interface is used for data transmission between the infrared camera and the ZYNQ development board, and between the ZYNQ development board and the output device; A power management module provides stable power supply for the whole device; A memory is used for temporarily storing data and algorithm models; The infrared camera and the ZYNQ development board are connected through a digital interface, and the image data captured by the infrared camera is transmitted to the ZYNQ development board for processing; the ARM processor and the FPGA hardware acceleration module in the ZYNQ development board are interconnected through an internal bus; the ARM processor is responsible for the processing logic of the algorithm, and the FPGA hardware acceleration module realizes hardware acceleration.

Images