An infrared image deblurring method and system for unmanned aerial vehicle platforms

Abstract

The present invention discloses an infrared image deblurring method and system for unmanned aerial vehicle platforms, comprising: obtaining gyroscope information during the period when the unmanned aerial vehicle shoots a motion video by shooting a clear image set and a motion video of a target area through the unmanned aerial vehicle; obtaining a pixel distribution intensity map according to the gyroscope information in combination with a residual connection network structure model; obtaining scene blur information according to the motion video; obtaining a first data set according to the clear image set and the scene blur information; obtaining a training set and a verification set according to the first data set and the pixel distribution intensity map; training a first model according to the training set to obtain a second model set; the first model includes a multi-scale encoder-decoder network model for removing image blur; calculating a first index of all second models according to the verification set and the second model set, and selecting a second model with the largest first index result to perform real-time infrared image deblurring of the unmanned aerial vehicle.

Description

[0001] The present invention relates to the field of infrared deblurring, and in particular to an infrared image deblurring method and system for an unmanned aerial vehicle platform. Background Art

[0002] Infrared thermal imagers are not affected by lighting conditions and can image at night or under low illumination conditions. Infrared thermal imagers also have a certain penetrating ability and can see through smoke to clearly image. UAV platforms equipped with infrared thermal imagers have been widely used in military fields such as battlefield reconnaissance and target identification and attack, as well as in civilian fields such as field search and rescue, forest fire warning, and security monitoring.

[0003] When the UAV platform is equipped with an infrared thermal imager for infrared imaging, the UAV platform needs to move quickly and rotate the infrared thermal imager at a high speed, which leads to the blurring of infrared imaging. Existing infrared deblurring methods mainly include methods based on motion blur kernel evaluation to remove infrared blur and methods based on deep network models to remove infrared blur. In the method based on motion blur kernel evaluation to remove infrared blur, the motion blur kernel corresponding to the image is first obtained by image flow or self-supervised regression, and then the infrared image blur is removed by the motion blur kernel. In the method based on deep network model to remove infrared blur, a fully supervised learning method is usually adopted, that is, a data set training model is constructed by a large number of blur-clear image pairs, and then infrared deblurring is performed according to the model. However, in the method based on motion blur kernel evaluation to remove infrared blur, a large amount of computing resources and time are consumed, which cannot meet the real-time deblurring requirements of the UAV platform, and the motion mode of the UAV platform is nonlinear, which makes it difficult to accurately evaluate the blur kernel; in the method based on deep network model to remove infrared blur, a large number of blur-clear image pairs are difficult to obtain and construct, and there is a large difference between the actual scene and the training data set, which leads to poor generalization of the model obtained by fully supervised training.

[0004] Therefore, a new technical solution is urgently needed to solve the technical problem of how to achieve real-time deblurring of infrared images taken in scenes with rapid movement of the UAV platform and rapid rotation of the infrared thermal imager. Summary of the invention

[0005] The present invention provides an infrared image deblurring method and system for unmanned aerial vehicle platforms, which are used to solve the technical problem of how to achieve real-time deblurring of infrared images taken in scenes where the unmanned aerial vehicle platform moves rapidly and the infrared thermal imager rotates rapidly.

[0006] To achieve the above object, the present invention provides an infrared image deblurring method for a UAV platform, comprising:

[0007] The clear image set and motion video of the target area taken by the drone are used to obtain the gyroscope information during the motion video shooting of the drone; the pixel distribution intensity map is obtained based on the gyroscope information combined with the residual connection network structure model.

[0008] Scene blur information is obtained based on motion video; a first data set is obtained based on a clear image set and the scene blur information; the first data set includes clear images and synthetic blurred images based on a one-to-one pairing of scenes; a training set and a verification set are obtained based on the first data set and a pixel distribution intensity map.

[0009] The first model is trained according to the training set, and the second model obtained by training is saved at intervals of preset rounds to obtain a second model set; the first model includes a multi-scale encoder-decoder network model for removing image blur.

[0010] Calculate the first index of all second models according to the validation set and the second model set. When there is a second model whose first index result is greater than a preset threshold, select the second model with the largest first index result to perform real-time infrared image deblurring of the drone. Calculate the first index of all second models according to the validation set and the second model set, including:

[0011] All second models in the second model set are verified according to the verification set, and the second models output verification results.

[0012] The first indicator is calculated based on the validation set and the validation results.

[0013] The first indicator includes the image restoration quality evaluation index PSNR, which can be expressed as:

[0014] ,

[0015] in, MSE represents the mean square error between the synthesized blurred image and the output result of the second model, where the output result of the second model is the clear image obtained after deblurring by the second model; MAX I Indicates the maximum color value of the synthesized blurred image.

[0016] Preferably, obtaining gyroscope information during the drone shooting of motion video; obtaining a pixel distribution intensity map based on the gyroscope information combined with the residual connection network structure model includes:

[0017] When a drone is shooting sports videos, the drone's gyroscope information is continuously collected through the IMU sensor installed in the drone; the gyroscope information includes angular velocity data.

[0018] The gyroscope information is input into the residual connection network structure model to obtain a fuzzy trajectory matrix; a pixel distribution intensity map is generated according to the fuzzy trajectory; and the fuzzy trajectory matrix includes the fuzzy trajectory of the fuzzy image.

[0019] Preferably, obtaining the first data set according to the clear image set and the scene blur information includes:

[0020] The scene blur information is used as a reference standard for the scene blur degree, and a time-averaged synthesis is performed on the clear image set to obtain a synthetic blur image set.

[0021] The clear images and synthetic fuzzy images corresponding to the scenes in the clear image set and the synthetic fuzzy image set are paired one-to-one to obtain a first data set.

[0022] Preferably, obtaining a training set and a validation set according to the first data set and the pixel distribution intensity map comprises:

[0023] The first data set is matched with the pixel distribution intensity map based on the scene according to the scene ambiguity information.

[0024] The data in the first data set are copied into three copies as data of three channels respectively, and the pixel distribution intensity map is used as data of the fourth channel, so as to obtain a second data set containing data of four channels.

[0025] The second data set is divided into a training set and a validation set according to a preset ratio.

[0026] Preferably, selecting the second model with the largest first indicator result to perform real-time infrared image deblurring of the drone includes:

[0027] The second model with the largest first indicator result is selected as the third model.

[0028] The infrared images collected by the UAV in real time are used as the input of the third model.

[0029] The third model outputs the processing results based on the infrared images collected by the UAV in real time.

[0030] Complete the real-time infrared image deblurring processing of UAV.

[0031] Preferably, before obtaining the pixel distribution intensity map according to the gyroscope information combined with the residual connection network structure model, the method further includes:

[0032] Construct a residual connection network structure model, including:

[0033] No.3 3 convolutional layers, Tanh nonlinear activation function layer, second 3 3 convolutional layers and the first 1 1 convolutional layer.

[0034] No.3 3 convolutional layers, Tanh nonlinear activation function layer and the second 3 3 convolutional layers are connected in sequence; the first 3 The output of the 3 convolutional layers is also connected to the second 3 The outputs of the 3 convolutional layers are added and connected to the first 1 1Convolutional layer input.

[0035] No.3 The 3 convolutional layers are used to receive the input gyroscope information and perform 3 3 After convolution processing, the output is sent to the Tanh nonlinear activation function layer, which is also used to connect the output to the second 3 through a residual connection The outputs of the 3 convolutional layers are added and connected to the first 1 1Convolutional layer input.

[0036] The Tanh nonlinear activation function layer is used to receive the first 3 The result sent by the 3 convolutional layer is processed and the output is sent to the second 3 3 convolutional layers.

[0037] No.3 The 3 convolutional layers are used to receive the output from the Tanh nonlinear activation function layer and perform 3 3 Output after convolution processing.

[0038] No.1 1 convolutional layer is used to receive the first 3 3 convolutional layer output and the second 3 3. Add the residual connection results of the convolutional layer outputs, and perform 1 1 After convolution, the fuzzy trajectory matrix is ​​output.

[0039] Preferably, before training the first model according to the training set, the method further includes:

[0040] Constructing the first model, that is, constructing a multi-scale encoder-decoder network model, including:

[0041] Construct a first encoding node, a second encoding node, a third encoding node, a neck layer node, a third decoding node, a second decoding node and a first decoding node that are connected in sequence.

[0042] The first encoding node is used to receive an initial input image, process the initial input image, and then output it.

[0043] The output of the first encoding node is down-sampled and connected to the initial input image proportionally reduced to 1 / 2 size to obtain the first input.

[0044] The second encoding node is used to receive the first input, process the first input and then output it.

[0045] The output of the second encoding node is downsampled and connected to the initial input image proportionally reduced to 1 / 4 of the size to obtain the second input.

[0046] The third encoding node is used to receive the second input, process the second input and output it to the neck layer node.

[0047] The neck layer node is used to receive the output of the third encoding node, and output it to the third decoding node after processing.

[0048] The third decoding node is used to receive the output of the neck layer node, process it and output it.

[0049] The output of the third decoding node is up-sampled to obtain a third input.

[0050] The second decoding node is used to receive the third input, process it and then output it.

[0051] The output of the second decoding node is up-sampled to obtain a fourth input.

[0052] The first decoding node is used to receive the fourth input, and output the infrared deblurring result of the initial input image size after processing.

[0053] When training the first model according to the training set, it also includes:

[0054] The second decoding node is also used to output an infrared deblurring result with a size of 1 / 2 of the original input image.

[0055] The third decoding node is also used to output an infrared deblurring result with a size of 1 / 4 of the original input image.

[0056] The infrared deblurring result of the original model input size output by the first decoding node, the infrared deblurring result of 1 / 2 of the initial input image size output by the second decoding node, and the infrared deblurring result of 1 / 4 of the initial input image size output by the third decoding node are respectively compared with the corresponding clear images of the same size; and the difference between the infrared deblurring result predicted by the model decoding node and the clear image is reduced through the optimization function of the first model.

[0057] Preferably, the first encoding node, the second encoding node, the third encoding node, the third decoding node, the second decoding node and the first decoding node have the same structure, and all include:

[0058] 3 connected in sequence 3 convolution blocks, global average pooling layer, second 1 1 convolutional layer, third 3 3 convolutional layers and depth-wise separable convolutional operation layers; 3 The output of the 3-convolution block is also added to the output of the depthwise separable convolution operation layer through a residual connection.

[0059] 3 The 3 convolutional block consists of 4 groups of 3 connected in sequence 3 convolutional layers, the first 3 The input of the 3 convolutional layers and the last 3 The outputs of the three convolutional layers are added together through residual connections as 3 3 Output of convolutional block.

[0060] 3 3 convolution blocks are used to receive node input features and perform 4 groups of 3 3. Convolution processing: the processing result is added to the node input feature through residual connection to obtain the spatial residual feature.

[0061] The global average pooling layer is used to receive 3 The spatial residual features output by the 3 convolutional blocks are processed and output to the second 1 1 convolutional layer.

[0062] Second 1 1 convolutional layer is used to receive the output of the global average pooling layer and perform 1 1 After convolution processing, output adaptive attention weights.

[0063] No.3 3 convolutional layers are used to receive adaptive attention weights and perform 3 3 After convolution processing, the output is sent to the depthwise separable convolution operation layer.

[0064] The depthwise separable convolution operation layer is used to receive the third 3. Output of the convolutional layer, processed and output.

[0065] The output of the depthwise separable convolution operation layer is The outputs of the 3 convolutional blocks are added through residual connections to obtain the node feature map output.

[0066] Neck nodes include:

[0067] First branch, second branch, third branch, third 1 1 convolutional layer and 3 3 deep convolutional layers.

[0068] In the neck layer node, the neck layer node input feature is processed by three cascaded frequency domain feature recovery branches, namely, by the first branch, the second branch and the third branch, including:

[0069] First branch: Input the neck layer node features through 3 3 dilated convolution operations will be processed through 3 The feature maps obtained by 3 dilated convolution are processed by Fourier transform and inverse Fourier transform respectively. 3 The feature maps obtained by the dilated convolution are added through residual connection to obtain the first branch features.

[0070] Second branch: Add the first branch feature to the neck layer node input feature and then pass through 6 6 dilated convolution operation, after Fourier transform and inverse Fourier transform, the output of the inverse Fourier transform in the second branch is combined with 6 6 The feature maps obtained by the dilated convolution operation are added through residual connection to obtain the second branch features.

[0071] The third branch: add the second branch feature to the neck layer node input feature and then pass through 9 9 The dilated convolution operation is processed, and after Fourier transform and inverse Fourier transform, the output of the inverse Fourier transform in the third branch is combined with 9 9 The feature maps obtained by the dilated convolution operation are added through residual connection to obtain the third branch feature.

[0072] After being processed by the first branch, the second branch, and the third branch, the first branch feature, the second branch feature, and the third branch feature are spliced, and the spliced ​​result is processed by the third branch. 1Convolutional layer after processing output; will be passed through the third 1 1 The feature map output after convolution layer processing is added to the input feature of the neck layer node through residual connection and then input 3 3 deep convolutional layers, 3 3 The deep convolutional layer outputs the feature map after processing, which is the output of the neck layer node.

[0073] The present invention also provides an infrared image deblurring system for an unmanned aerial vehicle platform, which is used for the method of the present invention. The system includes a unmanned aerial vehicle, an infrared thermal imager, an edge computing processing unit and a communication unit.

[0074] The infrared thermal imager, edge computing processing unit and communication unit are all installed on the drone.

[0075] The infrared thermal imager and edge computing processing unit are connected to the communication unit via a gigabit network cable.

[0076] The infrared thermal imager is used to capture clear image sets and motion videos of a target area; the infrared thermal imager includes an IMU sensor; the IMU sensor is used to obtain gyroscope information during the UAV's motion video capture, and send the gyroscope information to the edge computing processing unit through the communication unit.

[0077] The edge computing processing unit is used to obtain a pixel distribution intensity map based on gyroscope information combined with a residual connection network structure model; obtain scene blur information based on a motion video; obtain a first data set based on a clear image set and the scene blur information; the first data set includes clear images and synthetic blurred images based on a one-to-one pairing of scenes; obtain a training set and a verification set based on the first data set and the pixel distribution intensity map; train the first model based on the training set, and save the trained second model at intervals of preset rounds to obtain a second model set; the first model includes a multi-scale encoder-decoder network model for removing image blur; calculate the first index of all second models based on the verification set and the second model set, and when there is a second model with a first index result greater than a preset threshold, select the second model with the largest first index result to perform real-time infrared image deblurring of the drone; the first index includes an image restoration quality evaluation index.

[0078] The edge computing processing unit is also used to perform all data processing tasks in the system and store all mathematical models required for data processing in the system.

[0079] The communication unit is used to process information transmission between the edge computing unit and the infrared thermal imager, and is also used to process information transmission between the system and the outside world.

[0080] The present invention has the following beneficial effects:

[0081] The infrared image deblurring method for unmanned aerial vehicle platforms of the present invention obtains gyroscope information during the motion video shooting of the unmanned aerial vehicle through a clear image set and motion video of the target area shot by the unmanned aerial vehicle, and obtains a pixel distribution intensity map based on the gyroscope information in combination with a residual connection network structure model; obtains scene blur information based on the motion video, and obtains a clear image and a synthetic blurred image based on a one-to-one pairing of the scene based on the clear image set and the scene blur information, thereby providing a raw data basis for subsequent model training in scenes of rapid motion of the unmanned aerial vehicle platform and rapid rotation of the infrared thermal imager.

[0082] The training set and validation set are obtained based on the first data set and the pixel distribution intensity map, providing training data and prior data for subsequent model training. The first model is trained based on the training set, and the second model obtained by training is saved at intervals of preset rounds to obtain the second model set; this provides a range for the selection of subsequent models. The first indicators of all second models are calculated based on the validation set and the second model set, providing evaluation indicator data for the selection of models.

[0083] When there is a second model whose first indicator result is greater than a preset threshold, the second model with the largest first indicator result is selected to perform real-time infrared image deblurring of the UAV. The indicator result of the selected model is greater than the preset threshold to ensure the availability of the model; selecting the model with the largest indicator value to perform real-time infrared image deblurring of the UAV can ensure the accuracy and practicality of the method of the present invention to perform real-time infrared image deblurring of the UAV.

[0084] The method of the present invention can be applied to infrared images taken under scenes of rapid movement of the UAV platform and rapid rotation of the infrared thermal imager to achieve real-time deblurring processing; the method of the present invention has low computational complexity, high reasoning efficiency, strong generalization, and can meet the application scenarios of real-time infrared deblurring under different conditions.

[0085] The system of the present invention is used in the method of the present invention and has the same beneficial effects as the method of the present invention.

[0086] In addition to the above-described purposes, features and advantages, the present invention has other purposes, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0088] Figure 1 It is a flow chart of a preferred embodiment of the present invention.

[0089] Figure 2 It is a schematic diagram of the residual connection network structure model of a preferred embodiment of the present invention.

[0090] Figure 3 It is a schematic diagram of the first model structure of the preferred embodiment of the present invention.

[0091] Figure 4 It is a schematic diagram of the encoding node and decoding node structure of a preferred embodiment of the present invention.

[0092] Figure 5 Schematic diagram of the neck layer node of the preferred embodiment of the present invention.

[0093] Figure 6 It is a comparison diagram of infrared deblurring results of various methods in the preferred embodiment of the present invention.

[0094] Figure 7 It is a schematic diagram of the infrared deblurring result of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0095] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the present invention can be implemented in many different ways as defined and covered by the claims.

[0096] See also Figure 1 In a preferred embodiment of the present invention, a method for deblurring an infrared image for an unmanned aerial vehicle platform is provided, comprising the following steps:

[0097] S1. Obtain the gyroscope information during the motion video shooting of the drone by taking clear image sets and motion videos of the target area by the drone; obtain the pixel distribution intensity map based on the gyroscope information combined with the residual connection network structure model.

[0098] See also Figure 2 In a preferred embodiment of the present invention, before obtaining a pixel distribution intensity map according to the gyroscope information combined with the residual connection network structure model, the method further includes:

[0099] Construct a residual connection network structure model, including:

[0100] No.3 3 convolutional layers, Tanh nonlinear activation function layer, second 3 3 convolutional layers and the first 1 1 convolutional layer;

[0101] No.3 3 convolutional layers, Tanh nonlinear activation function layer and the second 3 3 convolutional layers are connected in sequence; the first 3 The output of the 3 convolutional layers is also connected to the second 3 The outputs of the 3 convolutional layers are added and connected to the first 1 1. Input of the convolutional layer;

[0102] No.3 The 3 convolutional layers are used to receive the input gyroscope information and perform 3 3 After convolution processing, the output is sent to the Tanh nonlinear activation function layer, which is also used to connect the output to the second 3 through a residual connection The outputs of the 3 convolutional layers are added and connected to the first 1 1. Input of the convolutional layer;

[0103] The Tanh nonlinear activation function layer is used to receive the first 3 The result sent by the 3 convolutional layer is processed and the output is sent to the second 3 3 convolutional layers;

[0104] No.3 The 3 convolutional layers are used to receive the output from the Tanh nonlinear activation function layer and perform 3 3 Output after convolution processing;

[0105] No.1 1 convolutional layer is used to receive the first 3 3 convolutional layer output and the second 3 3. Add the residual connection results of the convolutional layer outputs, and perform 1 1 After convolution, the fuzzy trajectory matrix is ​​output.

[0106] In a preferred embodiment of the present invention, gyroscope information is obtained during the drone shooting of motion video; and a pixel distribution intensity map is obtained according to the gyroscope information combined with a residual connection network structure model, specifically including:

[0107] When the drone is shooting sports videos, the drone's gyroscope information is continuously collected through the IMU sensor installed in the drone; the gyroscope information includes angular velocity data;

[0108] In a preferred embodiment of the present invention, the motion video includes infrared image data captured by the infrared thermal imager by controlling the drone to move quickly, or controlling the infrared thermal imager to rotate quickly within a large field of view.

[0109] The gyroscope information is input into the residual connection network structure model to obtain a fuzzy trajectory matrix; a pixel distribution intensity map is generated according to the fuzzy trajectory; and the fuzzy trajectory matrix includes the fuzzy trajectory of the fuzzy image.

[0110] S2. Obtain scene blur information based on motion video; obtain a first data set based on a clear image set and the scene blur information; the first data set includes clear images and synthetic blurred images based on a one-to-one pairing of scenes; obtain a training set and a validation set based on the first data set and a pixel distribution intensity map.

[0111] In a preferred embodiment of the present invention, obtaining the first data set according to the clear image set and the scene blur information specifically includes:

[0112] Using the scene blur information as a reference standard for the scene blur degree, a time-averaged synthesis is performed based on the clear image set to obtain a synthetic blur image set;

[0113] The clear images and synthetic fuzzy images corresponding to the scenes in the clear image set and the synthetic fuzzy image set are paired one-to-one to obtain a first data set.

[0114] In a preferred embodiment of the present invention, obtaining a training set and a validation set according to the first data set and the pixel distribution intensity map specifically includes:

[0115] performing scene-based matching of the first data set with the pixel distribution intensity map according to the scene ambiguity information;

[0116] The data in the first data set is copied into three copies as data of three channels respectively, and the pixel distribution intensity map is used as data of the fourth channel, so as to obtain a second data set containing data of four channels;

[0117] The second data set is divided into a training set and a validation set according to a preset ratio.

[0118] S3. Train the first model according to the training set, and save the trained second model at intervals of preset rounds to obtain a second model set; the first model includes a multi-scale encoder-decoder network model for removing image blur.

[0119] See also Figure 3 In a preferred embodiment of the present invention, before training the first model according to the training set, it also includes:

[0120] Constructing the first model, that is, constructing a multi-scale encoder-decoder network model, including:

[0121] Constructing a first encoding node, a second encoding node, a third encoding node, a neck layer node, a third decoding node, a second decoding node and a first decoding node connected in sequence;

[0122] The first encoding node is used to receive an initial input image, process the initial input image and then output it;

[0123] The output of the first encoding node is downsampled and connected to the initial input image proportionally reduced to 1 / 2 size to obtain a first input;

[0124] The second encoding node is used to receive the first input, process the first input and then output;

[0125] The output of the second encoding node is downsampled and connected to the initial input image proportionally reduced to 1 / 4 of the size to obtain the second input;

[0126] The third encoding node is used to receive the second input, process the second input and output it to the neck layer node;

[0127] The neck layer node is used to receive the output of the third encoding node, and output it to the third decoding node after processing;

[0128] The third decoding node is used to receive the output of the neck layer node, process it and output it;

[0129] The output of the third decoding node is up-sampled to obtain a third input;

[0130] The second decoding node is used to receive the third input, process it and output it;

[0131] The output of the second decoding node is up-sampled to obtain a fourth input;

[0132] The first decoding node is used to receive the fourth input, and output the infrared deblurring result of the size of the initial input image after processing;

[0133] When training the first model according to the training set, it also includes:

[0134] The second decoding node is also used to output an infrared deblurring result of 1 / 2 the size of the initial input image;

[0135] The third decoding node is also used to output an infrared deblurring result of 1 / 4 the size of the initial input image;

[0136] The infrared deblurring result of the original model input size output by the first decoding node, the infrared deblurring result of 1 / 2 of the initial input image size output by the second decoding node, and the infrared deblurring result of 1 / 4 of the initial input image size output by the third decoding node are respectively compared with the corresponding clear images of the same size; and the difference between the infrared deblurring result predicted by the model decoding node and the clear image is reduced through the optimization function of the first model.

[0137] In the preferred embodiment of the present invention, see Figure 4 The first encoding node, the second encoding node, the third encoding node, the third decoding node, the second decoding node and the first decoding node have the same structure, and all include:

[0138] 3 connected in sequence 3 convolution blocks, global average pooling layer, second 1 1 convolutional layer, third 3 3 convolutional layers and depth-wise separable convolutional operation layers; 3 The output of the 3 convolutional block is also added to the output of the depthwise separable convolutional operation layer through a residual connection;

[0139] 3 The 3 convolutional block consists of 4 groups of 3 connected in sequence 3 convolutional layers, the first 3 The input of the 3 convolutional layers and the last 3 The outputs of the three convolutional layers are added together through residual connections as 3 3. Output of convolutional block;

[0140] 3 3 convolution blocks are used to receive node input features and perform 4 groups of 3 3. Convolution processing: adding the processing result to the node input feature through residual connection to obtain the spatial residual feature;

[0141] The global average pooling layer is used to receive 3 The spatial residual features output by the 3 convolutional blocks are processed and output to the second 1 1 convolutional layer;

[0142] Second 1 1 convolutional layer is used to receive the output of the global average pooling layer and perform 1 1. Output adaptive attention weight after convolution processing;

[0143] No.3 3 convolutional layers are used to receive adaptive attention weights and perform 3 3. After convolution processing, the output is sent to the depth-separable convolution operation layer;

[0144] The depthwise separable convolution operation layer is used to receive the third 3. Output of the convolutional layer, processed and output;

[0145] The output of the depthwise separable convolution operation layer is 3. The outputs of the convolutional blocks are added through residual connections to obtain the node feature map output;

[0146] See also Figure 5 , the neck layer nodes include:

[0147] First branch, second branch, third branch, third 1 1 convolutional layer and 3 3 deep convolutional layers;

[0148] In the neck layer node, the neck layer node input feature is processed by three cascaded frequency domain feature recovery branches, namely, by the first branch, the second branch and the third branch, including:

[0149] First branch: Input the neck layer node features through 3 3 dilated convolution operations will be processed through 3 The feature maps obtained by 3 dilated convolution are processed by Fourier transform and inverse Fourier transform respectively. 3. The feature maps obtained by the dilated convolution are added through residual connection to obtain the first branch features;

[0150] Second branch: Add the first branch feature to the neck layer node input feature and then pass through 6 6 dilated convolution operation, after Fourier transform and inverse Fourier transform, the output of the inverse Fourier transform in the second branch is combined with 6 6. The feature maps obtained by the dilated convolution operation are added through residual connection to obtain the second branch features;

[0151] The third branch: add the second branch feature to the neck layer node input feature and then pass through 9 9 The dilated convolution operation is processed, and after Fourier transform and inverse Fourier transform, the output of the inverse Fourier transform in the third branch is combined with 9 9 The feature maps obtained by the dilated convolution operation are added through residual connection to obtain the third branch feature;

[0152] After being processed by the first branch, the second branch, and the third branch, the first branch feature, the second branch feature, and the third branch feature are spliced, and the spliced ​​result is processed by the third branch. 1 The output after convolutional layer processing; will be passed through the third 1 1 The feature map output after convolution layer processing is added to the input feature of the neck layer node through residual connection and then input 3 3 deep convolutional layers, 3 3 The deep convolutional layer outputs the feature map after processing, which is the output of the neck layer node.

[0153] S4. Calculate the first index of all second models according to the validation set and the second model set. When there is a second model whose first index result is greater than a preset threshold, select the second model with the largest first index result to perform real-time infrared image deblurring of the UAV; the first index includes an image restoration quality evaluation index.

[0154] In a preferred embodiment of the present invention, calculating the first index of all second models according to the validation set and the second model set specifically includes:

[0155] Verify all second models in the second model set according to the verification set, and the second models output verification results;

[0156] Calculate the first indicator based on the validation set and the validation results;

[0157] The first indicator includes the image restoration quality evaluation index PSNR, which can be expressed as:

[0158] ,

[0159] in, MSE represents the mean square error between the synthesized blurred image and the output result of the second model, where the output result of the second model is the clear image obtained after deblurring by the second model; MAX I Indicates the maximum color value of the synthesized blurred image.

[0160] In a preferred embodiment of the present invention, selecting the second model with the largest first indicator result to perform real-time infrared image deblurring of the drone specifically includes:

[0161] Select the second model with the largest first indicator result as the third model;

[0162] The infrared images collected by the UAV in real time are used as the input of the third model;

[0163] The third model outputs the processing results based on the infrared images collected by the drone in real time;

[0164] Complete the real-time infrared image deblurring processing of UAV.

[0165] Verification part:

[0166] The preferred embodiment of the present invention carries out experimental testing in a certain flight test field, and the range of the test scene is 5km 5km, the field test area includes water scenes and ground scenes such as grassland and woods. The UAV flight altitude is 300m, the infrared thermal imager mounted on the UAV has a wavelength of 8 to 14um, the obtained video frame rate is 30Hz, and the resolution is 640×480 pixels.

[0167] In a preferred embodiment of the present invention, the third model is tested by a test data set to calculate the PSNR index. At the same time, SRN_DeblurNet, FFANet, MPRNet, GridDehazeNet, MIMO-UNet, Dehamer, DeepRFT, AirNet, OKNet, DSANet, and FocalNet are used as comparison algorithms for training and testing.

[0168] The comparison results of PSNR, SSIM, MAE, and LPIPS image deblurring performance evaluation indicators of the comparison algorithms are shown in Table 1.

[0169] ,

[0170] In Table 1, an ↑ next to an indicator indicates that the larger the value of the indicator, the better the infrared image deblurring performance; ↓ indicates that the smaller the value of the indicator, the better the infrared image deblurring performance. It can be seen from Table 1 that all evaluation indicators of the method of the present invention are at the highest level, indicating that the image deblurring performance of the method of the present invention is the best. In addition, the parameter amount of the method of the present invention is 6.53M, and the calculation amount is 57.4G. The parameter amount is small and the calculation complexity is low. The method of the present invention can meet the requirements of real-time infrared deblurring of drones. In addition, the preferred embodiment of the present invention compares the deblurring effect of the method of the present invention with blurred images, AirNet, GridDehazeNet, MPRNet and true values. The comparison results are shown in Figure 6 In order to see the details clearly, Figure 6 The red rectangular area in the left figure is Figure 6 Zoom in on the right to better compare the deblurring effect. Figure 6 It can be seen that the method of the present invention can obtain a clearer infrared deblurred picture, which shows that the method of the present invention is effective in removing infrared motion blur.

[0171] In summary, the infrared image deblurring method for UAV platforms of the present invention obtains gyroscope information during the UAV shooting of motion videos through clear image sets and motion videos of the target area shot by the UAV, and obtains a pixel distribution intensity map based on the gyroscope information combined with the residual connection network structure model; obtains scene blur information based on the motion video, and obtains a clear image and a synthetic blurred image based on a one-to-one pairing of the scene based on the clear image set and the scene blur information, which provides the original data basis for subsequent model training in the scene of rapid movement of the UAV platform and rapid rotation of the infrared thermal imager.

[0172] The training set and validation set are obtained based on the first data set and the pixel distribution intensity map, providing training data and prior data for subsequent model training. The first model is trained based on the training set, and the second model obtained by training is saved at intervals of preset rounds to obtain the second model set; this provides a range for the selection of subsequent models. The first indicators of all second models are calculated based on the validation set and the second model set, providing evaluation indicator data for the selection of models.

[0173] When there is a second model whose first indicator result is greater than a preset threshold, the second model with the largest first indicator result is selected to perform real-time infrared image deblurring of the UAV. The indicator result of the selected model is greater than the preset threshold to ensure the availability of the model; selecting the model with the largest indicator value to perform real-time infrared image deblurring of the UAV can ensure the accuracy and practicality of the method of the present invention to perform real-time infrared image deblurring of the UAV.

[0174] The method of the present invention can be applied to infrared images taken under scenes of rapid movement of the UAV platform and rapid rotation of the infrared thermal imager to achieve real-time deblurring processing; the method of the present invention has low computational complexity, high reasoning efficiency, strong generalization, and can meet the application scenarios of real-time infrared deblurring under different conditions.

[0175] In a preferred embodiment of the present invention, an infrared image deblurring system for a UAV platform is also provided, which is used in the method of the present invention, and the system includes a UAV, an infrared thermal imager, an edge computing processing unit, and a communication unit;

[0176] The infrared thermal imager, edge computing processing unit and communication unit are all installed on the drone;

[0177] The infrared thermal imager and edge computing processing unit are connected to the communication unit via a gigabit network cable;

[0178] The infrared thermal imager is used to capture clear image sets and motion videos of the target area; the infrared thermal imager includes an IMU sensor; the IMU sensor is used to obtain gyroscope information during the UAV shooting of the motion video, and send the gyroscope information to the edge computing processing unit through the communication unit;

[0179] The edge computing processing unit is used to obtain a pixel distribution intensity map based on gyroscope information combined with a residual connection network structure model; obtain scene blur information based on motion video; obtain a first data set based on a clear image set and scene blur information; the first data set includes clear images and synthetic blurred images based on a one-to-one pairing of scenes; obtain a training set and a verification set based on the first data set and the pixel distribution intensity map; train the first model based on the training set, and save the trained second model at intervals of preset rounds to obtain a second model set; the first model includes a multi-scale encoder-decoder network model for removing image blur; calculate the first index of all second models based on the verification set and the second model set, and when there is a second model with a first index result greater than a preset threshold, select the second model with the largest first index result to perform real-time infrared image deblurring of the drone; the first index includes an image restoration quality evaluation index;

[0180] The edge computing processing unit is also used to perform all data processing tasks in the system and store all mathematical models required for data processing in the system;

[0181] The communication unit is used to process information transmission between the edge computing unit and the infrared thermal imager, and is also used to process information transmission between the system and the outside world.

[0182] In a preferred embodiment of the present invention, the communication unit is used to process information transmission between the system and the outside world, send information in the system to external equipment, and receive information from external equipment.

[0183] See also Figure 7 The communication unit sends the infrared image deblurring result to the external device for display. It can be seen that the method of the present invention can meet the actual application needs, and can realize the infrared image motion deblurring function in the scenario of the actual rapid flight of the UAV and the rapid rotation of the infrared thermal imager, and has a strong practical application value.

[0184] The system of the present invention is used in the method of the present invention and has the same beneficial effects as the method of the present invention.

[0185] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for deblurring infrared images for UAV platforms, characterized in that: include: By taking clear images and motion videos of the target area by the drone, gyroscope information during the motion video shooting by the drone is obtained; Obtain a pixel distribution intensity map based on the gyroscope information combined with a residual connection network structure model; Obtain scene blur information according to the motion video; obtain a first data set according to the clear image set and the scene blur information; the first data set includes a clear image and a synthetic blur image based on a one-to-one pairing of scenes; Obtaining a first data set according to the clear image set and the scene blur information includes: Using the scene blur information as a reference standard for the scene blur degree, performing time-averaged synthesis according to the clear image set to obtain a synthetic blur image set; One-to-one pairing of clear images and synthetic fuzzy images corresponding to scenes in the clear image set and the synthetic fuzzy image set to obtain a first data set; Obtaining a training set and a validation set according to the first data set and the pixel distribution intensity map; The first model is trained according to the training set, and a second model obtained by the training is saved at intervals of preset rounds to obtain a second model set; the first model includes a multi-scale encoder-decoder network model for removing image blur; Calculate the first index of all second models according to the validation set and the second model set, and when there is a second model whose first index result is greater than a preset threshold, select the second model with the largest first index result to perform real-time infrared image deblurring of the drone; Calculate the first index of all second models according to the validation set and the second model set includes: Verify all second models in the second model set according to the verification set, and the second models output verification results; Calculate a first indicator according to the verification set and the verification result; The first indicator includes the image restoration quality evaluation indicator PSNR, which can be expressed as: ， in, MSE represents the mean square error between the synthesized blurred image and the output result of the second model, where the output result of the second model is the clear image obtained after deblurring by the second model; MAX I Indicates the maximum color value of the synthesized blurred image.

2. The infrared image deblurring method for UAV platform according to claim 1 is characterized in that: The gyroscope information of the drone during shooting of sports video is obtained; The pixel distribution intensity map obtained according to the gyroscope information combined with the residual connection network structure model includes: During the period when the drone shoots the motion video, the gyroscope information of the drone is continuously collected through the IMU sensor loaded in the drone; the gyroscope information includes angular velocity data; The gyroscope information is input into the residual connection network structure model to obtain a fuzzy trajectory matrix; the pixel distribution intensity map is generated according to the fuzzy trajectory; the fuzzy trajectory matrix includes a fuzzy trajectory of a fuzzy image.

3. The infrared image deblurring method for UAV platform according to claim 2 is characterized in that: Obtaining a training set and a validation set according to the first data set and the pixel distribution intensity map includes: performing scene-based matching of the first data set with the pixel distribution intensity map according to the scene blur information; The data in the first data set is copied into three copies as data of three channels respectively, and the pixel distribution intensity map is used as data of the fourth channel to obtain a second data set containing data of four channels; The second data set is divided into the training set and the validation set according to a preset ratio.

4. The infrared image deblurring method for UAV platform according to claim 3 is characterized in that: The selecting the second model with the largest first indicator result to perform real-time infrared image deblurring of the UAV comprises: Select the second model with the largest first indicator result as the third model; Using the infrared image collected by the drone in real time as input of the third model; The third model outputs processing results according to the infrared images collected by the drone in real time; Complete the real-time infrared image deblurring processing of UAV.

5. The infrared image deblurring method for UAV platform according to claim 4 is characterized in that: Before obtaining the pixel distribution intensity map according to the gyroscope information combined with the residual connection network structure model, it also includes: Constructing the residual connection network structure model includes: No.3 3 convolutional layers, Tanh nonlinear activation function layer, second 3 3 convolutional layers and the first 1 1 convolutional layer; No.3 3 convolutional layers, Tanh nonlinear activation function layer and the second 3 3 convolutional layers are connected in sequence; wherein the first 3 The output of the 3 convolutional layers is also connected to the second 3 The outputs of the 3 convolutional layers are added and connected to the first 1 1. Input of the convolutional layer; The first 3 3 convolutional layers are used to receive the input gyroscope information and perform 3 3 After convolution processing, the output is sent to the Tanh nonlinear activation function layer, which is also used to connect the output to the second 3 through a residual connection The outputs of the 3 convolutional layers are added and connected to the first 1 1. Input of convolutional layer; The Tanh nonlinear activation function layer is used to receive the first 3 The result sent by the 3 convolutional layer is processed and the output is sent to the second 3 3 convolutional layers; The second 3 The 3 convolutional layers are used to receive the output from the Tanh nonlinear activation function layer and perform 3 3 Output after convolution processing; The first 1 1 convolutional layer is used to receive the first 3 3 convolutional layer output and the second 3 3. The residual connection method of the convolution layer output is added, and the residual connection method addition result is performed 1 1 After convolution, the fuzzy trajectory matrix is ​​output.

6. The infrared image deblurring method for UAV platform according to claim 5, characterized in that: Before training the first model according to the training set, the method further includes: Constructing the first model, that is, constructing the multi-scale encoder-decoder network model, includes: Constructing a first encoding node, a second encoding node, a third encoding node, a neck layer node, a third decoding node, a second decoding node and a first decoding node connected in sequence; The first encoding node is used to receive an initial input image, process the initial input image and then output it; The output of the first encoding node is downsampled and connected to the initial input image proportionally reduced to 1 / 2 size to obtain a first input; The second encoding node is used to receive the first input, process the first input and then output it; The output of the second encoding node is downsampled and connected to the initial input image proportionally reduced to 1 / 4 of the size to obtain a second input; The third encoding node is used to receive the second input, process the second input and output it to the neck layer node; The neck layer node is used to receive the output of the third encoding node, and output it to the third decoding node after processing; The third decoding node is used to receive the output of the neck layer node, process it and then output it; The output of the third decoding node is up-sampled to obtain a third input; The second decoding node is used to receive the third input, process it and output it; The output of the second decoding node is up-sampled to obtain a fourth input; The first decoding node is used to receive the fourth input, and output an infrared deblurring result of the size of the initial input image after processing; When training the first model according to the training set, it also includes: The second decoding node is also used to output an infrared deblurring result with a size of 1 / 2 of the initial input image; The third decoding node is also used to output an infrared deblurring result of 1 / 4 the size of the initial input image; The infrared deblurring result of the original model input size output by the first decoding node, the infrared deblurring result of 1 / 2 of the initial input image size output by the second decoding node, and the infrared deblurring result of 1 / 4 of the initial input image size output by the third decoding node are respectively compared with corresponding clear images of the same size; and the difference between the infrared deblurring result predicted by the model decoding node and the clear image is reduced by the optimization function of the first model.

7. The infrared image deblurring method for UAV platform according to claim 6 is characterized in that: The first encoding node, the second encoding node, the third encoding node, the third decoding node, the second decoding node and the first decoding node have the same structure and all include: 3 connected in sequence 3 convolution blocks, global average pooling layer, second 1 1 convolutional layer, third 3 3 convolutional layers and depth-wise separable convolutional operation layers; wherein the 3 3. The output of the convolution block is also added to the output of the depthwise separable convolution operation layer through a residual connection; 3 The 3 convolutional block consists of 4 groups of 3 connected in sequence 3 convolutional layers, the first 3 The input of the 3 convolutional layers and the last 3 The outputs of the three convolutional layers are added together through residual connections as described above.

3. Output of convolutional block; 3 3 convolution blocks are used to receive node input features and perform 4 groups of 3 3. Convolution processing, adding the processing result to the node input feature through residual connection to obtain the spatial residual feature; The global average pooling layer is used to receive 3 The spatial residual features output by the 3 convolutional blocks are processed and output to the second 1 1 convolutional layer; The second 1 1 convolutional layer is used to receive the output of the global average pooling layer and perform 1 1. Output adaptive attention weight after convolution processing; The third 3 convolutional layers are used to receive the adaptive attention weights and perform 3 3. After convolution processing, the output is sent to the depthwise separable convolution operation layer; The depthwise separable convolution operation layer is used to receive the third 3. Output of the convolutional layer, processed and output; The output of the depthwise separable convolution operation layer is 3. The outputs of the convolutional blocks are added through residual connections to obtain the node feature map output; The neck layer nodes include: First branch, second branch, third branch, third 1 1 convolutional layer and 3 3 deep convolutional layers; In the neck layer node, the neck layer node input feature is processed by three cascaded frequency domain feature recovery branches, that is, processed by the first branch, the second branch and the third branch, including: The first branch: the neck layer node input feature passes 3 3 dilated convolution operations will be processed through 3 The feature maps obtained by the 3-hole convolution process are respectively processed by Fourier transform and inverse Fourier transform and then compared with the 3-hole convolution process.

3. The feature maps obtained by the dilated convolution are added through residual connection to obtain the first branch features; Second branch: The first branch feature is added to the neck layer node input feature and then processed through 6 6 dilated convolution operation, after Fourier transform and inverse Fourier transform, the output of the inverse Fourier transform in the second branch is combined with 6 6. The feature maps obtained by the dilated convolution operation are added through residual connection to obtain the second branch features; The third branch: The second branch feature is added to the neck layer node input feature and then sequentially passes through 9 9 The dilated convolution operation is processed, and after Fourier transform and inverse Fourier transform, the output of the inverse Fourier transform in the third branch is combined with 9 9 The feature maps obtained by the dilated convolution operation are added through residual connection to obtain the third branch feature; After being processed by the first branch, the second branch and the third branch, the first branch feature, the second branch feature and the third branch feature are spliced, and the spliced ​​result is processed by the third branch. 1 The output after convolutional layer processing; will be passed through the third 1 The feature map output after 1 convolution layer processing and the input feature of the neck layer node are added through residual connection and then input into the 3 3 deep convolutional layers, the 3 3. After the deep convolution layer is processed, the feature map is output, that is, the output of the neck layer node.

8. An infrared image deblurring system for unmanned aerial vehicle platforms, used in the method according to any one of claims 1 to 7, characterized in that: The system includes a drone, an infrared thermal imager, an edge computing processing unit, and a communication unit; The infrared thermal imager, edge computing processing unit and communication unit are all installed on the drone; The infrared thermal imager and the edge computing processing unit are both connected to the communication unit via a gigabit network cable; The infrared thermal imager is used to capture clear image sets and motion videos of the target area; the infrared thermal imager includes an IMU sensor; the IMU sensor is used to obtain gyroscope information during the drone shooting of motion videos, and send the gyroscope information to the edge computing processing unit through the communication unit; The edge computing processing unit is used to obtain a pixel distribution intensity map based on the gyroscope information combined with the residual connection network structure model; obtain scene blur information based on the motion video; obtain a first data set based on the clear image set and the scene blur information; the first data set includes clear images and synthetic blurred images based on a one-to-one pairing of scenes; obtain a training set and a verification set based on the first data set and the pixel distribution intensity map; train the first model based on the training set, and save the trained second model at intervals of preset rounds to obtain a second model set; the first model includes a multi-scale encoder-decoder network model for removing image blur; calculate the first index of all second models based on the verification set and the second model set, and when there is a second model with a first index result greater than a preset threshold, select the second model with the largest first index result to perform real-time infrared image deblurring of the drone; the first index includes an image restoration quality evaluation index; The edge computing processing unit is also used to perform all data processing tasks in the system and store all mathematical models required for data processing in the system; The communication unit is used to process information transmission between the edge computing unit and the infrared thermal imager, and is also used to process information transmission between the system and the outside world.

Images