A method for local motion perception of infrared small target feature enhancement

Abstract

The present invention discloses a method for enhancing the features of infrared small targets with local motion perception, and the method comprises the following steps: step 1: loading infrared sequence images, selecting continuous T-frame images as network input, and extracting image features using a backbone network; step 2: extracting rough target inter-frame motion information using a coarse motion estimation module CME, and generating forward optical flow and backward optical flow; step 3: aligning multi-frame sequences using an energy enhancement module EnE in combination with optical flow and deformable convolution, and enhancing target features using a 3D convolution with a convolution kernel size of 1×1; step 4: outputting an enhanced infrared image using a feature enhancement task head that guides optical flow learning, fusing multi-scale features through a target segmentation task head, outputting a target segmentation result, and then guiding the learning of a feature enhancement network. The method can effectively adapt to scenes with dim targets and background motion, and output high-quality enhanced images of infrared multi-frame moving small targets.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and relates to a multi-frame infrared small target feature enhancement method, and specifically to a local motion-aware infrared small target feature enhancement method. Background Art

[0002] Infrared imaging systems overcome the limitations of visible light imaging at night or in bad weather conditions by detecting the difference in radiation energy between the target and the background, and provide high-quality detection capabilities in a variety of environments. They are widely used in disaster prevention, weather forecasting, target detection and other fields. However, in a large number of application scenarios, the detection platform is often far away from the target, resulting in the lack of target structure and texture in the infrared image. In addition, the imaging scene is complex, and the target is submerged in the background clutter, showing weak and small characteristics, which brings great application difficulties to subsequent target detection, recognition and other advanced visual tasks. Therefore, the infrared small target feature enhancement method is committed to improving the saliency of weak edges and texture details in the image, thereby improving the visibility and detectability of small targets and optimizing the overall quality of infrared images.

[0003] Infrared small target feature enhancement methods can be divided into two categories: traditional methods and deep learning methods. Traditional infrared small target feature enhancement methods include spatial domain transformation and frequency domain filtering, but this method requires the use of prior knowledge of the image to improve the performance of the enhancement method, and the model parameters need to be manually adjusted as the application scenario changes, and the algorithm robustness is poor. The infrared small target feature enhancement method based on deep learning automatically mines and enhances weak edge and weak texture features in the image through a convolutional neural network (CNN), making full use of the powerful feature extraction ability and adaptability of CNN, and can effectively enhance the visibility and detectability of infrared small targets in complex backgrounds.

[0004] In recent years, researchers have proposed many infrared small target feature enhancement algorithms, such as: frequency-domain-based image registration and fusion methods to generate high-resolution images; modeling the temporal correlation of video sequences using recurrent neural networks to make full use of temporal information; designing motion compensation residual networks and using optical flow for motion estimation compensation. However, the above methods do not solve the problems of moving background and multi-frame feature registration, and are difficult to adapt to scenes with dim targets and moving backgrounds. Summary of the invention

[0005] Aiming at the problem of missing structure and texture of small and weak targets under infrared moving background, the present invention estimates the inter-frame motion of multiple infrared frames to align the features of multiple frames, uses deformable convolution to fine-tune the alignment results, and converts the time domain characteristics of small and weak targets into spatial domain saliency through target energy accumulation, thereby providing a local motion-aware infrared small target feature enhancement method. This method can effectively adapt to scenes with dim targets and moving backgrounds, and output high-quality enhanced images of small moving targets in multiple infrared frames.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] A method for enhancing the characteristics of infrared small targets based on local motion perception comprises the following steps:

[0008] Step 1: Load the infrared sequence image, select T consecutive frames of images as network input, and use the backbone network to extract image features;

[0009] Step 2: Use the coarse motion estimation module (CME) to extract rough target inter-frame motion information and generate forward optical flow and backward optical flow;

[0010] Step 3: Use the energy enhancement module (EnE) combined with optical flow and deformable convolution to align multi-frame sequences, and use 3D convolution with a convolution kernel size of 1×1 to enhance the target features;

[0011] Step 4: Use the feature enhancement task head that guides optical flow learning to output the enhanced infrared image, fuse the multi-scale features through the target segmentation task head, output the target segmentation result, and then guide the learning of the feature enhancement network.

[0012] Compared with the prior art, the present invention has the following advantages:

[0013] 1. A Local Motion Cue Representation Network (LMCRN) is proposed. The multi-frame features extracted by the backbone network are used for inter-frame motion estimation to align the multi-frame features, and deformable convolution is used to fine-tune the alignment results. Finally, 3D convolution is used to accumulate target energy and convert the temporal characteristics of faint targets into spatial saliency, thereby solving the problem of weak target feature extraction and improving the performance of infrared small target feature enhancement methods.

[0014] 2. A coarse motion estimation module (CME) is proposed. It stacks the features of adjacent frames together as input, uses a lightweight convolutional neural network to map the differences between frames into the horizontal and vertical components of the optical flow, and extracts the rough motion direction vector of the feature differences between frames. Compared with other optical flow extraction methods, CME only provides rough motion direction without outputting precise optical flow information, so the lightweight structure has almost no computational cost.

[0015] 3. An energy enhancement module (EnE) is proposed, which uses roughly extracted optical flow to roughly align multi-frame features, then learns the target morphology and accurately aligns the features of each frame through a deformable attention mechanism, and finally uses 3D convolution to weight each frame to achieve target energy collection and enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 The overall architecture diagram of the local motion cue representation network;

[0017] Figure 2 It is a schematic diagram of CME module;

[0018] Figure 3 This is the schematic diagram of the EnE module;

[0019] Figure 4 This is a graph of the experimental results. DETAILED DESCRIPTION

[0020] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0021] The present invention provides a method for enhancing the features of infrared small targets with local motion perception. The method designs a local motion cue representation network (LMCRN). First, the multi-frame features extracted by the backbone network are subjected to inter-frame motion estimation to align the multi-frame features. Next, deformable convolution is used to fine-tune the alignment results. Finally, 3D convolution is used to accumulate target energy, and the time domain characteristics of weak targets are converted into spatial domain significance, thereby solving the problem that it is difficult to extract features of weak targets. Figure 1 As shown, the specific steps include:

[0022] Step 1: Load the infrared sequence image, select T consecutive frames of images as network input, and use the backbone network to extract image features. The specific steps are as follows:

[0023] Step 1: Load multiple frames of images As the backbone network input, T represents the number of infrared sequence image frames contained in each batch in the network input, H1 represents the input image height, W1 represents the input image width, 3 represents that the number of infrared image channels is mapped to 3 during data loading, and T is set to 5 in the present invention.

[0024] Step 1 and 2: The backbone network uses the Video Swin Transformer network to input multiple frames of images The image blocks are divided into T / 2×H1 / 4×W1 / 4 blocks at a scale of 2×4×4×3. Each image block contains 96-dimensional features, which are then mapped to dimension C1 after embedding. The network uses 3D non-overlapping windows and translation windows to divide the image blocks to reduce the complexity of Transformer calculation and extract the global features of the image sequence. Finally, four layers of features F1, F2, F3 and F4 are output, with corresponding dimensions of T×H1 / 8×W1 / 8×2C1, T×H1 / 16×W1 / 16×4C1, T×H1 / 32×W1 / 32×4C1 and T×H1 / 32×W1 / 32×4C1, respectively. The multi-frame top-level feature F4 containing the richest semantic information is input into the subsequent network to extract the target spatiotemporal correlation features.

[0025] Step 2: Use the CME module to extract rough target inter-frame motion information and generate forward optical flow and backward optical flow. The specific steps are as follows:

[0026] Step 21: If Figure 2 As shown in Figure 1, the CME module consists of four layers of convolution. The first three layers of convolution use expansion coefficients of 8, 16, and 1 to capture inter-frame displacements of different sizes, respectively. The last layer of convolution maps the inter-frame differences into the horizontal and vertical components of the optical flow, and the final output is used as a rough motion pointing vector.

[0027] Step 22: Input features The adjacent frame features are stacked together to extract the feature differences between frames, where {·} represents the set, C2 represents the number of feature channels, H2 represents the feature map height, W2 represents the feature map width, and F 4，i Represents the fourth layer i-th frame feature output by the backbone network. Using the parameter-sharing CME module, the forward optical flow is output and backward optical flow in represents the optical flow from the pth frame to the qth frame, and N represents the total number of frames. The dimensions of the forward and backward optical flows are the same, both Among them, T-1 represents the number of optical flows, H3 represents the height of the optical flow map, W3 represents the width of the optical flow map, and 2 represents the horizontal and vertical components of the optical flow.

[0028] Step 3: Use the EnE module to combine optical flow and deformable convolution to align multi-frame sequences, and use 3D convolution with a convolution kernel size of 1×1 to enhance target features. The specific structure of the EnE module is as follows: Figure 3 The specific steps are as follows:

[0029] Step 3: Input the forward and backward optical flow and multi-frame features into the EnE module, align the multi-frame features and enhance the target features. The specific structure of the EnE module is as follows: Figure 3 The specific steps are as follows:

[0030] Step 31: Use the rough optical flow to perform a backward warp operation on the F4 features of multiple frames to achieve rough alignment. When processing the features of the adjacent 5 frames, first use the backward warp operation to convert the forward optical flow and backward optical flow Expanded to two adjacent frames, the expanded optical flow and They are as follows:

[0031]

[0032] In the formula Represents the optical flow from the p-th frame to the q-th frame.

[0033] Step 32: After aligning each frame to the middle frame, use the concatenate operation to splice them together to obtain the alignment feature F a :

[0034]

[0035] Where {·} represents a set, F 4，i It is the fourth layer i-th frame feature output by the backbone network. Alignment feature F a satisfy Where T represents the number of infrared sequence image frames, C3 represents the number of feature channels of the fourth layer, H4 represents the height of the fourth layer feature map, and W4 represents the width of the fourth layer feature map.

[0036] Step 33: Align the feature F a , the intermediate frame of the input feature F4 and the forward and backward optical flow along the first dimension are spliced ​​together to form the feature Where C4 represents the number of spliced ​​feature channels, H5 represents the height of the spliced ​​feature map, and W5 represents the width of the spliced ​​feature map.

[0037] Step 3 and 4: Use the deformable attention function (DAF) to learn the target shape and accurately align the features of each frame. The specific steps are as follows:

[0038] Step 341: Use a convolutional layer with a convolution kernel size of 3×3 and a Leaky ReLU activation function to concatenate the features F b It is divided into two branches, one of which is added to the optical flow offset to generate the sampling point offset The other branch generates normalized attention weights Among them, T represents the number of frames, H6 represents the height of the sampling point offset tensor, W6 represents the width of the sampling point offset tensor, H7 represents the height of the attention weight matrix, W7 represents the width of the attention weight matrix, M represents the number of attention heads, and P represents the number of sampling points. Used to find pixels with the same name in adjacent frames, Used to characterize the possibility that a pixel is a target.

[0039] Step 342: The original input feature F4 is flattened after two layers of convolution T represents the number of frames, H8 represents the tensor height, W8 represents the tensor width, and C5 represents the number of tensor channels.

[0040] Step 343: and The calculation methods are as follows:

[0041]

[0042] Where cv represents a convolutional layer with a kernel size of 3×3, LR represents the LeakyReLu activation function, cat represents the concatenation operation, and Fgo represents the optical flow offset. The offset across frames is calculated by the optical flow of adjacent frames. The calculation formula is the same as step 31.

[0043] Step 344: Transform the attention module and Acts on And output features Among them, T represents the number of frames, C6 represents the number of output feature channels, H9 represents the height of the output feature map, and W9 represents the width of the output feature map. At the same time, residual connections are used to improve the optimization efficiency of the network, and the output F4 is combined with the convolution Perform addition operation to generate And will After connecting with F4, two layers of convolution operations are performed to generate To fuse the original features and the enhanced features, and After addition, 3D convolution is used to accumulate the target energy, and the sigmoid activation function is used to normalize the output of the local spatiotemporal attention weight. The specific process is as follows:

[0044]

[0045] Where sg represents the sigmoid activation function, cv3 is a 3D convolutional layer with a convolution kernel size of 1×1×5, cv represents a convolutional layer with a convolution kernel size of 3×3, and cat represents the concatenation operation.

[0046] Step 4: Use the feature enhancement task head that guides optical flow learning to output the enhanced infrared image, fuse the multi-scale features through the target segmentation task head, output the target segmentation result, and then guide the learning of the feature enhancement network. The specific steps are as follows:

[0047] Step 41: Feature enhancement to guide optical flow learning The task head consists of upsampling, skip connection structure, and convolutional layers with a convolution kernel size of 1×1. First, the feature map output by the EnE module Use an upsampling to restore the original image size, and then use the jump connection structure to connect the input feature F4 and the output feature Finally, the convolution layer with a convolution kernel size of 1×1 outputs the enhanced single infrared image.

[0048] Step 42: Use motion-aware loss to constrain network training and decompose the input multi-frame image into two parts: the front image and the following image use and Perform backward warp operations on them respectively to obtain the estimated posterior image and the estimated front image for:

[0049]

[0050]

[0051] Mean Squared Error (MSE) is used to measure the similarity between the original image and the reconstructed image, so the final forward flow loss is And backward Flow loss They are:

[0052]

[0053]

[0054] Where H 10 and W 10 represents the height and width of the output enhanced image, and t, h, and w represent the h-th row and w-th column of the t-th frame image respectively.

[0055] The final optical flow loss for:

[0056]

[0057] Step 43: The target segmentation head structure consists of a two-dimensional convolution layer with a convolution kernel of 3×3, a batch normalization layer, and a two-dimensional convolution layer with a convolution kernel of 1×1. The target segmentation head uses the concatenate operation to splice multi-scale features. The two-dimensional convolution layer with a convolution kernel of 3×3 extracts local features of the image. The batch normalization layer standardizes the input to make the training process more stable. The two-dimensional convolution layer with a convolution kernel of 1×1 fuses multi-scale features and finally outputs the target segmentation result. The loss is calculated for the segmentation result and the target segmentation true value, thereby optimizing the network weights.

[0058] Step 5: Use two public moving infrared small target data sets to test and verify the algorithm. The specific steps are as follows:

[0059] This paper uses the public TSIRMT and NUDT-MIRSDT datasets to conduct experiments. The two datasets are divided into training sets and test sets with a ratio of 7:3. In all experiments, the batch size is set to 1, the number of frames of multi-frame moving infrared small target images is set to 5, and the AdamW optimization algorithm is used for training 30 rounds. The initial learning rate is set to 1×10 -4 , using a polynomial strategy to decay to 1×10 -7 , the decay index is 0.9. SCRG is used to quantitatively evaluate the algorithm performance. The SCRG calculation formula is as follows:

[0060]

[0061] SCR input Represents the signal-to-clutter ratio of the input infrared image, SCR output Indicates the signal-to-noise ratio of the output infrared image.

[0062] The experimental results are as follows Figure 4 As shown, it can be observed that the local motion-aware infrared small target feature enhancement method proposed in the present invention has excellent performance, and the infrared image output after enhancement has obvious target features, and can well adapt to scenes with dim targets and background motion.

Claims

1. A method for enhancing the characteristics of infrared small targets based on local motion perception, characterized in that The method comprises the following steps: Step 1: Load the infrared sequence image, select T consecutive frames of images as network input, and use the backbone network to extract image features; Step 2: Use the coarse motion estimation module CME to extract rough target inter-frame motion information and generate forward optical flow and backward optical flow. The specific steps are as follows: Step 21: The CME module consists of 4 layers of convolution. The first three layers of convolution use expansion coefficients of 8, 16, and 1 to capture different sizes of inter-frame displacements. The last layer of convolution maps the inter-frame differences into the horizontal and vertical components of the optical flow. The final output is used as a rough motion pointing vector. Step 22: Input features The adjacent frame features are stacked together to extract the feature differences between frames, where {·} represents the set, C2 represents the number of feature channels, H2 represents the feature map height, W2 represents the feature map width, and F 4，i Represents the fourth layer i-th frame feature output by the backbone network, using the parameter-sharing CME module to output the forward optical flow and backward optical flow in represents the optical flow from the pth frame to the qth frame, N represents the total number of frames, and the dimensions of the forward and backward optical flows are the same, both are Where T-1 represents the number of optical flows, H3 represents the height of the optical flow map, W3 represents the width of the optical flow map, and 2 represents the horizontal and vertical components of the optical flow; Step 3: Use the energy enhancement module EnE to combine optical flow and deformable convolution to align multi-frame sequences, and use 3D convolution with a convolution kernel size of 1×1 to enhance the target features. The specific steps are as follows: Step 31: When processing the features of the adjacent 5 frames, first use the backward warp operation to convert the forward optical flow and backward optical flow Expanded to two adjacent frames, the expanded optical flow and They are as follows: In the formula Represents the optical flow from the pth frame to the qth frame; Step 32: After aligning each frame to the middle frame, use the concatenate operation to splice them together to obtain the alignment feature F a : Where {·} represents the set, and the alignment feature F a satisfy Where C3 represents the number of feature channels in the fourth layer, H4 represents the height of the feature map in the fourth layer, and W4 represents the width of the feature map in the fourth layer; Step 33: Align the feature F a , the intermediate frame of the input feature F4 and the forward and backward optical flow along the first dimension are spliced ​​together to form the feature Where C4 represents the number of spliced ​​feature channels, H5 represents the height of the spliced ​​feature map, and W5 represents the width of the spliced ​​feature map; Step 3 and 4: Use the deformation attention module DAF to learn the target shape and accurately align the features of each frame; Step 4: Use the feature enhancement task head that guides optical flow learning to output the enhanced infrared image, fuse the multi-scale features through the target segmentation task head, output the target segmentation result, and then guide the learning of the feature enhancement network.

2. The local motion sensing infrared small target feature enhancement method according to claim 1 is characterized in that The specific steps of step one are as follows: Step 1: Load multiple frames of images As the backbone network input, T represents the number of infrared sequence image frames contained in each batch in the network input, H1 represents the input image height, W1 represents the input image width, and 3 represents that the number of infrared image channels is mapped to 3 during data loading; Step 1 and 2: The backbone network uses the Video Swin Transformer network to input multiple frames of images The image blocks are divided into T / 2×H1 / 4×W1 / 4 blocks at a scale of 2×4×4×3. Each image block contains 96-dimensional features, which are then mapped to dimension C1 after embedding. The network uses 3D non-overlapping windows and translation windows to divide the image blocks to reduce the complexity of Transformer calculation and extract the global features of the image sequence. Finally, four layers of features F1, F2, F3 and F4 are output, with corresponding dimensions of T×H1 / 8×W1 / 8×2C1, T×H1 / 16×W1 / 16×4C1, T×H1 / 32×W1 / 32×4C1 and T×H1 / 32×W1 / 32×4C1, respectively. The multi-frame top-level feature F4 containing the richest semantic information is input into the subsequent network to extract the target spatiotemporal correlation features.

3. The local motion sensing infrared small target feature enhancement method according to claim 1 is characterized in that The specific steps of steps three and four are as follows: Step 341: Use a convolution layer with a convolution kernel size of 3×3 and a Leaky ReLU activation function to concatenate the features F b It is divided into two branches, one of which is added to the optical flow offset to generate the sampling point offset The other branch generates normalized attention weights Among them, H6 represents the height of the sampling point offset tensor, W6 represents the width of the sampling point offset tensor, H7 represents the height of the attention weight matrix, W7 represents the width of the attention weight matrix, M represents the number of attention heads, and P represents the number of sampling points. Used to find pixels with the same name in adjacent frames, Used to characterize the possibility of a pixel being a target; Step 342: The original input feature F4 is flattened after two layers of convolution Among them, H8 represents the tensor height, W8 represents the tensor width, and C5 represents the number of tensor channels; Step 343: and The calculation methods are as follows: Where cv represents the convolution layer with a convolution kernel size of 3×3, LR represents the LeakyReLu activation function, cat represents the concatenation operation, and Fgo represents the optical flow offset; Step 344: Transform the attention module and Acts on And output features Among them, C6 is the number of output feature channels, H9 is the height of the output feature map, and W9 is the width of the output feature map. At the same time, residual connections are used to improve the optimization efficiency of the network. The output F4 is combined with the convolution Perform addition operation to generate And will After connecting with F4, two layers of convolution operations are performed to generate To fuse the original features and the enhanced features, and After addition, 3D convolution is used to accumulate the target energy, and the sigmoid activation function is used to normalize the output of the local spatiotemporal attention weight. The specific process is as follows: Where sg represents the sigmoid activation function, cv3 is a 3D convolutional layer with a convolution kernel size of 1×1×5, cv represents a convolutional layer with a convolution kernel size of 3×3, and cat represents the concatenation operation.

4. The method for enhancing the characteristics of infrared small targets based on local motion perception according to claim 3 is characterized in that The specific steps of step 4 are as follows: Step 41: Feature enhancement to guide optical flow learning The task head consists of upsampling, skip connection structure, and convolutional layers with a convolution kernel size of 1×1. First, the feature map output by the EnE module Use an upsampling to restore the original image size, and then use the jump connection structure to connect the input feature F4 and the output feature Finally, the convolution layer with a convolution kernel size of 1×1 outputs the enhanced single infrared image; Step 42: Use motion-aware loss to constrain network training and decompose the input multi-frame image into two parts: the front image and the following image use and Perform backward warp operations on them respectively to obtain the estimated posterior image and the estimated front image for: MSE is used to measure the similarity between the original image and the reconstructed image, and the forward flow loss is obtained. And backward Flow loss They are: Where H 10 and W 10 represents the height and width of the output enhanced image, t, h, and w represent the h-th row and w-th column of the t-th frame image respectively; The final optical flow loss for: Step 43: The target segmentation head structure consists of a two-dimensional convolution layer with a convolution kernel of 3×3, a batch normalization layer, and a two-dimensional convolution layer with a convolution kernel of 1×1. The target segmentation head uses the concatenate operation to splice multi-scale features. The two-dimensional convolution layer with a convolution kernel of 3×3 extracts local features of the image. The batch normalization layer standardizes the input to make the training process more stable. The two-dimensional convolution layer with a convolution kernel of 1×1 fuses multi-scale features and finally outputs the target segmentation result. The loss is calculated for the segmentation result and the target segmentation true value, thereby optimizing the network weights.

Images