An infrared image background suppression method based on zero-order learning

Abstract

The application provides an infrared image background suppression method based on zero-order learning, comprising the following steps: (1) reading in infrared time-series images of visible classes, corresponding labels of each class and corresponding spectral data of each class; (2) extracting infrared image features, label text features, time-series features and spectral features by using the data in step (1); (3) realizing zero-order learning training based on the features in step (2), and obtaining a function group composed of target and background feature mapping functions; (4) reading in infrared images of invisible classes, constructing time-space local contrast, and judging whether each region belongs to a background class according to a threshold value; (5) judging whether it belongs to a visible class background by similarity based on the feature mapping function group obtained in step (3); (6) adding a new feature mapping function obtained by step (3) to the function group for the background divided into invisible classes; and (7) reconstructing the background based on the function group, and realizing background suppression by using background difference and non-local mean filtering.

Description

Technical Field

[0001] This invention discloses an infrared image background suppression method based on zero-learning, belonging to the field of intelligent image data processing technology. It is applicable to the research of intelligent removal methods for complex backgrounds in infrared images and precise intelligent target recognition methods. Background Technology

[0002] During the jamming phase, space targets fly in a deep-space environment, which is subject to various background interferences. These include sunlight, darkness, and starlight, all of which can affect target identification. Additionally, strong light interference and the uneven light spots and noise generated by the detector's imaging mechanism can also interfere with identification. Furthermore, the impact of infrared image background on target identification is caused by multiple complex factors, and the relationships and relationships between these factors are not fully understood. This results in incomplete background features in infrared images. Current research on space-based infrared image backgrounds is limited, with insufficient data, leading to limited learning of background features and inadequate background analysis.

[0003] For intelligent processing of infrared images, especially in target recognition and detection tasks in complex environments, the diversity and complexity of image backgrounds pose a significant challenge to the robustness of algorithms. The complex backgrounds of deep space and the influence of sensors both increase the difficulty of image processing. Although deep learning technology has made significant progress in image processing, existing models may still experience performance degradation when faced with unknown or complex backgrounds due to the diversity and incompleteness of background features. Furthermore, the scarcity of deep space image datasets limits the generalization ability of models in these scenarios.

[0004] Background modeling is an important area in intelligent infrared image processing. It involves estimating the background and then distinguishing it from the foreground to reduce its influence. Initially, background modeling mainly relied on traditional methods, such as filtering and thresholding, to analyze the background and separate the target from the background. With the development of research on the background of spatial targets in infrared images, more and more methods have begun to use neural networks to mine the latent characteristics of the background and the relationship between background factors and the target to model and analyze the background. Most current methods need to consider the correlation between image features and then realize the application requirements of background suppression or target recognition by mining the differences between the target and the background.

[0005] Zero-time learning can enable the recognition of invisible images by constructing a mapping relationship between the semantic features of visible data labels and image features during the training phase, even when the recognition data and training data are unrelated. However, infrared spatial target detection and recognition in deep space environments suffers from significant interference due to the small size of the targets, lack of texture and shape characteristics, and complex background conditions. Stars with similar brightness and pixel size to the target in the starry background, high-brightness noise points, and non-uniform image patches can all cause considerable interference.

[0006] However, existing deep space infrared image background modeling and suppression methods still have the following problems: (1) Existing methods do not make full use of the relationship between the background and the target. Most of them only conduct separate studies on the background or the target, ignoring the mutual relationship between their features; (2) The influencing factors of deep space background features are complex and the influencing factors are coupled with each other to affect the image quality. Some background features are difficult to effectively represent and many influencing factors are not clear; (3) Due to the incomplete background features, there are few effective features, making it difficult to construct a complete background feature model. Summary of the Invention

[0007] The purpose of this invention is to address the problems faced in deep space science research, such as complex and coupled factors affecting image background, incomplete background features, and space targets being submerged in the background, by providing an infrared image background suppression method based on zero-step learning.

[0008] The technical solution of this invention is as follows: A background suppression method for infrared images based on zero-shot learning is proposed. Semantic representations of visible image features and label text features are extracted as the main features for the visible and invisible classes in the zero-shot learning network structure. Simultaneously, temporal and spectral features are extracted as auxiliary features for the data representation. The feature mapping group for the visible class is obtained through model training. In the detection stage, the traditional temporal-spatial double-frame local contrast method is combined to determine which regions of the image belong to the background. Furthermore, the feature mapping function group is used to calculate distances to determine whether the background region belongs to the visible background. By combining traditional methods with neural network methods, background modeling under zero-shot conditions is finally achieved. Finally, by setting a feedback function, the invisible class is added to the visible class feature mapping group.

[0009] This invention is an infrared image background suppression method based on zero-learning, the steps of which are as follows:

[0010] (1) Read in visible infrared time-series images, corresponding labels for each type, and corresponding spectral data for each type;

[0011] (2) Construct a zero-learning model. Using the data from step (1), extract infrared image features based on the VGG-16 model with dilated convolution, extract label text features based on the bidirectional encoder model of Transformer and use them as the main features of the zero-learning model. Extract temporal features of temporal images based on dilated convolution and causal convolution, and extract spectral features based on the encoder of Transformer network and use them as auxiliary features of the zero-learning model.

[0012] (3) Construct a zero-time learning model. Based on the main features and auxiliary features extracted in step (2), train the zero-time learning model based on Transformer and obtain the target and background feature mapping function to form a function group.

[0013] (4) Read in the invisible infrared image, construct the temporal-spatial local contrast, and determine whether the regions traversed by the invisible data sliding box are judged as invisible regions or background regions.

[0014] (5) Based on whether the data read in step (4) has corresponding auxiliary features, use the matrix to adjust the mapping function group obtained in step (3), and use the feature mapping group to calculate the similar distance to determine whether the background area belongs to the visible or invisible class by performing a sliding box traversal on the image.

[0015] (6) The backgrounds classified as invisible are obtained by step (3) and then added to the visible background feature mapping function group;

[0016] (7) The background image is reconstructed based on the background feature mapping function group. The background is removed by difference between the original image and the background image and optimized by non-local mean filtering to achieve background suppression of infrared image.

[0017] In step (1), the visible infrared time-series image, the corresponding labels of each type, and the corresponding spectral data of each type are read in.

[0018] In step (2), a zero-learning model is constructed. Using the data from step (1), infrared image features are extracted based on the VGG-16 model with dilated convolution, and label text features are extracted based on the bidirectional encoder model of Transformer. These are used as the main features of the zero-learning model. Temporal features of the temporal image are extracted based on dilated convolution and causal convolution, and spectral features are extracted based on the encoder of the Transformer network. These are used as auxiliary features of the zero-learning model. In this invention, the VGG-16 model based on dilated convolution is used to extract infrared image features W.

[0019]

[0020]

[0021] Where K is the filter sequence, a is the dilation rate of the dilated convolution, k is the original kernel size of the network, w is the size of the input image or feature map, S is the stride, and P is the feature map boundary padding. The image feature extraction network structure consists of three convolutional layers, two multi-scale feature fusion modules based on dilated convolution, and a fully connected layer. The node output of each layer of the network... as follows:

[0022]

[0023] Where x is the feature input, d is the layer number, and t is the input sequence. For kernel function weights, The correlation function is defined as the number of layers and the dilation rate. The image target description is used as the semantic label corresponding to the background and target features. For the label text, a Transformer-based bidirectional encoder model is used to embed the pre-trained text to obtain the label text features. The algorithm employs label embedding, segment embedding, and positional embedding. The input labels are in the form of (batch_size, seq_len). Label embedding maps the input to a vector of dimension d_model, where batch_size is the number of samples in each batch and seq_len is the maximum length of the label text, thus obtaining the label embedding features. Segment embedding maps the information of sentences in the input into a vector of dimension d_model. Therefore, after this layer, the form becomes (batch_size, seq_len, d_model). However, since the natural language in this paper consists of feature labels for the target and background, and there are no sentences, the segment embedding will not be modified to [1,1,...1], resulting in segment embedding features. Position embedding maps the positional information of the input background and the target word to a vector of d_model, thus obtaining the position embedding features. This leads to the high-dimensional vector. :

[0024]

[0025] The Transformer-based bidirectional encoder model consists of L layers of Transformers, ultimately yielding... The system utilizes dilated convolution and causal convolution to operate on the input sequence of images, extracting time-series features corresponding to the image features, and outputting the results at each time step. All depend only on the num times before that moment. The value of:

[0026]

[0027] in, For causal convolutional networks, For the first num features of the current value, after each layer of dilated causal convolution, the output is normalized and ReLU nonlinear calculation is performed; the encoder of the Transformer network is selected to extract the spectral features of the spatial target, and the final sequence input into the network can be represented as:

[0028]

[0029] in, Represents the embedding vector of the text input sequence, where For sequence index, For time indexing, This represents the projection layer that maps word vectors to D-dimensional vectors, where P represents the word relative position embedding. The last layer of the position embedding network outputs the spectral feature embedding vector. It can be represented as:

[0030]

[0031] After spectral features are extracted using a Transformer, a real-time updated soft attention mechanism is employed. The similarity function of this spectral attention mechanism... and characteristics Represented as:

[0032]

[0033]

[0034]

[0035] in, For key vectors, For query vector, For value vectors, The weight matrix represents the weights of the first layer of the Transformer network. The self-attention mechanism in the network is replaced by a simulated spectral spectra. The encoder output of the Transformer network is as follows:

[0036]

[0037]

[0038] in, For the output characteristics of the encoder, For the final spectral characteristics, For spectral characteristics, as well as As weight, as well as This is the bias value.

[0039] In step (3), a zero-order learning model is constructed. Based on the main features and auxiliary features extracted in step (2), the zero-order learning model based on Transformer is trained, and the target and background feature mapping functions are obtained to form a function group. After extracting the features required to represent the data, a zero-order learning model is constructed. By constructing a mapping relationship between the image features and auxiliary features of the target and background and the label text features of the target, the visible domain data is defined as... ,in, Image features for the visible class, For the complete set of visible image features, For visible label text features, The complete set of visible class label text features, For image temporal features, For the complete set of temporal features of visible images, For spectral characteristics, The visible class is the complete set of spectral features, while the invisible class is defined as... ,in, Image features for invisible classes, For the complete set of features of the invisible class image, For invisible text features, For the complete set of invisible class label text features, Temporal features of invisible images For the complete set of temporal features of invisible images, It is an invisible spectral feature. The complete set of invisible spectral features is used; a preliminary mapping function is formed by nonlinear mapping of image features, auxiliary features, and label text features using a multilayer perceptron.

[0040]

[0041] in, For image feature mapping function, As the auxiliary feature mapping function, p and q are the weights, p+q=1; then a Transformer network is used to train the mapping model, and the loss function is established based on the cross-entropy loss function. By introducing a similarity metric, the zero-step learning network learns the visible class feature mapping function by optimizing the similarity between features, thus improving the similarity between images and text, and between auxiliary features and image features. and loss function The calculation formula is as follows, where These are the weights, and :

[0042]

[0043]

[0044] Finally, the final mapping function is obtained by training the mapping function through a Transformer network:

[0045]

[0046] in, To ultimately train and obtain the image feature mapping function, This is used to obtain the auxiliary feature mapping function for the final training.

[0047] In step (4), the invisible infrared image is read in, a temporal-spatial local contrast is constructed, and it is determined whether the regions traversed by the invisible data sliding frame are background regions. The invisible infrared image is read in, and the outer frame uses the changes between the inner and outer frames and the comparison between the four corners of the outer frame to compare the changes between the inner and outer frames. The inner frame calculates the difference between the center point of each sliding frame region and the average value of the sliding frame, and multiplies it with the difference between the center region and the opposite direction to obtain the contrast value. as follows:

[0048]

[0049] in, For slider direction index, Center of the image slider Pixel value, as well as For the grayscale value of each slider region at that point, a temporal feature is introduced. The difference between the previous frame and the current frame, and between the next frame and the current frame, are compared. The preceding and following frames are selected as the temporal range, and the updated comparison value is... The formula is as follows:

[0050]

[0051] Where t is the current frame, the updated local contrast is represented as follows. To update the coefficients, the updated overall judgment index is... as follows:

[0052]

[0053] in, ,when Less than If there is no change between the preceding and following frames, it is judged as the background; set the grayscale ratio between the two frames in the current frame and the change in grayscale values ​​with the preceding and following frames, and set the fluctuation of the product of the changes between the preceding and following frames to be greater than a threshold. At that time, it is considered that the background has changed significantly:

[0054]

[0055]

[0056]

[0057] in, Here, represents the inner and outer lengths of the double-layered frame; aver1 is the average value of the inner frame; aver2 is the average value of the outer frame; and gray represents the pixel grayscale value within the area. These are the values ​​for the four corners of the outer frame. The ratio of the two frames at frame t is given by , and 'dif' is the product of the grayscale value ratio changes between the current frame and the previous and next frames. Furthermore, three boundary points are set on each side of the outer frame. When the background boundary point is located on any one of the four edges AB, BC, CD, or DA, the difference between these boundary points can be used to determine whether the boundary will pass through any of these four edges.

[0058]

[0059] Where diff represents the difference between the two points. and These are the values ​​of a certain side point. , Introducing target motion information, this involves combining the target's motion distance, direction and its weights, velocity, and acceleration features. The average distance between the target point in each of the previous five frames is used as the target motion distance between the current frame and the previous frame. Simultaneously, the target movement changes from the previous two frames are also considered. The movement changes of the target in the current frame are used to predict the target's motion state and added to a temporary storage group as a reference for subsequent motion.

[0060]

[0061]

[0062] Where dis is the average distance of the previous five frames, dirx is the average distance in the X direction of the previous five frames (X,Y), diry is the average distance in the Y direction of the previous five frames (X,Y), and dir_mom is the position state of the current frame. From this, the predicted position of the target in the current frame is obtained. The target's velocity spe and acceleration acc are calculated using the relative distance between targets in different frames and the frame rate.

[0063]

[0064]

[0065] in, This refers to the positions of the current frame and the previous frame. fre is the frame rate.

[0066] In step (5), depending on whether the data read in step (4) has corresponding auxiliary features, the mapping function group obtained in step (3) is adjusted using a matrix. By performing a sliding box traversal on the image, the similarity distance is calculated using the feature mapping group to determine whether the background region belongs to the visible or invisible class. In the recognition stage, since the invisible class and the visible class share the semantic space, the obtained mapping function group can be transferred to the invisible domain. At the same time, when facing the lack of auxiliary features in the recognition of invisible class images, the auxiliary features of each mapping function group are adjusted to form a new mapping function by adjusting the feature mapping function group obtained by this method to form an identity matrix to solve the problem of lack of auxiliary features. Then, the label text features obtained by the new image through the mapping function are compared with the label text features of the trained feature mapping function group. The minimum distance between two points is used to determine whether an image belongs to a known domain, thereby enabling the recognition of unknown image backgrounds. The formula for distance D is as follows:

[0067]

[0068] in, , For visible class label text features, For invisible class label text features, These are the predicted label text features. These are the actual label text features. It is a similarity function, calculated using the cosine function. When D is greater than the threshold... When the time is visible, it is less than the threshold. When the invisible class is identified, the identified invisible class and auxiliary features are added to the training network to obtain a new background feature mapping function and add it to the feature mapping function group.

[0069] In step (6), the background is classified as invisible. A new background feature mapping function is obtained through step (3) and then added to the visible feature mapping function group: the network structure is set as a recurrent feedback structure, and the background features of the invisible class are classified and formed into a new mapping function and added to the feature mapping function group.

[0070] In step (7), the background image is reconstructed based on the background feature mapping function set. The background is removed by difference between the original image and the background image and optimized by non-local mean filtering to achieve infrared image background suppression. Feature fusion between infrared image features and label text features is achieved by element-wise multiplication to generate features. Where h is the height of the feature map, w is the width of the feature map, and c is the number of channels, a lightweight U-Net generator network is introduced to decode it into a background image. z is progressively upsampled through multiple transposed convolutions, and finally an infrared background image matching the target size is output. , Generate a network for U-Net. The parameters of the U-Net network are generated by combining pixel-level L1 loss and structural similarity loss to obtain the loss function. Optimize the similarity of the model at the pixel level and the similarity of structural information:

[0071]

[0072] in, For weight hyperparameters, It is a structural similarity index. For the generated background image, To obtain a more refined differential image, the original infrared image is subtracted from the infrared background image to obtain a more realistic background image. For each pixel in the difference image Calculate its relationship with other pixels in the neighborhood. Similarity weight :

[0073]

[0074] in, In pixels The image patch centered on For each pixel of the difference image, the filtering parameters are... Perform filtering operation:

[0075]

[0076] in, For the search window, cover pixels The neighborhood of the target image is then used to obtain a clean target image. This achieves background suppression.

[0077] Compared with existing background modeling and suppression methods, the advantages of this invention are: (1) Based on the principle that target and background research are inseparable and complementary, the zero-learning method is used to study complex background features in reverse through target features and visible background features, which can make full use of the features of the target and background and the relationship between them; (2) The image features and label semantic features of the target are extracted as data representations of visible and invisible classes respectively. At the same time, the temporal features of the target and background and the spectral features of the spatial target are used as auxiliary features to enhance the effectiveness of the features, so as to learn the background features of the invisible class more fully and comprehensively and to represent them more effectively through the feature mapping function; (3) The traditional algorithm is combined with the neural network technology to realize the identification of the invisible background category based on the zero-learning model and to construct the background feature mapping. In addition, a loop structure is added to integrate the invisible background into the visible class, so as to continuously expand the background feature mapping group in the process of identifying the invisible background features and enhance the generalization of the model. Detailed Implementation

[0078] This invention is an infrared image background suppression method based on zero-learning, and the specific implementation steps are as follows:

[0079] (1) Read in the visible infrared time series image, the corresponding labels of each type and the corresponding spectral data of each type.

[0080] (2) Construct a zero-learning model. Using the data from step (1), extract infrared image features based on the VGG-16 model with dilated convolution, extract label text features based on the bidirectional encoder model of Transformer, and use these as the main features of the zero-learning model. Extract temporal features of the temporal image based on dilated convolution and causal convolution, and extract spectral features based on the encoder of the Transformer network, and use these as auxiliary features of the zero-learning model. This invention uses the VGG-16 model based on dilated convolution to extract infrared image features W.

[0081]

[0082]

[0083] Where K is the filter sequence, a is the dilation rate of the dilated convolution, k is the original kernel size of the network, w is the size of the input image or feature map, S is the stride, and P is the feature map boundary padding. The image feature extraction network structure consists of three convolutional layers, two multi-scale feature fusion modules based on dilated convolution, and a fully connected layer. The node output of each layer of the network... as follows:

[0084]

[0085] Where x is the feature input, d is the layer number, and t is the input sequence. For kernel function weights, The correlation function is defined as the number of layers and the dilation rate. The image target description is used as the semantic label corresponding to the background and target features. For the label text, a Transformer-based bidirectional encoder model is used to embed the pre-trained text to obtain the label text features. The algorithm employs label embedding, segment embedding, and positional embedding. The input labels are in the form of (batch_size, seq_len). Label embedding maps the input to a vector of dimension d_model, where batch_size is the number of samples in each batch and seq_len is the maximum length of the label text, thus obtaining the label embedding features. Segment embedding maps the information of sentences in the input into a vector of dimension d_model. Therefore, after this layer, the form becomes (batch_size, seq_len, d_model). However, since the natural language in this paper consists of feature labels for the target and background, and there are no sentences, the segment embedding will not be modified to [1,1,...1], resulting in segment embedding features. Position embedding maps the positional information of the input background and the target word to a vector of d_model, thus obtaining the position embedding features. This leads to the high-dimensional vector. :

[0086]

[0087] The Transformer-based bidirectional encoder model consists of L layers of Transformers, ultimately yielding... The system utilizes dilated convolution and causal convolution to operate on the input sequence of images, extracting time-series features corresponding to the image features, and outputting the results at each time step. All depend only on the num times before that moment. The value of:

[0088]

[0089] in, For causal convolutional networks, For the first num features of the current value, after each layer of dilated causal convolution, the output is normalized and ReLU nonlinear calculation is performed; the encoder of the Transformer network is selected to extract the spectral features of the spatial target, and the final sequence input into the network can be represented as:

[0090]

[0091] in, Represents the embedding vector of the text input sequence, where For sequence index, For time indexing, This represents the projection layer that maps word vectors to D-dimensional vectors, where P represents the word relative position embedding. The last layer of the position embedding network outputs the spectral feature embedding vector. It can be represented as:

[0092]

[0093] After spectral features are extracted using a Transformer, a real-time updated soft attention mechanism is employed. The similarity function of this spectral attention mechanism... and characteristics Represented as:

[0094]

[0095]

[0096]

[0097] in, For key vectors, For query vector, For value vectors, The weight matrix represents the weights of the first layer of the Transformer network. The self-attention mechanism in the network is replaced by a simulated spectral spectra. The encoder output of the Transformer network is as follows:

[0098]

[0099]

[0100] in, For the output characteristics of the encoder, For the final spectral characteristics, For spectral characteristics, as well as As weight, as well as This is the bias value.

[0101] (3) Construct a zero-order learning model. Based on the main features and auxiliary features extracted in step (2), train the zero-order learning model based on Transformer and obtain the target and background feature mapping function to form a function group: After extracting the features required to represent the data, construct a zero-order learning model. By constructing a mapping relationship between the image features and auxiliary features of the target and background and the label text features of the target, define the visible domain data as ,in, Image features for the visible class, For the complete set of visible image features, For visible label text features, The complete set of visible class label text features, For image temporal features, For the complete set of temporal features of visible images, For spectral characteristics, The visible class is the complete set of spectral features, while the invisible class is defined as... ,in, Image features for invisible classes, For the complete set of features of the invisible class image, For invisible text features, For the complete set of invisible class label text features, Temporal features of invisible images For the complete set of temporal features of invisible images, It is an invisible spectral feature. The complete set of invisible spectral features is used; a preliminary mapping function is formed by nonlinear mapping of image features, auxiliary features, and label text features using a multilayer perceptron.

[0102]

[0103] in, For image feature mapping function, As the auxiliary feature mapping function, p and q are the weights, p+q=1; then a Transformer network is used to train the mapping model, and the loss function is established based on the cross-entropy loss function. By introducing a similarity metric, the zero-step learning network learns the visible class feature mapping function by optimizing the similarity between features, thus improving the similarity between images and text, and between auxiliary features and image features. and loss function The calculation formula is as follows, where These are the weights, and :

[0104]

[0105]

[0106] Finally, the final mapping function is obtained by training the mapping function through a Transformer network:

[0107]

[0108] in, To ultimately train and obtain the image feature mapping function, This is used to obtain the auxiliary feature mapping function for the final training.

[0109] (4) Read in the invisible infrared image, construct the temporal-spatial local contrast, and determine whether the regions traversed by the invisible data sliding box are judged as invisible regions or background regions: Read in the invisible infrared image, the outer layer of the sliding box uses the changes between the inner and outer frames and the comparison between the four corners of the outer frame, and the inner frame calculates the difference between the center point of each sliding box region and the average value of the sliding box, and multiplies it with the difference between the center region in the opposite direction to obtain the contrast value. The formula is as follows:

[0110]

[0111] in, For slider direction index, Center of the image slider Pixel value, as well as For each slider region at a given point, a temporal feature is introduced. The differences between the previous and current frames, and between the next and current frames, are compared. The preceding and following frames are selected as the temporal range, and the updated judgment parameters are then used. The formula is as follows:

[0112]

[0113] Where t is the current frame, the updated local contrast is represented as follows. To update the coefficients, the updated overall judgment index is... as follows:

[0114]

[0115] in, ,when Less than If there is no change between the preceding and following frames, it is judged as the background; set the grayscale ratio between the two frames in the current frame and the change in grayscale values ​​with the preceding and following frames, and set the fluctuation of the product of the changes between the preceding and following frames to be greater than a threshold. At that time, it is considered that the background has changed significantly:

[0116]

[0117]

[0118]

[0119] in, Let m be the inner and outer dimensions of the double-layered frame, where m is 5, n is 7, aver1 is the average value of the inner frame, aver2 is the average value of the outer frame, and gray is the pixel grayscale value within the region. These are the values ​​for the four corners of the outer frame. The ratio of the two frames at frame t is given by , and 'dif' is the product of the grayscale value ratio changes between the current frame and the previous and next frames. Furthermore, three boundary points are set on each side of the outer frame. When the background boundary point is located on any one of the four edges AB, BC, CD, or DA, the difference between these boundary points can be used to determine whether the boundary will pass through any of these four edges.

[0120]

[0121] Where diff represents the difference between the two points. and These are the values ​​of a certain side point. , Introducing target motion information, this involves combining the target's motion distance, direction and its weights, velocity, and acceleration features. The average distance between the target point in each of the previous five frames is used as the target motion distance between the current frame and the previous frame. Simultaneously, the target movement changes from the previous two frames are also considered. The movement changes of the target in the current frame are used to predict the target's motion state and added to a temporary storage group as a reference for subsequent motion.

[0122]

[0123]

[0124] Where dis is the average distance of the previous five frames, dirx is the average distance in the X direction of the previous five frames (X,Y), diry is the average distance in the Y direction of the previous five frames (X,Y), and dir_mom is the position state of the current frame. From this, the predicted position of the target in the current frame is obtained. The target's velocity spe and acceleration acc are calculated using the relative distance between targets in different frames and the frame rate.

[0125]

[0126]

[0127] in, This refers to the positions of the current frame and the previous frame. fre is the frame rate, set to 50.

[0128] (5) Based on whether the data read in step (4) has corresponding auxiliary features, adjust the mapping function group obtained in step (3) using a matrix. By performing a sliding box traversal on the image, use the feature mapping group to calculate the similarity distance to determine whether the background region belongs to the visible or invisible class. In the recognition stage, since the invisible class and the visible class share the semantic space, the obtained mapping function group can be transferred to the invisible domain. At the same time, when facing the lack of auxiliary features in the recognition of invisible class images, the auxiliary features of each mapping function group are adjusted to form a new mapping function by adjusting the feature mapping function group obtained by this method to form an identity matrix to solve the problem of lack of auxiliary features. Then, the label text features obtained by the new image through the mapping function are compared with the label text features of the trained feature mapping function group. The minimum distance between two points is used to determine whether an image belongs to a known domain, thereby enabling the recognition of unknown image backgrounds. The formula for distance D is as follows:

[0129]

[0130] in, , For visible class label text features, For invisible class label text features, These are the predicted label text features. These are the actual label text features. It is a similarity function, calculated using the cosine function. When D is greater than the threshold... When the time is visible, it is less than the threshold. When the invisible class is identified, the identified invisible class and auxiliary features are added to the training network to obtain a new background feature mapping function and add it to the feature mapping function group.

[0131] (6) The background is classified as invisible. A new background feature mapping function is obtained through step (3) and then added to the visible class feature mapping function group: The network structure is set as a recurrent feedback structure, and the background features of the invisible class are classified and formed into a new mapping function and added to the feature mapping function group.

[0132] (7) Reconstruct the background image based on the background feature mapping function set, remove the background by difference between the original image and the background image and optimize by non-local mean filtering to achieve infrared image background suppression: use element-wise multiplication to achieve feature fusion between infrared image features and label text features to generate features. Where h is the height of the feature map, w is the width of the feature map, and c is the number of channels, a lightweight U-Net generator network is introduced to decode it into a background image. z is then progressively upsampled through multiple transposed convolutions to finally output an infrared background image that matches the target size. , Generate a network for U-Net. The parameters of the U-Net network are generated by combining pixel-level L1 loss and structural similarity loss to obtain the loss function. Optimize the similarity of the model at the pixel level and the similarity of structural information:

[0133]

[0134] in, For weight hyperparameters, It is a structural similarity index. For the generated background image, To obtain a more refined differential image, the original infrared image is subtracted from the infrared background image to obtain a more realistic background image. For each pixel in the difference image Calculate its relationship with other pixels in the neighborhood. Similarity weight :

[0135]

[0136] in, In pixels The image patch centered on For each pixel of the difference image, the filtering parameters are... Perform filtering operation:

[0137]

[0138] in, For the search window, cover pixels The neighborhood of the target image is then used to obtain a clean target image. This achieves background suppression.

[0139] The embodiments of the present invention have been described in detail above, but the content described is only a preferred embodiment of the present invention and should not be considered as limiting the scope of the present invention. The implementation of each step can vary, and all equivalent changes and improvements made within the scope of the present invention should still fall within the scope of this patent.

Claims

1. A background suppression method for infrared images based on zero-learning, characterized in that, It includes the following steps: (1) Read in the infrared time series images of the visible class, the corresponding labels of each class and the corresponding spectral data of each class. Using the read data, extract the infrared image features based on the VGG-16 model of dilated convolution, extract the label text features based on the bidirectional encoder model of Transformer and use them as the main features of the zero-time learning model. Extract the time series features of the time series images based on dilated convolution and causal convolution, and extract the spectral features based on the encoder of Transformer network and use them as the auxiliary features of the zero-time learning model. (2) Construct a zero-time learning model. Based on the main features and auxiliary features extracted in step (1), train the zero-time learning model based on Transformer and obtain the target and background feature mapping function to form a function group. (3) Read in the invisible infrared image, construct the temporal-spatial local contrast, and determine whether the regions traversed by the invisible data sliding box are judged as invisible regions or background regions. (4) Based on whether the data read in step (3) has corresponding auxiliary features, use the matrix to adjust the mapping function group obtained in step (2), and use the feature mapping group to calculate the similar distance to determine whether the background area belongs to the visible or invisible class by performing a sliding box traversal on the image. (5) The backgrounds classified as invisible are obtained by step (2) and then added to the visible background feature mapping function group; (6) The background image is reconstructed based on the background feature mapping function group. The background is removed by difference between the original image and the background image and optimized by non-local mean filtering to achieve background suppression of infrared image.

2. The infrared image background suppression method based on zero-learning according to claim 1, characterized in that, Step (1) reads in the visible infrared time-series images, corresponding labels, and corresponding spectral data. Using the read data, infrared image features are extracted based on the VGG-16 model with dilated convolution, and label text features are extracted based on the bidirectional encoder model of Transformer, which are used as the main features of the zero-learning model. Temporal features of the time-series images are extracted based on dilated convolution and causal convolution, and spectral features are extracted based on the encoder of the Transformer network, which are used as auxiliary features of the zero-learning model. The VGG-16 model based on dilated convolution is used to extract infrared image features W. ； ； Where K is the filter sequence, a is the dilation rate of the dilated convolution, k is the original kernel size of the network, w is the size of the input image or feature map, S is the stride, and P_boundary is the feature map boundary padding. The image feature extraction network structure consists of three convolutional layers, two multi-scale feature fusion modules based on dilated convolution, and a fully connected layer. The output of each node in the network layer... as follows: ； Where x is the feature input, d is the layer number, and t is the input sequence. For kernel function weights, The correlation function is defined as the number of layers and the dilation rate. The image target description is used as the semantic label corresponding to the background and target features. For the label text, a Transformer-based bidirectional encoder model is used to embed the pre-trained text to obtain the label text features. The algorithm employs label embedding, segment embedding, and positional embedding. The input labels are in the form of (batch_size, seq_len). Label embedding maps the input to a vector of dimension d_model, where batch_size is the number of samples in each batch and seq_len is the maximum length of the label text, thus obtaining the label embedding features. Segment embedding maps the information of the sentences in the input into a vector of dimension d_model, so after this layer, the form becomes ( batch_size , seq_len , d_ model However, since natural language consists entirely of target and background feature labels and is not segmented into sentences, the segment embedding will not be modified to [1,1,...1], resulting in segment embedding features. Position embedding maps the positional information of the input background and the target word to a vector of d_model, thus obtaining the position embedding features. This leads to the high-dimensional vector. : ； The Transformer-based bidirectional encoder model consists of L layers of Transformers, ultimately yielding... The system utilizes dilated convolution and causal convolution to operate on the input sequence of images, extracting time-series features corresponding to the image features, and outputting the results at each time step. All depend only on the num times before that moment. Value: ； in, For causal convolutional networks, For the first num features of the current value, after each layer of dilated causal convolution, the output is normalized and ReLU nonlinear calculation is performed; the encoder of the Transformer network is selected to extract the spectral features of the spatial target, and the final sequence input into the network is represented as: ； in, Represents the embedding vector of the text input sequence, where For sequence index, For time indexing, This represents the projection layer that maps word vectors to D-dimensional vectors. P_embedding represents the word relative position embedding. The last layer of the position embedding network outputs the spectral feature embedding vector. Represented as: ； After spectral features are extracted using a Transformer, a real-time updated soft attention mechanism is employed. The similarity function of this spectral attention mechanism... and characteristics Represented as: ； ； ； in, For key vectors, For query vector, For value vectors, The weight matrix represents the weights of the first layer of the Transformer network. The self-attention mechanism in the network is replaced by a simulated spectral spectra. The encoder output of the Transformer network is as follows: ； ； in, For the output characteristics of the encoder, For the final spectral characteristics, For spectral characteristics, as well as As weight, as well as This is the bias value.

3. The infrared image background suppression method based on zero-learning according to claim 1, characterized in that, Step (2) constructs a zero-order learning model. Based on the main features and auxiliary features extracted in step (1), the zero-order learning model based on Transformer is trained, and the target and background feature mapping functions are obtained to form a function group. After extracting the features required to represent the data, a zero-order learning model is constructed. By constructing a mapping relationship between the image features and auxiliary features of the target and background and the label text features of the target, the visible domain data is defined as... ,in, Image features for the visible class, For the complete set of features of the visible class image, For visible label text features, The complete set of visible class label text features, For image temporal features, For the complete set of temporal features of visible images, For spectral characteristics, The visible class is the complete set of spectral features, while the invisible class is defined as... ,in, Image features for invisible classes, For the complete set of features of the invisible class image, For invisible text features, For the complete set of invisible class label text features, Temporal features of invisible images For the complete set of temporal features of invisible images, It is an invisible spectral feature. The complete set of invisible spectral features is used; a preliminary mapping function is formed by nonlinear mapping of image features, auxiliary features, and label text features using a multilayer perceptron. ； in, For image feature mapping function, As the auxiliary feature mapping function, p and q are the weights, p+q=1; then a Transformer network is used to train the mapping model, and the loss function is established based on the cross-entropy loss function. By introducing a similarity metric, the zero-step learning network learns the visible class feature mapping function by optimizing the similarity between features, thus improving the similarity between images and text, and between auxiliary features and image features. and loss function The calculation formula is as follows, where These are the weights, and : ； ； Finally, the final mapping function is obtained by training the mapping function through a Transformer network: ； in, To ultimately train and obtain the image feature mapping function, This is used to obtain the auxiliary feature mapping function for the final training.

4. The infrared image background suppression method based on zero-learning according to claim 1, characterized in that, Step (3) involves reading in the invisible infrared image, constructing a temporal-spatial local contrast, and determining whether the regions traversed by the invisible data sliding frame are background regions: The invisible infrared image is read in, and the outer frame utilizes the changes between frames and the comparison between the four corners of the outer frame. The inner frame calculates the difference between the center point of each sliding frame region and the average value of the sliding frame, and multiplies this difference with the difference between the center region and the opposite direction to obtain the contrast value. The formula is as follows: ； in, For slider direction index, Center of the image slider Pixel values, incorporating temporal features, are compared by contrasting the previous and current frames, as well as the next and current frames. The preceding and following frames are selected as the temporal range, resulting in updated comparison values. The formula is as follows: ； Where t is the current frame, To update the coefficients, the updated overall judgment index is... as follows: ； in, ,when Less than If there is no change between the preceding and following frames, it is judged as the background; set the grayscale ratio between the two frames in the current frame and the change in grayscale values ​​with the preceding and following frames, and set the fluctuation of the product of the changes between the preceding and following frames to be greater than a threshold. At that time, it is considered that the background has changed significantly: ； ； ； in, Here, represents the inner and outer lengths of the double-layered frame; aver1 is the average value of the inner frame; aver2 is the average value of the outer frame; and gray represents the pixel grayscale value within the area. These are the values ​​for the four corners of the outer frame. The ratio of the two frames at frame t is given, and dif is the product of the grayscale value ratio changes between the current frame and the previous and next frames. Furthermore, three boundary points are set on each side of the outer frame. When the background boundary point is located on any one of the four edges AB, BC, CD, or DA, the difference between these boundary points is used to determine whether the boundary will pass through these four edges. ； Where diff represents the difference between the two points. and These are the values ​​of a certain side point. , Introducing target motion information, this involves combining the target's motion distance, direction and its weights, velocity, and acceleration features. The average distance between the target point in each of the previous five frames is used as the target motion distance between the current frame and the previous frame. Simultaneously, the target movement changes from the previous two frames are also considered. The movement changes of the target in the current frame are used to predict the target's motion state and added to a temporary storage group as a reference for subsequent motion. ； ； Where dis is the average distance of the previous five frames, dirx is the average distance in the X direction of the previous five frames (X,Y), diry is the average distance in the Y direction of the previous five frames (X,Y), and dir_mom is the position state of the current frame. From this, the predicted position of the target in the current frame is obtained. The target's velocity spe and acceleration acc are calculated using the relative distance between targets in different frames and the frame rate. ； ； in, This refers to the positions of the current frame and the previous frame. fre is the frame rate.

5. The infrared image background suppression method based on zero-learning according to claim 1, characterized in that, Step (4) adjusts the mapping function group obtained in step (2) based on whether the data read in step (3) has corresponding auxiliary features. By performing a sliding box traversal on the image, the similarity distance is calculated using the feature mapping group to determine whether the background region belongs to the visible or invisible class. In the recognition stage, since the invisible class and the visible class share the semantic space, the obtained mapping function group is transferred to the invisible domain. At the same time, when facing the lack of auxiliary features in the recognition of invisible class images, the auxiliary features of each mapping function group are adjusted to form a new mapping function by adjusting the feature mapping function group obtained by this method to form an identity matrix to solve the problem of lack of auxiliary features. Then, the label text features obtained by the new image through the mapping function are compared with the label text features of the trained feature mapping function group. The minimum distance between two points is used to determine whether an image belongs to a known domain, thereby enabling the recognition of unknown image backgrounds. The formula for distance D is as follows: ； in, , For visible class label text features, For invisible class label text features, These are the predicted label text features. These are the actual label text features. It is a similarity function, calculated using the cosine function. When D is greater than the threshold... When the time is visible, it is less than the threshold. When the invisible class is identified, the identified invisible class and auxiliary features are added to the training network to obtain a new background feature mapping function and add it to the feature mapping function group.

6. The infrared image background suppression method based on zero-learning according to claim 1, characterized in that, Step (5) distinguishes the background as invisible class and obtains a new background feature mapping function through step (2), and then adds it to the visible class feature mapping function group: set the network structure to a recurrent feedback structure, and add the background features of the distinguished invisible class into a new mapping function to the feature mapping function group.

7. The infrared image background suppression method based on zero-learning according to claim 1, characterized in that, Step (6) reconstructs the background image based on the background feature mapping function set, removes the background by difference between the original image and the background image, and optimizes it using non-local mean filtering to achieve infrared image background suppression: feature fusion between infrared image features and label text features is achieved by using element-wise multiplication to generate features. Where h is the height of the feature map, w is the width of the feature map, and c is the number of channels, a lightweight U-Net generator network is introduced to decode it into a background image. z is then progressively upsampled through multiple transposed convolutions to finally output an infrared background image that matches the target size. , Generate a network for U-Net. The parameters of the U-Net network are generated by combining pixel-level L1 loss and structural similarity loss to obtain the loss function. Optimize the similarity of the model at the pixel level and the similarity of structural information: ； in, For weight hyperparameters, It is a structural similarity index. For the generated background image, To obtain a more refined differential image, the original infrared image is subtracted from the infrared background image to obtain a more realistic background image. For each pixel in the difference image Calculate its relationship with other pixels in the neighborhood. Similarity weight : ； in, In pixels The image patch centered on For each pixel of the difference image, the filtering parameters are... Perform filtering operation: ； in, For the search window, cover pixels The neighborhood of the target image is then used to obtain a clean target image. This achieves background suppression.

Images