A cross-modal generation method of infrared images based on feature mapping

Abstract

The present invention provides a method for cross-modal generation of infrared images based on feature mapping, comprising the following steps: (1) reading a visible light image; (2) simulating the radiation transmission mechanism of the read visible light image to obtain a physically simulated infrared image; (3) decomposing the read visible light image into high-frequency and low-frequency information components by using Laplace pyramid decomposition; (4) extracting features from the information components obtained in step (3) based on multi-level coding to obtain a frequency domain feature map; (5) performing feature mapping on the frequency domain feature map obtained in step (4) to obtain a high-frequency and low-frequency global feature map; (6) introducing cross-attention to fuse the global feature map obtained in step (5) to generate a feature fusion image; (7) extracting and comparing feature blocks from the feature fusion image of step (6) and the visible light image of step (1); and (8) training and reasoning an adversarial model based on the comparison result of step (7) and the simulated infrared image of step (2) to generate an image that conforms to infrared characteristics.

Description

Technical Field

[0001] The invention relates to a cross-modal generation method of infrared images based on feature mapping, belongs to the field of visible light-infrared cross-modal data intelligent processing methods and application technologies, and is suitable for cross-modal data intelligent generation. Background Art

[0002] In the field of computer vision, visible light and infrared data are two important cross-modal image data. Visible light images contain rich texture structures and geometric features, and infrared images reflect the temperature distribution of the target and the background. Due to the limitations of the shooting conditions of infrared imagers or the lack of a reliable and accurate infrared simulation system, the amount of infrared data collected in many downstream application scenarios is insufficient to support model learning and training, and the downstream models cannot perform well. Therefore, these scenarios face the huge challenge of a lack of infrared image data. Current research on solving the problem of lack of infrared data mainly focuses on generating corresponding infrared images from visible light images. How to learn the mapping association between cross-modal data during training and generate high-quality infrared images from visible light images has become an important research direction that needs to be solved urgently.

[0003] At present, there are different solutions for cross-modal generation from visible light to infrared images. One idea is to use physical model methods to artificially simulate hot areas, but this method is inefficient and inaccurate. Another popular idea is to use intelligent algorithms to embed the input visible light image into the latent feature space, and then reconstruct the corresponding infrared image from the nonlinear conversion relationship. For example, the visible light domain is converted to infrared style using generative adversarial networks, unsupervised learning, self-supervised learning, etc. These methods have alleviated the problem of lack of infrared image data to a certain extent. Although the above conversion methods can obtain outputs similar to real infrared images, they focus more on the overall image and ignore the conversion of infrared thermal features. Some studies based on contrastive learning have attempted to overcome these shortcomings and use feature blocks to compare and supplement details, but the semantic information between the visible light domain and the infrared domain is quite different, and these methods still lack additional constraints. It is worth noting that the visible light and infrared data collected in actual scenes are often non-aligned, and there are certain differences in perspective or resolution. These limitations increase the difficulty of cross-modal data conversion. In addition, many popular image translation methods currently focus more on the diversity of translation styles. If they are directly applied to visible light-infrared cross-modal conversion tasks, it is also difficult to obtain output images that conform to the infrared feature laws.

[0004] In summary, the existing methods for generating images from visible light to infrared have the following three difficulties that have not yet been resolved: (1) It is difficult to generate effectively under non-aligned training data; (2) The generated infrared data only has the style of infrared images and is difficult to conform to the physical laws of the infrared modality; (3) The generation method is difficult to have data generalization capabilities. Summary of the invention

[0005] The purpose of the present invention is to address the problem of insufficient infrared data collected for target detection and recognition tasks and to propose a method for cross-modal intelligent generation of infrared images based on visible light images.

[0006] The technical solution of the present invention is to design an intelligent network model for feature mapping and image generation, which consists of a generator and a discriminator. Its goal is to maximize the mutual information of feature block mapping between input and output images, and to perform adversarial training with infrared images generated based on radiation transfer mechanism simulation to obtain a final generated image that meets the infrared thermal characteristics.

[0007] The present invention is a cross-modal generation method of infrared images based on feature mapping, and the steps are as follows:

[0008] (1) Read in visible light images;

[0009] (2) simulating the radiation transmission mechanism of the visible light image read in step (1) to obtain a physically simulated infrared image;

[0010] (3) Decomposing the visible light image read in step (1) into high-frequency and low-frequency information components using Laplace pyramid decomposition;

[0011] (4) Based on multi-level coding, feature extraction is performed on the high and low frequency information components obtained in step (3) to obtain a frequency domain feature map;

[0012] (5) Introduce sliding window self-attention to perform feature mapping on the frequency domain feature map obtained in step (4) to obtain high- and low-frequency global feature maps;

[0013] (6) Introduce cross attention to fuse the high-frequency and low-frequency global feature maps obtained in step (5) to generate a feature fused image;

[0014] (7) performing feature block extraction and comparison calculation on the feature fusion image of step (6) and the visible light image of step (1);

[0015] (8) Based on the comparative calculation results of step (7) and the simulated infrared image of step (2), adversarial model training and reasoning are performed to generate an image that conforms to infrared characteristics.

[0016] Wherein, the step (1) reads in the visible light image: reads in the visible light image data collected from the real scene.

[0017] In which, the step (2) simulates the radiation transmission mechanism of the visible light image read in the step (1) to obtain a physically simulated infrared image: first, different types of objects in the image are identified and segmented to obtain a semantic segmentation result; second, transmission modeling and simulation are performed on the target, background, atmosphere and sensor in infrared radiation transmission to obtain simulation images with different grayscale differences; then, radiation characteristic models of different targets and their surrounding environments are constructed for radiation characteristic analysis and simulation; finally, an atmospheric simulation model is constructed to simulate the influence of the atmosphere on infrared radiation, including atmospheric absorption and scattering, and then the infrared radiation amount is calculated to finally obtain a physically simulated infrared image.

[0018] The step (3) uses Laplacian pyramid decomposition to decompose the visible light image read in step (1) into high-frequency and low-frequency information components: the original visible light image read in step (1) is used as the bottom layer of the Gaussian pyramid, and a convolution filtering operation is performed on the image using a Gaussian kernel to smooth the image and remove noise; the filtered image is convolutionally downsampled with a sampling interval of 2 to halve the image resolution, and the Gaussian filtering and downsampling operations are repeated twice; after obtaining the Gaussian pyramid, a Laplacian pyramid is constructed by calculating the difference between adjacent levels; first, starting from the top layer of the Gaussian pyramid, the resolution of each layer of the image is upsampled, and the upsampled image is Gaussian filtered to match its size and resolution with the next layer of the Gaussian pyramid image; then, the obtained image is subtracted from the next layer of the Gaussian pyramid image to obtain the current Laplacian pyramid difference image layer, which represents the high-frequency feature map of the visible light image at the current scale, and is recorded as The top layer of the Laplacian pyramid is used as the low-frequency feature map of the visible light image, denoted as .

[0019] Wherein, the step (4) extracts features of the high- and low-frequency information components obtained in step (3) based on multi-level coding to obtain a frequency domain feature map: , where H, W, and C represent the vertical width, horizontal length, and number of channels of the image, respectively. First, a continuous encoder structure is used to expand its channels to obtain a low-frequency feature map ,in Indicates the encoder level, the value is The encoder consists of a convolutional layer, a normalization layer, and an activation function layer for feature extraction; then position encoding is added to retain spatial information and express the positional relationship of features; for the high-frequency information components of visible light images , this component is downsampled using an encoder with the same structure to obtain a high-frequency feature map.

[0020] Among them, the step (5) introduces sliding window self-attention, performs feature mapping on the frequency domain feature map obtained in step (4), and obtains high- and low-frequency global feature maps: after the low-frequency feature map position encoding operation and after the high-frequency feature map, a continuous self-attention calculation module is introduced to perform long-distance feature extraction and mapping; the continuous self-attention module calculation process for the low-frequency feature branch and the high-frequency feature branch can be expressed as:

[0021]

[0022] In the formula, and Respectively represent conventional and sliding window multi-head self-attention calculations, represents the normalization layer calculation, represents the multi-layer perceptron computation, Indicates Level self-attention module structure, Represents the structure of the previous level self-attention module, and is the process quantity of feature calculation, representing the The output of the conventional window multi-head self-attention calculation of the module structure and the output of the multi-layer perceptron calculation, Represents the output of the previous level self-attention module structure, and Respectively represent The output of the sliding window multi-head self-attention calculation of a module structure and the output of its multi-layer perceptron calculation; after the above feature mapping calculation, high-frequency and low-frequency global feature maps are obtained respectively.

[0023] Among them, the step (6) introduces cross attention to fuse the high-frequency and low-frequency global feature maps obtained in step (5) to generate a feature fusion image: first, a continuous decoder structure is used to decode the high-frequency and low-frequency global feature maps obtained in step (5) step by step, and the decoder structure consists of a convolution layer, a normalization layer and an activation function layer; then, a sliding window-based cross attention is introduced to fuse the high-frequency and low-frequency global feature maps obtained in step (5) with a resolution of The high- and low-frequency global feature maps are fused and calculated, so that the key matrix and value matrix in the window come from the low-frequency branch, and the query matrix comes from the high-frequency branch, realizing the fusion and refinement of different feature sequences; among them, the fusion calculation process of high- and low-frequency components in the cross attention based on the sliding window is:

[0024]

[0025] In the formula, and They represent conventional and sliding window multi-head cross self-attention calculations respectively, represents the normalization layer, represents a multi-layer perceptron, Indicates Level self-attention module structure, Represents the structure of the previous level self-attention module, The subscript indicates that the process quantity belongs to the low-frequency feature branch. The subscript indicates that the process quantity belongs to the high-frequency feature branch. and Indicates the first The output of the conventional window multi-head cross self-attention calculation of the module structure and the output of the multi-layer perceptron calculation, Represents the output of the previous level self-attention module structure, and They represent the first The sliding window multi-head self-attention calculation output of a module structure and the output of the multi-layer perceptron calculation; finally, the output of the cross-attention is convolved and up-sampled to generate a feature fusion image.

[0026] In which, the step (7) extracts feature blocks and performs contrast calculation on the feature fusion image of step (6) and the visible light image of step (1): introduces contrast learning, uses a feature block extraction module to extract feature blocks from the real visible light image input in step (1) and the fusion image output in step (6), and obtains a stack of two feature blocks. as well as ,in represents the visible light mode, represents any visible light image in the visible light modality dataset, Indicates infrared mode, represents the number of feature layers in the stack, represents the number of feature encoding layers, For the generator, and Indicates the two modes The feature map after layer feature encoding, and Indicates the two modes The layer features are the features calculated by the multi-layer perceptron; the feature block extraction module is composed of a continuous convolutional coding structure and a multi-layer perceptron network; a query sample is selected each time in the feature block stack of the fused image, and the feature block at the corresponding position in the input real visible light image is designated as a positive sample, while the feature block at the non-corresponding position is recorded as a negative sample; the query sample, positive sample, and negative sample are mapped to dimensional space vector; on this basis, the feature block matching problem is transformed into a classification problem, and the probability that the positive sample exceeds the negative sample is calculated; the multi-layer contrast loss based on feature block matching is used as the mapping contrast loss from the visible light modality to the infrared modality , as the feature mapping constraint of the network to evaluate the mapping effect of the generator:

[0027]

[0028] In the formula is the mapping function, represents the number of feature layers in the stack, is the maximum number of feature layers, Represents the number of spatial locations of features at each layer, is the maximum number of spatial locations, represents the infrared modal feature stack, and Represents the positive sample feature stack and negative sample feature stack of the visible light modality, represents the visible light mode, represents any visible light image in the visible light modality dataset, Indicates infrared mode, and It represents the result of all feature layers of the two modal images after being calculated by the multi-layer perceptron. is the generator; after the feature block extraction and comparison calculation steps, it serves as the feature mapping constraint of the network and performs feature optimization and update on the feature fusion image obtained in step (6).

[0029] In which, the step (8) performs adversarial model training and reasoning based on the comparative calculation results of step (7) and the simulated infrared image of step (2) to generate an image that meets the infrared characteristics: based on the comparative calculation results of step (7), the feature fusion image obtained in step (6) is optimized and updated in detail, and an adversarial network loss calculation method is introduced. The similarity between the updated feature fusion image and the physical simulated infrared image outputted in step (2) is optimized through the discriminator constraint step as the network's adversarial loss; the cross-modal infrared image intelligent generation model based on frequency domain feature mapping and contrastive learning is trained, and the inference output obtains a generated image that meets the infrared characteristics.

[0030] Compared with the prior art, the advantages of the present invention are: (1) the model of the present method is an intelligent generation method, which can achieve the effect of converting visible light domain to infrared domain image without aligned infrared training data; (2) a conversion method based on feature mapping is proposed, which can make the infrared image generated from the visible light image more consistent with the physical laws of the infrared modality, for example, hot objects are brighter, while cold objects are darker, avoiding the cross-modal distortion problem caused by the general image conversion method; (3) the low-frequency feature mapping captures the structure and color features of the image, etc. These structure and color features may show local similarities between different images, so that the model can be more effectively extended to new data sets and has a certain data generalization ability. DETAILED DESCRIPTION

[0031] In order to better illustrate the infrared image feature mapping and intelligent generation method involved in the present invention, a high-resolution continuous zoom medium-wave cooled infrared thermal imager and a high-definition visible light camera sensor are used to fly on a drone platform to collect real visible light and infrared data. An aerial photography data set was constructed for the docked ships and aircraft stationed in the Qingdao Naval Museum, and 1,103 pairs of visible light and non-aligned infrared images were collected. 939 pairs of images were randomly selected as training sets and 164 pairs of images were randomly selected as test sets. The image resolution size was 640×512 for model training and reasoning. The specific implementation steps are as follows:

[0032] (1) Reading in visible light images: Read in visible light image data collected from real scenes. The training set size is 939 visible light images.

[0033] (2) The radiation transmission mechanism of the visible light image read in step (1) is simulated to obtain a physically simulated infrared image: first, the different types of objects in the image are identified and segmented to obtain semantic segmentation results; second, the transmission modeling and simulation of the target, background, atmosphere and sensor in infrared radiation transmission are performed to obtain simulation images with different grayscale differences; then, radiation characteristic models of different targets and their surrounding environments are constructed for radiation characteristic analysis and simulation; finally, an atmospheric simulation model is constructed to simulate the influence of the atmosphere on infrared radiation, including atmospheric absorption and scattering, and then the infrared radiation amount is calculated to finally obtain a physically simulated infrared image.

[0034] (3) Decompose the visible light image read in step (1) into high-frequency and low-frequency information components using Laplacian pyramid decomposition: Set the number of Laplacian pyramid layers to 2, use the original visible light image read in step (1) as the bottom layer of the Gaussian pyramid, and use the Gaussian kernel to perform a convolution filtering operation on the image to smooth the image and remove noise; perform convolution downsampling on the filtered image with a sampling interval of 2 to halve the image resolution, and repeat the Gaussian filtering and downsampling operations twice; after obtaining the Gaussian pyramid, construct a Laplacian pyramid by calculating the difference between adjacent layers; first, starting from the top layer of the Gaussian pyramid, upsample the resolution of each layer of the image, and perform Gaussian filtering on the upsampled image to match its size and resolution with the next layer of the Gaussian pyramid image; then, subtract the obtained image from the next layer of the Gaussian pyramid image to obtain the current Laplacian pyramid difference image layer, which represents the high-frequency feature map of the visible light image at the current scale, denoted as The top layer of the Laplacian pyramid is used as the low-frequency feature map of the visible light image, denoted as .

[0035] (4) Based on multi-level coding, feature extraction is performed on the high- and low-frequency information components obtained in step (3) to obtain a frequency domain feature map: the number of encoder structures for high-frequency components is set to 1, and the number of encoder structures for low-frequency components is set to 3; for the low-frequency information components of the visible light image, , where H, W, and C represent the vertical width, horizontal length, and number of channels of the image, respectively. First, three consecutive encoder structures are used to expand the channels to obtain the low-frequency feature map ,in Indicates the encoder level, the value is The encoder consists of a convolutional layer, a normalization layer, and an activation function layer for feature extraction; then position encoding is added to retain spatial information and express the positional relationship of features; for the high-frequency information components of visible light images , an encoder with the same structure is used to downsample this component to obtain a high-frequency feature map.

[0036] (5) Introduce sliding window self-attention, perform feature mapping on the frequency domain feature map obtained in step (4), and obtain high- and low-frequency global feature maps: after the position encoding operation of the low-frequency feature map and after the high-frequency feature map, introduce six consecutive sliding window-based self-attention calculation modules to perform long-distance feature extraction and mapping; the continuous self-attention module calculation process for the low-frequency feature branch and the high-frequency feature branch can be expressed as:

[0037]

[0038] In the formula ,and Respectively represent conventional and sliding window multi-head self-attention calculations, represents the normalization layer calculation, represents the multi-layer perceptron computation, Indicates Level self-attention module structure, Represents the structure of the previous level self-attention module, and is the process quantity of feature calculation, representing the The output of the conventional window multi-head self-attention calculation of the module structure and the output of the multi-layer perceptron calculation, Represents the output of the previous level self-attention module structure, and Respectively represent The output of the sliding window multi-head self-attention calculation of a module structure and the output of its multi-layer perceptron calculation; after the above feature mapping calculation, high-frequency and low-frequency global feature maps are obtained respectively.

[0039] (6) Introduce cross attention to fuse the high- and low-frequency global feature maps obtained in step (5) to generate a feature fusion image: First, the high-frequency and low-frequency global feature maps obtained in step (5) are decoded step by step, where three consecutive decoders are used for the low-frequency global feature map and one decoder is used for the high-frequency global feature map. The decoder structure consists of a convolutional layer, a normalization layer, and an activation function layer. Then, introduce the cross attention based on the sliding window to fuse the high-frequency global feature map with a resolution of The high- and low-frequency global feature maps are fused and calculated, so that the key matrix and value matrix in the window come from the low-frequency branch, and the query matrix comes from the high-frequency branch, realizing the fusion and refinement of different feature sequences; specifically, the cross attention calculation process is as follows:

[0040]

[0041] In the formula, , , , Respectively represent the query, key, and value matrices contained in the representation window, as well as their dimensions, is the bias matrix, is the activation function operation; in addition, the fusion calculation process of high-frequency and low-frequency components in the sliding window-based cross attention is:

[0042]

[0043] In the formula, and They represent conventional and sliding window multi-head cross self-attention calculations respectively, represents the normalization layer, represents a multi-layer perceptron, Indicates Level self-attention module structure, Represents the structure of the previous level self-attention module, The subscript indicates that the process quantity belongs to the low-frequency feature branch. The subscript indicates that the process quantity belongs to the high-frequency feature branch. and Indicates the first The output of the conventional window multi-head cross self-attention calculation of the module structure and the output of the multi-layer perceptron calculation, Represents the output of the previous level self-attention module structure, and They represent the first The sliding window multi-head self-attention calculation output of a module structure and the output of the multi-layer perceptron calculation; finally, the output of the cross-attention is convolved and up-sampled to generate a feature fusion image.

[0044] (7) Extracting and contrasting the feature blocks of the feature fusion image of step (6) and the visible light image of step (1): Introducing contrast learning, using the feature block extraction module to extract the feature blocks of the real visible light image input in step (1) and the fusion image output in step (6), respectively, to obtain a stack of two feature blocks. as well as ,in represents the visible light mode, represents any visible light image in the visible light modality dataset, Indicates infrared mode, represents the number of feature layers in the stack, represents the number of feature encoding layers, For the generator, and Indicates the two modes The feature map after layer feature encoding, and Indicates the two modes The feature block extraction module consists of three consecutive convolutional coding structures and two multi-layer perceptron networks. A query sample is selected each time in the feature block stack of the feature fusion image, and the feature block at the corresponding position in the input real visible light image is designated as a positive sample, while the feature block at the non-corresponding position is recorded as a negative sample. The query sample, positive sample, and negative sample are mapped to dimensional space vector; on this basis, the feature block matching problem is transformed into a classification problem, and the probability that the positive sample exceeds the negative sample is calculated; the multi-layer contrast loss based on feature block matching is used as the mapping contrast loss from the visible light modality to the infrared modality , as the feature mapping constraint of the network to evaluate the mapping effect of the generator:

[0045]

[0046] in is the mapping function, represents the visible light mode, represents any visible light image in the visible light modality dataset, Indicates infrared mode, represents the number of feature layers in the stack, is the maximum number of feature layers, Represents the number of spatial locations of features at each layer, is the maximum number of spatial locations, and It represents the result of all feature layers of the two modal images after being calculated by the multi-layer perceptron. For the generator; represents the infrared modal feature stack, and The positive sample feature stack and the negative sample feature stack representing the visible light modality, the stack selection and classification probability calculation process are as follows:

[0047] Map queries, positive samples, and negative samples to K-dimensional vectors respectively , , , Represents the maximum number of negative samples; uses L2 normalization to normalize the vector to the unit sphere; establishes a The classification problem in the direction of the positive sample and the probability of calculating the positive sample exceeding the negative sample is mathematically defined as follows:

[0048]

[0049] In the formula, Parameter representing the scaling distance value, Indicates negative samples; after the above feature block extraction and comparison calculation steps, the feature fusion image obtained in step (6) is optimized and updated as the feature mapping constraint of the network.

[0050] (8) Based on the comparison calculation results of step (7) and the simulated infrared image of step (2), the adversarial model training and reasoning are performed to generate an image that conforms to the infrared characteristics: based on the comparison calculation results of step (7), the feature fusion image obtained in step (6) is optimized and updated in detail, and the adversarial network loss calculation method is introduced. The similarity between the updated feature fusion image and the physical simulated infrared image output by step (2) is optimized through the discriminator constraint step as the adversarial loss of the network; the infrared image cross-modal generation method model based on feature mapping is constructed based on the Pytorch deep learning framework, the model training cycle is iterated 200 times, the batch size is set to 1, and the initial learning rate is set to The learning rate decays by 50% every 50 training cycles, the model training process is carried out and the intelligent generation network and parameters are obtained after the training is completed; then 164 Qingdao scene visible light test set images are input, the network model is inferred and tested, and image results that meet the infrared characteristics are generated.

Claims

1. A method for cross-modal generation of infrared images based on feature mapping, characterized in that: The following steps are involved: (1) Read in visible light images; (2) Simulate the radiation transmission mechanism of the input visible light image to obtain a physically simulated infrared image; (3) Decompose the input visible light image into high-frequency and low-frequency information components using Laplace pyramid decomposition; (4) Based on multi-level coding, feature extraction is performed on the information component obtained in step (3) to obtain a frequency domain feature map; (5) Performing feature mapping on the frequency domain feature map obtained in step (4) to obtain high- and low-frequency global feature maps; (6) Introduce cross attention to fuse the global feature map obtained in step (5) to generate a feature fusion image; (7) Extracting and comparing feature blocks of the feature fusion image in step (6) and the visible light image in step (1); (8) Based on the comparison results of step (7) and the simulated infrared image of step (2), adversarial model training and reasoning are performed to generate an image that conforms to infrared characteristics.

2. The method for cross-modal generation of infrared images based on feature mapping according to claim 1, characterized in that: The step (1) reading in the visible light image: reading in the visible light image data collected from the real scene.

3. The method for generating infrared images cross-modally based on feature mapping according to claim 1, characterized in that: The step (2) simulates the radiation transmission mechanism of the read-in visible light image to obtain a simulated infrared image: first, the different types of ground objects in the image are identified and segmented to obtain a semantic segmentation result; second, the transmission modeling simulation is performed on the target, background, atmosphere and sensor in the infrared radiation transmission to obtain a simulation image with different grayscale differences; then, the radiation characteristic model of different targets and their surrounding environment is constructed for radiation characteristic analysis and simulation; finally, an atmospheric simulation model is constructed to simulate the influence of the atmosphere on infrared radiation, including atmospheric absorption and scattering, and then the infrared radiation amount is calculated to finally obtain a physically simulated infrared image.

4. The method for cross-modal generation of infrared images based on feature mapping according to claim 1, characterized in that: The step (3) uses Laplacian pyramid decomposition to decompose the read visible light image into high-frequency and low-frequency information components: the visible light image read in step (1) is used as the bottom layer of the Gaussian pyramid, and a convolution filtering operation is performed on the image using a Gaussian kernel to smooth the image and remove noise; the filtered image is convolutionally downsampled with a sampling interval of 2 to halve the image resolution, and the Gaussian filtering and downsampling operations are repeated twice; after obtaining the Gaussian pyramid, a Laplacian pyramid is constructed by calculating the difference between adjacent levels; first, starting from the top layer of the Gaussian pyramid, the resolution of each layer of the image is upsampled, and the upsampled image is Gaussian filtered to match its size and resolution with the next layer of the Gaussian pyramid image; then, the obtained image is subtracted from the next layer of the Gaussian pyramid image to obtain the current Laplacian pyramid difference image layer, which represents the high-frequency information component of the visible light image at the current scale, and is recorded as The top layer of the Laplacian pyramid is taken as the low-frequency information component of the visible light image, denoted as .

5. The method for cross-modal generation of infrared images based on feature mapping according to claim 1, characterized in that: The step (4) extracts features from the information components obtained in step (3) based on multi-level coding to obtain a frequency domain feature map: , where H, W, and C represent the vertical width, horizontal length, and number of channels of the image, respectively. First, a continuous encoder structure is used to expand its channels to obtain a low-frequency feature map ,in Indicates the encoder level, the value is ,The encoder consists of a convolutional layer, a normalization layer, and an activation function layer to extract features; then position encoding is added to retain spatial information and express the positional relationship of features; High-frequency information components for visible light images , this component is downsampled using an encoder with the same structure to obtain a high-frequency feature map.

6. The method for cross-modal generation of infrared images based on feature mapping according to claim 1, characterized in that: The step (5) performs feature mapping on the frequency domain feature map obtained in step (4) to obtain high- and low-frequency global feature maps: a continuous self-attention calculation module is introduced after the position encoding operation of the low-frequency feature map, and the same continuous self-attention module calculation is directly performed on the high-frequency feature map to perform long-distance feature extraction and mapping; wherein the continuous self-attention module calculation process for the low-frequency feature branch and the high-frequency feature branch is expressed as: ； In the formula, and Respectively represent conventional and sliding window multi-head self-attention calculations, represents the normalization layer calculation, represents the multi-layer perceptron computation, Indicates Level self-attention module structure, Represents the structure of the previous level self-attention module, and is the process quantity of feature calculation, representing the The output of the conventional window multi-head self-attention calculation of the module structure and the output of the multi-layer perceptron calculation, Represents the output of the previous level self-attention module structure, and Respectively represent The output of the sliding window multi-head self-attention calculation of a module structure and the output of its multi-layer perceptron calculation; after the above feature mapping calculation, high-frequency and low-frequency global feature maps are obtained respectively.

7. The method for cross-modal generation of infrared images based on feature mapping according to claim 1, characterized in that: The step (6) introduces cross attention to fuse the global feature map obtained in step (5) to generate a feature fusion image: first, a continuous decoder structure is used to decode the high-frequency and low-frequency global feature maps obtained in step (5) step by step, and the decoder structure consists of a convolution layer, a normalization layer and an activation function layer; then, a sliding window-based cross attention is introduced to fuse the global feature map obtained in step (5) with a resolution of The high- and low-frequency global feature maps are fused and calculated, so that the key matrix and value matrix in the window come from the low-frequency branch, and the query matrix comes from the high-frequency branch, realizing the fusion and refinement of different feature sequences; among them, the fusion calculation process of high- and low-frequency components in the cross attention based on the sliding window is: ； In the formula, and They represent conventional and sliding window multi-head cross self-attention calculations respectively, represents the normalization layer, represents a multi-layer perceptron, Indicates Level self-attention module structure, Represents the structure of the previous level self-attention module, The subscript indicates that the process quantity belongs to the low-frequency feature branch. The subscript indicates that the process quantity belongs to the high-frequency feature branch. and Indicates the first The output of the conventional window multi-head cross self-attention calculation of the module structure and the output of the multi-layer perceptron calculation, Represents the output of the previous level self-attention module structure, and They represent the first The sliding window multi-head self-attention calculation output of a module structure and the output of the multi-layer perceptron calculation; finally, the output of the cross-attention is convolved and up-sampled to generate a feature fusion image.

8. The method for cross-modal generation of infrared images based on feature mapping according to claim 1, characterized in that: The step (7) extracts and compares feature blocks of the feature fusion image of step (6) and the visible light image of step (1): introduces contrast learning, uses a feature block extraction module to extract feature blocks of the real visible light image input in step (1) and the fusion image output in step (6), respectively, and obtains a stack of two feature blocks. as well as ,in represents the visible light mode, represents any visible light image in the visible light modality dataset, Indicates infrared mode, represents the number of feature layers in the stack, represents the number of feature encoding layers, For the generator, and Indicates the two modes The feature map after layer feature encoding, and Indicates the two modes Layer features are features calculated by multi-layer perceptron; The feature block extraction module is composed of a continuous convolutional coding structure and a multi-layer perceptron network. A query sample is selected each time in the feature block stack of the fused image. The feature block at the corresponding position in the input real visible light image is designated as a positive sample, while the feature block at the non-corresponding position is recorded as a negative sample. The query sample, positive sample, and negative sample are mapped to dimensional space vector; on this basis, the feature block matching problem is transformed into a classification problem, and the probability that the positive sample exceeds the negative sample is calculated; the multi-layer contrast loss based on feature block matching is used as the mapping contrast loss from the visible light modality to the infrared modality , as the feature mapping constraint of the network to evaluate the mapping effect of the generator: ； In the formula is the mapping function, represents the number of feature layers in the stack, is the maximum number of feature layers, Represents the number of spatial locations of features at each layer, is the maximum number of spatial locations, represents the infrared modal feature stack, and Represents the positive sample feature stack and negative sample feature stack of the visible light modality, represents the visible light mode, represents any visible light image in the visible light modality dataset, Indicates infrared mode, and It represents the result of all feature layers of the two modal images after being calculated by the multi-layer perceptron. For the generator; After the feature block extraction and comparison calculation steps, the feature fusion image obtained in step (6) is optimized and updated as the feature mapping constraint of the network.

9. The method for cross-modal generation of infrared images based on feature mapping according to claim 1, characterized in that: The step (8) performs adversarial model training and reasoning based on the comparison result of step (7) and the simulated infrared image of step (2) to generate an image that meets the infrared characteristics: based on the comparison calculation result of step (7), the feature fusion image obtained in step (6) is optimized and updated in detail, and an adversarial network loss calculation method is introduced. The similarity between the updated feature fusion image and the physical simulated infrared image outputted in step (2) is optimized through the discriminator constraint step as the network's adversarial loss; the cross-modal infrared image intelligent generation model based on frequency domain feature mapping and contrastive learning is trained, and the generated image that meets the infrared characteristics is obtained by reasoning output.

Images