Visible light-infrared image cross-modal conversion method and device based on attention

Abstract

The invention discloses a visible light-infrared image cross-modal conversion method and device based on attention, and relates to the technical field of cross-modal image processing. The method comprises the following steps: acquiring a training sample data set and a to-be-trained initial image cross-modal conversion generator, wherein the image cross-modal conversion generator is used for converting a visible light image and an infrared image; according to the training sample data set, training the initial image cross-modal conversion generator to obtain a trained image cross-modal conversion generator; obtaining a to-be-processed image, wherein the to-be-processed image is a visible light image or an infrared image; and inputting the to-be-processed image into a trained image cross-modal conversion generator to obtain a modal conversion image corresponding to the to-be-processed image. According to the invention, the infrared image with low cost and high quality can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of cross-modal image processing, and in particular to an attention-based visible light-infrared image cross-modal conversion method and device. Background Art

[0002] Infrared images are grayscale images composed of infrared light intensity. They have extremely high application value in fields such as fault diagnosis, species monitoring, and disaster relief, and have long attracted attention from all parties. For example, in the field of traffic safety, infrared images can not only capture heat source information in low-light environments to generate images, fuse them with visible light images, perform target detection, and assist vehicles in making intelligent decisions, but can also be used to monitor traffic flow, thereby helping managers optimize intersection efficiency. Research on infrared imaging technology generally relies on a large amount of infrared experimental data. The most common method is to set up infrared sensors for actual shooting. Even so, it requires a lot of manpower, equipment, and costs, and the quality and quantity of the data obtained are also very limited. With the development of computer technology, using cutting-edge technology to simulate infrared images and expand data samples to assist in traffic flow control has become a feasible solution.

[0003] Infrared image simulation can be performed using empirical or physics-based models. However, while these methods successfully simulate reliable infrared images, they suffer from long modeling cycles and limited scenarios, making them unsuitable for generating large numbers of infrared images in a short period of time. Furthermore, the high cost and closed-source nature of simulation platforms hinder researchers' access to development tools and further research.

[0004] In recent years, research on infrared-visible cross-modal conversion tasks has grown significantly, and these works can be categorized using the methods described above. Although these algorithms rely on paired data to achieve pixel-level mapping, they face two challenges in practical applications: first, data collection for visible-infrared cross-modal image pairs is difficult, making it difficult to construct large-scale training sets; second, cross-modal data must maintain strict geometric alignment, and spatial misalignment will affect the pixel-level loss function in adversarial generation, thereby affecting the quality of the generated image.

[0005] In the case of less paired data, unsupervised learning is more reasonable and natural for infrared image simulation. Although existing solutions have made contributions to the visible-to-infrared image translation task, the lack of corresponding semantic information in the network still has certain limitations in target simulation and is relatively costly.

[0006] The training of network models is inseparable from rich datasets. However, existing datasets often face three problems in the actual application of traffic flow monitoring: first, some datasets are so old that the original web pages can no longer be opened; second, some of these datasets are driving records and fail to meet the needs of actual applications; third, these datasets often do not have paired visible light images, making it impossible to reasonably evaluate the quality of the generated images, resulting in low sample quality. Summary of the Invention

[0007] To address the technical issues of high cost and low sample quality in existing technologies, the present invention provides an attention-based visible-to-infrared image cross-modal conversion method and device. The technical solution is as follows:

[0008] In one aspect, a method for attention-based visible light-infrared image cross-modal conversion is provided. The method is implemented by an attention-based visible light-infrared image cross-modal conversion device, and the method includes:

[0009] S1. Obtain a training sample dataset and an initial image cross-modal conversion generator to be trained, where the image cross-modal conversion generator is used to convert visible light images and infrared images;

[0010] S2. Train the initial image cross-modal conversion generator based on the training sample dataset to obtain a trained image cross-modal conversion generator;

[0011] S3. Acquire an image to be processed, where the image to be processed is a visible light image or an infrared image;

[0012] S4. Input the image to be processed into a trained image cross-modal conversion generator to obtain a modality conversion image corresponding to the image to be processed.

[0013] On the other hand, a device for attention-based visible light-infrared image cross-modal conversion is provided. The device is applied to the attention-based visible light-infrared image cross-modal conversion method. The device includes:

[0014] A first acquisition submodule is used to acquire a training sample data set and an initial image cross-modal conversion generator to be trained, where the image cross-modal conversion generator is used to convert visible light images and infrared images;

[0015] A training submodule is used to train the initial image cross-modal conversion generator based on the training sample data set to obtain a trained image cross-modal conversion generator;

[0016] An acquisition submodule is used to acquire an image to be processed, where the image to be processed is a visible light image or an infrared image;

[0017] The conversion submodule is used to input the image to be processed into a trained image cross-modal conversion generator to obtain a modality conversion image corresponding to the image to be processed.

[0018] On the other hand, a device for cross-modal conversion of visible light to infrared images based on attention is provided, and the device comprises: a processor; and a memory, wherein computer-readable instructions are stored on the memory, and when the computer-readable instructions are executed by the processor, any one of the above-mentioned methods for cross-modal conversion of visible light to infrared images based on attention is implemented.

[0019] On the other hand, a computer-readable storage medium is provided, wherein the storage medium stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement any one of the above-mentioned attention-based visible light-infrared image cross-modal conversion methods.

[0020] The beneficial effects brought about by the technical solution provided by the embodiment of the present invention include at least:

[0021] The embodiment of the present invention proposes a multi-scale densely linked generative adversarial network based on an attention mechanism, and applies it to the generation of overhead visible light vehicle images to infrared vehicle images. It improves the loss calculation function of the original CycleGAN network and introduces a style loss function through weighted superposition, thereby establishing a connection between the generated image and the unpaired real image. The improved network designed based on the CycleGAN network can learn features between different domains without relying on paired data for training. In order to further extract the features of vehicle targets in the image, the embodiment of the present invention also improves the structure of the generator network, introduces an attention mechanism, performs dense links through the Unet++ structure, and uses void convolution to improve the discriminator PatchGAN network, thereby enhancing the discriminator's perception of multiple scales. In addition, by introducing a style loss function for calculation, the limitation of unpaired pixels that make the two unable to be compared is broken, and low-cost, high-quality infrared images and visible light-infrared sample data can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a flow chart of an attention-based visible light-infrared image cross-modal conversion method provided by an embodiment of the present invention;

[0024] Figure 2 This is a diagram of the overall architecture of a training model provided by an embodiment of the present invention;

[0025] Figure 3 1 is a schematic structural diagram of a visible light-infrared generator provided by an embodiment of the present invention;

[0026] Figure 4 1 is a schematic structural diagram of an encoder provided by an embodiment of the present invention;

[0027] Figure 5 1 is a schematic structural diagram of a decoder provided by an embodiment of the present invention;

[0028] Figure 6 is a structural diagram of an upsampling module provided by an embodiment of the present invention;

[0029] Figure 7 Schematic diagram of the structure of a CBAM module provided by an embodiment of the present invention;

[0030] Figure 8 is a schematic structural diagram of a discriminator provided by an embodiment of the present invention;

[0031] Figure 9 Schematic diagram of the principle of a style loss function provided by an embodiment of the present invention;

[0032] Figure 10 This is a block diagram of an attention-based visible light-infrared image cross-modal conversion device provided by an embodiment of the present invention;

[0033] Figure 11 This is a structural diagram of an attention-based visible light-infrared image cross-modal conversion device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The technical solution of the present invention is described below in conjunction with the accompanying drawings.

[0035] In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as an "exemplary" in the present invention should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of the word "exemplary" is intended to present concepts in a concrete manner. Furthermore, in the embodiments of the present invention, "and / or" can mean both or either of the two.

[0036] In the embodiments of the present invention, the terms "image" and "picture" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same. The terms "of," "corresponding," and "corresponding" may be used interchangeably. It should be noted that, when the distinction between them is not emphasized, their intended meanings are the same.

[0037] In the embodiments of the present invention, sometimes a subscript such as W1 may be written as a non-subscript such as W1. When the difference is not emphasized, the meanings to be expressed are the same.

[0038] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

[0039] The embodiment of the present invention provides a method for cross-modal conversion of visible light to infrared images based on attention. The method can be implemented by a device for cross-modal conversion of visible light to infrared images based on attention. The device for cross-modal conversion of visible light to infrared images based on attention can be a terminal or a server. Figure 1 The flowchart of the attention-based visible light-infrared image cross-modal conversion method shown in FIG. 1 may include the following steps:

[0040] S1. Obtain a training sample dataset and an initial image cross-modal conversion generator to be trained. The image cross-modal conversion generator is used to convert visible light images and infrared images.

[0041] Optionally, the training sample data set may include visible light sample images and infrared sample images that do not correspond to the visible light sample images.

[0042] For example, consider three pairs of images, A, B, and C, taken at different locations. Each pair includes a visible light image and its corresponding infrared image, forming the visible light and infrared sample datasets, respectively. During training, if the visible light sample image is the visible light image in A, the infrared sample image can be the infrared image in B or the infrared image in C.

[0043] S2. Based on the training sample data set, the initial image cross-modal conversion generator is trained to obtain a trained image cross-modal conversion generator.

[0044] Optionally, the initial image cross-modal conversion generator may include an initial visible light-infrared generator and an initial infrared-visible light generator.

[0045] It should be noted that the structures of the visible light-infrared generator and the infrared-visible light generator are completely symmetrical.

[0046] Optionally, in order to improve the training effect of the image generator, an infrared discriminator, a visible light discriminator, an infrared feature extractor, and a visible light feature extractor can be added during training to form a training model, and then the adversarial loss function, the cycle consistency loss function, the identity consistency loss function, and the style loss function are calculated through the training model, and the training model is trained to obtain a trained visible light-infrared generator and a trained infrared-visible light generator; specifically, Figure 2 As shown in the overall architecture diagram of the training model, the model training process of S2 may include the following steps S21-S29:

[0047] S21. Construct a model to be trained, where the model to be trained includes an initial visible light-infrared generator, an infrared discriminator, a visible light discriminator, an initial infrared-visible light generator, an infrared feature extractor, and a visible light feature extractor.

[0048] S22. Input the visible light sample image into the initial visible light-infrared generator to obtain an infrared predicted image corresponding to the visible light sample image. Input the infrared sample image into the initial infrared-visible light generator to obtain a visible light predicted image corresponding to the infrared sample image.

[0049] In a feasible implementation, since the visible light-infrared generator and the infrared-visible light generator are completely symmetrical structures, each generator consists of 8 encoders, 20 decoders and 16 upsampling modules, and are connected in a step-by-step or jump connection manner.

[0050] The generator plays a vital role in image generation. In an embodiment of the present invention, the U-Net++ structure is used as the basis of the generator network. The U-Net network connects the encoder and the decoder through jump connection operations to ensure information transmission between different layers, and uses deconvolution and upsampling in the decoder to gradually extract image features, and finally uses splicing operations to combine them to retain more information, which makes it widely used in GAN networks. The U-Net++ network combines the dense connection characteristics of the DenseNet structure on the basis of the U-Net network, so that each layer can save as much detail information as possible, thereby obtaining a richer representation. However, too dense connections will make the network parameter model too large, which greatly affects the training time of the model. Past studies have shown that appropriate pruning will not affect the model of the entire network. Based on Wang's work on infrared ship target generation, a consistent U-Net++ structure is adopted on the basis of CycleGAN's U-Net256 generator, and the decoder branches corresponding to the even-layer downsampling encoder are pruned to reduce the dependence on the number of parameters and ensure the training effect. The outputs of the four-layer decoder are fused to more effectively capture features from different scales. At the same time, the Unet++ architecture has more upsampling modules and improves the upsampling module, so that the target extraction ability can be improved by calling the module multiple times.

[0051] The following first introduces the structure of the initial visible light-infrared generator:

[0052] Alternatively, as Figure 3 As shown, the initial visible light-infrared generator may include a four-layer restoration structure. Based on the four-layer restoration structure, the specific operation process of S22 may include S221-S225:

[0053] S221 . Input the visible light sample image into the first-layer restoration structure to obtain a first-layer restoration result.

[0054] S222. Obtain a second-layer restoration result based on the intermediate quantity of the first-layer restoration structure, the second-layer restoration structure, and the jump connection operation.

[0055] S223. Obtain a third-layer restoration result according to the intermediate quantity of the first-layer restoration structure, the intermediate quantity of the second-layer restoration structure, the third-layer restoration structure, and the jump connection operation.

[0056] S224. Obtain a fourth-layer restoration result based on the intermediate quantity of the first-layer restoration structure, the intermediate quantity of the second-layer restoration structure, the intermediate quantity of the third-layer restoration structure, the fourth-layer restoration structure, and the jump connection operation.

[0057] S225 , inputting the first layer restoration result, the second layer restoration result, the third layer restoration result and the fourth layer restoration result into an averaging module, and using the output of the averaging module as the infrared prediction image corresponding to the visible light sample image.

[0058] Optionally, the specific structure of each layer of the restoration structure is introduced below:

[0059] The first-layer restoration structure consists of 2 encoders, 2 decoders, and 1 upsampling module;

[0060] The second-layer restoration structure consists of 2 encoders, 4 decoders, and 3 upsampling modules;

[0061] The third-layer restoration structure consists of 2 encoders, 6 decoders, and 5 upsampling modules;

[0062] The fourth-layer restoration structure consists of 2 encoders, 8 decoders, and 7 upsampling modules.

[0063] In one feasible implementation, each layer of the encoder uses convolution to extract the features of the input image. As the number of encodings increases, the encoder output features gradually change from shallow image features (edges, textures, colors, etc.) to high-level features (also known as semantic features, such as the vehicle category in the image, which can help identify specific objects in the target image). The function of the decoder is to gradually restore the spatial information from these gradually abstracted features, and finally form a complete image.

[0064] The role of the skip connection is to connect the feature directly to the decoder of the same layer after it is encoded by the previous encoder. This has the advantage of helping to alleviate the gradient disappearance problem. Because the feature is not further encoded, it retains some of the underlying features. After being fused with the features processed by the encoder and decoder of the next layer, both high-level and shallow features can be retained.

[0065] The four branches can represent different feature fusions. They will output four restored images and focus on different details in the image respectively. For example, the branch with the least encoding times can focus on the shallow details of the image, thereby depicting the approximate edge of the image, while the branch with the deepest encoding times focuses on the semantic features of the image. Averaging the four images pixel by pixel is also a method for multi-scale fusion of shallow and high-level features, which can also improve the robustness of the output.

[0066] like Figure 3 As shown, in order to better introduce the structure and specific operation of the visible light-infrared generator, the first two layers of the four-layer restoration structure are numbered as follows:

[0067] The visible light sample image is input to the first encoder of the first-layer restoration structure, and the output first encoding result is passed to the upsampling of the first layer, the last upsampling of the second layer, the last upsampling of the third layer, and the last upsampling of the fourth layer through a skip connection; the first encoding result is input to the second encoder of the first layer, and the output second encoding result is passed to the second-to-last upsampling of the second layer, the second-to-last upsampling of the third layer, and the second-to-last upsampling of the fourth layer through a skip connection; the second encoding result is input to the first decoder of the first layer to obtain a first decoding result, the first encoding result and the first decoding result are input to the upsampling of the first layer, and the obtained first upsampling result is input to the second decoder of the first layer to obtain a second decoding result, i.e., the restoration result of the first layer;

[0068] The second encoding result is input into the first encoder of the second-layer restoration structure to obtain a third encoding result, and the third encoding result is transmitted to the third-to-last upsampling of the second layer, the third-to-last upsampling of the third layer, and the third-to-last upsampling of the fourth layer through jump connections; the third encoding result is input into the second encoder of the second-layer restoration structure to obtain a fourth encoding result, the fourth encoding result is input into the first decoder of the second-layer restoration structure to obtain a third decoding result, the third decoding result and the third encoding result are input into the first upsampling of the second layer, and the output second upsampling result is input into the second decoder of the second-layer restoration structure to obtain a fourth decoding result, the fourth decoding result and the second encoding result are input into the second upsampling of the second layer to obtain a third upsampling result, the third upsampling result is input into the third decoder of the second-layer restoration structure to obtain a fifth decoding result, the fifth decoding result and the first encoding result are input into the third upsampling of the second layer, and the fourth upsampling result is input into the fourth decoder of the second-layer restoration structure to obtain a sixth decoding result, i.e., the second-layer restoration result.

[0069] The third and fourth restoration structures are analogous. Through skip connections, the upsampling module in each restoration structure combines the low-level features in the encoder (containing more spatial details) with the high-level features in the decoder (containing more semantic information), and finally obtains the restoration result of each layer. The specific execution process is as follows: Figure 3 As shown, no further description is given here.

[0070] Alternatively, as Figure 4 、 Figure 5 as well as Figure 6 , the following introduces the structure of each encoder, each decoder and each upsampling module in the four-layer restoration structure:

[0071] Each of the above encoders includes a first input unit, a convolution unit, a first instance normalization unit, a leaky rectified linear unit, and a first output unit;

[0072] Each decoder includes a second input unit, a deconvolution unit, a second instance normalization unit, a linear rectification function, and a second output unit;

[0073] Each upsampling module includes a third input unit, an upsampling unit, a 1*1 convolution unit, a CBAM module and a third output unit.

[0074] In a feasible implementation, although the jump connection ensures the information transmission between different layers, as the number of sampling increases, the image gradually transforms from shallow texture, shape, and edge features to deep information such as style, semantics, and modality. A simple splicing operation cannot fully capture the rich features in the deep network, especially in the visible light to infrared cross-modal image style generation task. Multiple downsampling causes the gap in deep information between visible light and infrared images to become increasingly larger. In order to better extract the features fused after the jump connection, the embodiment of the present invention introduces an upsampling module CFUS (CBAM-Fuse Upsampling Block) with convolutional block attention features in the jump connection part, and uses the CBAM (Convolutional Block Attention Module) module to improve the performance of the generator network. CBAM is a lightweight and effective attention mechanism that is widely added to various networks to enhance performance in tasks. Generally speaking, CBAM helps the network focus on more important feature areas by connecting the CAM (Channel Attention Module) and SAM (Spatial Attention Module) modules in series. Optionally, such as Figure 7 As shown, the following is a detailed introduction to the CBAM module:

[0075] The CBAM module includes a channel attention CAM unit and a spatial attention SAM unit; the expression of CBAM is defined as follows (1):

[0076]

[0077] in, Represents element-by-element multiplication, M C () indicates the channel attention weighted operation through the CAM unit, M S () indicates the spatial attention weighted operation through the SAM unit, F represents the input feature map, and F' represents the channel feature map after attention weighting.

[0078] In one feasible implementation, a cascaded CBAM-Fuse Upsampling Block (CFUS) is designed. First, information at different levels is connected via channels, and upsampling is performed. Because upsampling can affect the number of channels, hindering subsequent convolution operations, a 1x1 convolution is performed to adjust the number of channels. To better extract the information after feature fusion, the cascaded CBAM module is placed after the upsampling operation, enriching the information obtained.

[0079] (1) The channel attention CAM unit is introduced below:

[0080] The channel attention CAM unit includes: the first global maximum pooling subunit, the first global average pooling subunit, the shared fully connected layer, and the first Sigmoid activation function;

[0081] Furthermore, the specific steps of performing the channel attention weighting operation through the CAM unit can be as follows:

[0082] Input the input feature map of size H*W*C into the first global maximum pooling subunit and the first global average pooling subunit respectively, and obtain the first global maximum eigenvector and the first global average eigenvector respectively.

[0083] The first global maximum eigenvector and the first global average eigenvector are input into the shared fully connected layer to obtain the attention weight vector of each channel. The attention weight vectors of each channel are fused, and the final fused channel attention weight vector is input into the Sigmoid function for activation. The activated vector is multiplied by the input feature map to obtain the attention-weighted channel feature map. The channel feature vector M of the attention-weighted channel feature map is C It is defined as the following formula (2):

[0084]

[0085] Among them, σ represents the sigmoid activation function, f MLP is a function learned by sharing a fully connected layer, W1 represents, W0 represents, F represents the input feature map, F max represents the global maximum eigenvector, F acg represents the global average eigenvector.

[0086] In one feasible implementation, CAM performs global maximum pooling and global average pooling on the input H*W*C feature map, generating two feature maps of shape 1*1*C, representing the global maximum eigenvector and the global average eigenvector, respectively. These maps are then passed through a shared fully connected layer to obtain the attention weights for each channel, thereby adaptively learning which channels are more important for the current visible-to-infrared cross-modal image generation task. The two feature vectors are then fused to obtain the final channel attention weight vector, which is activated using a Sigmoid function. The activated vector is multiplied by the input feature map to obtain the attention-weighted channel feature map, which helps suppress channels irrelevant to the generation task.

[0087] (2) The spatial attention SAM unit is introduced below:

[0088] The spatial attention SAM unit includes: a second global maximum pooling subunit, a second global average pooling subunit, a convolution subunit, and a second Sigmoid activation function;

[0089] Furthermore, the specific steps of performing the spatial attention weighting operation through the SAM unit can be as follows:

[0090] The channel feature map after attention weighting is input into the second global maximum pooling subunit and the second global average pooling subunit respectively. The second global maximum feature vector and the second global average feature vector are spliced along the channel dimension to obtain feature maps with different scale context information. According to the feature maps with different scale context information and the convolution subunit, a spatial attention weight vector is formed. The spatial attention weight vector is input into the sigmoid function for activation to obtain the attention-weighted spatial feature map. The spatial attention vector M of the spatial feature map S It is defined as the following formula (3):

[0091]

[0092] Among them, σ represents the sigmoid activation function, fs is the convolution function, and the specification is 7×7. represents the second global maximum eigenvector, represents the second global average eigenvector.

[0093] In one feasible implementation, SAM performs max pooling and average pooling on the attention-weighted channel feature maps, concatenating them along the channel dimension to produce a feature map with contextual information at different scales. Spatial attention weights are then formed through convolution, followed by a sigmoid activation to emphasize the importance of different locations in the image.

[0094] S23. Input the infrared prediction image and the infrared sample image into the infrared discriminator to obtain the infrared discrimination result, input the visible light prediction image and the visible light sample image into the visible light discriminator to obtain the visible light discrimination result, and calculate the adversarial loss function based on the infrared discrimination result and the visible light discrimination result.

[0095] In one possible implementation, the original discriminator is composed of a Markov discriminator, which maps the input to a k*k matrix X, X ij represents a receptive field in the original image, which represents the discrimination result of this patch. By finding the mean of the matrix X, the discrimination result of this generated image is obtained. This method can make the model pay more attention to image details during training. That is to say, the formula for the real image or the generated image through the discriminator can be expressed as the following formula (4):

[0096]

[0097] Where x represents the input image, P ij represents the probability that the patch in row i and column j cut out from x is judged to be true, and N represents the total number of patches. It is generally believed that N = k 2 ,D represents the discriminator, p(x) represents the probability value output by the discriminator, and each output matrix element corresponds to a 70*70 size patch.

[0098] However, the traditional PatchGAN only downsamples layer by layer through fixed convolution kernels, which may ignore the contextual associations of objects of different scales. To this end, the embodiment of the present invention introduces the ASPP (Atrous Spatial Pyramid Pooling) module to construct an A-Triple-P discriminator (Residual-fused Atrous Spatial Pyramid Pooling PatchGAN, combined with a residual fused pooling pyramid void convolution structure block discriminator). Through a parallel multi-branch design, while retaining the output resolution of PatchGAN unchanged, it captures a wider range of context, and at the same time uses a residual structure to maintain the original discriminator features. The improved discriminator network is as follows: Figure 8 shown.

[0099] Optionally, the infrared discriminator and the visible light discriminator are both A-Triple-P discriminators, each of which includes a PatchGAN module and a dilated spatial pyramid convolution attention ASPP module; wherein:

[0100] The PatchGAN module includes the first convolution block, the second convolution block, the third convolution block, the fourth convolution block and the fifth convolution block;

[0101] The ASPP module includes a first branch, a second branch, a third branch, a fourth branch, a fifth branch, a 1*1 convolutional layer, a linear rectification function, and a dropout layer.

[0102] In one feasible implementation, after passing through the 4-layer convolution of the PatchGAN module, the feature map of PatchGAN will enter the improved void spatial pyramid convolution attention mechanism ASPP module, such as Figure 8 As shown in the figure, it consists of five branches. B1, as the basic branch, only provides the original input feature map features through 1*1 convolution and activation function, while B2, B3, and B4 use different expansion rates to perceive the high-level information of other surrounding patches. B5, as a pooling branch, supplements global information, thereby preventing the discriminator from over-focusing on local information and ignoring overall consistency. Finally, the five branches are spliced and multi-scale fused again through 1*1 convolution to generate a more refined discriminant block, which is beneficial for the discriminant block to determine whether the vehicle target in the patch is accurately generated. In order to enhance the robustness of the model, a Dropout layer (i.e., a discard layer) is also added to avoid overfitting.

[0103] The expression of the ASPP module can be simply summarized as the following formula (5):

[0104]

[0105] Among them, F in represents the input features, F out represents the output features, B1-B5 represent different branches;

[0106] With the definition of the ASPP layer and the residual module, the formula for the modified real image or generated image passing through the discriminator can be modified to the following formula (6):

[0107]

[0108] Among them, D 1-4 represents the convolution module, X ij Represents the Patch block in the i-th row and j-th column of an image.

[0109] Optionally, since the structures of the infrared discriminator and the visible light discriminator are completely consistent, the embodiment of the present invention only describes the execution process of the infrared discriminator in detail. Specifically, Figure 8 As shown, the specific execution process of S23 can be as follows:

[0110] Input the infrared prediction image into the infrared discriminator to obtain the infrared first discrimination result, and input the infrared sample image into the infrared discriminator to obtain the infrared second discrimination result;

[0111] Input the infrared prediction image into the infrared discriminator to obtain the infrared first discrimination result, including:

[0112] The infrared prediction image is input into the first convolution block, the output result of the first convolution block is input into the second convolution block, the output result of the second convolution block is input into the third convolution block, the output result of the third convolution block is input into the fourth convolution block, the output result of the fourth convolution block is input into the first branch, the second branch, the third branch, the fourth branch and the fifth branch respectively, the output result of the first branch, the output result of the second branch, the output result of the third branch, the output result of the fourth branch and the output result of the fifth branch are connected, the connection result is input into the 1*1 convolution layer, the output result of the 1*1 convolution layer is input into the linear rectification function, the output result of the linear rectification function is input into the drop layer, the output result of the drop layer is added to the output result of the fourth convolution block, and the addition result is input into the fifth convolution block to obtain the first infrared discrimination result.

[0113] Optionally, each convolutional block of the PatchGAN module and each branch of the ASPP module are described in detail below:

[0114] The first convolutional block includes a convolutional layer with 3->64 channels, a kernel size of 4, a stride of 2, a padding of 1, and a leaky linear rectification function with a slope of 0.2;

[0115] The second convolutional block includes a convolutional layer with 64->128 channels, a kernel size of 4, a stride of 2, and a padding of 1, a leaky linear rectification function with a slope of 0.2, and an instance normalization layer;

[0116] The third convolutional block includes a convolutional layer with a channel number of 128->256, a kernel size of 4, a stride of 2, and a padding of 1, a leaky linear rectification function with a slope of 0.2, and an instance normalization layer;

[0117] The fourth convolutional block includes a convolutional layer with a channel number of 256->512, a kernel size of 4, a stride of 1, and a padding of 1, a leaky linear rectification function with a slope of 0.2, and an instance normalization layer;

[0118] The fifth convolutional block includes a convolutional layer with a channel number of 512->1, a kernel size of 4, a stride of 1, and a padding of 1;

[0119] The first branch includes a 1*1 convolutional layer and a linear rectification function;

[0120] The second branch includes a dilated convolution with a kernel size of 3 and a dilation rate of 1, as well as a linear rectification function;

[0121] The third branch includes a dilated convolution with a kernel size of 3 and a dilation rate of 3, as well as a linear rectification function;

[0122] The fourth branch includes a dilated convolution with a kernel size of 3 and a dilation rate of 6, as well as a linear rectification function;

[0123] The fifth branch includes average pooling, 1*1 convolution layer, linear rectification function and upsampling unit.

[0124] The following describes the adversarial loss function:

[0125] Assume that A represents the visible light domain, B represents the infrared domain, and G AB Represents visible light-infrared generator, G BA L represents the infrared-visible light generator, x represents the visible light image input to the network, and y represents the infrared image input to the network. adv The purpose of is to make the data generated by the generator close to the real data distribution, which is accomplished by the generator and the discriminator, and can be expressed as the following formula (7):

[0126]

[0127] Among them, G, D, E represent the generator, discriminator, and expectation respectively, x~p data (x) represents the data distribution between the input visible light image and the real visible light data, y~p data (y) represents the data distribution between the input infrared image and the real infrared data, which is encouraged to Judged as true, Judged to be false.

[0128] S24. Input the infrared predicted image into the initial infrared-visible light generator to obtain a visible light reconstructed image. Input the visible light predicted image into the initial visible light-infrared generator to obtain an infrared reconstructed image.

[0129] S25. Calculate a cycle consistency loss function based on the visible light reconstructed image, the visible light sample image, the infrared reconstructed image, and the infrared sample image.

[0130] In one feasible implementation, the cycle consistency loss function L cyc It is designed to ensure that the generated image and the original image are paired with each other, that is, to ensure that G BA (G AB (x))≈x,G AB (G BA (y))≈y,L cyc It can be expressed as the following formula (8):

[0131]

[0132] S26 , input the visible light sample image into an initial infrared-visible light generator to obtain a visible light feature-preserving image, and input the infrared sample image into an initial visible light-infrared generator to obtain an infrared feature-preserving image.

[0133] S27. Calculate the identity consistency loss function based on the visible light feature-preserving image, the visible light sample image, the infrared feature-preserving image, and the infrared sample image.

[0134] In one feasible implementation, the identity consistency loss function L idt The purpose is to keep the image in the domain unchanged after passing through the generator, that is, the infrared image passes through G AB When , the infrared characteristics should still be retained, which can be expressed as the following formula (9):

[0135]

[0136] Among them, ||G BA (x)-x||1,||G AB (y)-y||1 represents the L1 norm between the generated image converted from this domain to other domains and the original image.

[0137] S28. Input the infrared prediction image and the infrared sample image into the infrared feature extractor to obtain infrared features, input the visible light prediction image and the visible light sample image into the visible light feature extractor to obtain visible light features, and calculate the style loss function based on the infrared features and the visible light features.

[0138] Optionally, the infrared feature extractor and the visible light feature extractor are both VGG-19 models.

[0139] In one possible implementation, an interesting phenomenon can be noticed from the flowchart of the cycle consistency loss. In fact, there is no relevant connection path between the real image and the generated image, and L adv Although relevant connections are established, they focus on the high-level features of the image to judge the authenticity. Inspired by Yang’s work on infrared to visible light image conversion, the present invention introduces a style loss L sty , to compensate for the focus on low-level style features and thus generate higher quality images, such as Figure 9 shown.

[0140] The style loss can be calculated using the Gram matrix and has been used in previous generative adversarial networks. It can be expressed as follows (10):

[0141]

[0142] Among them, x1-x nThe vector representing the inner product space, (x i |x j ) represents the direct inner product of two vectors.

[0143] If two images are similar in low-level space, it means that they are similar in color, contour, edge and other information, that is, they have similar style features. Therefore, the embodiment of the present invention extracts low-level features of the image by introducing the VGG-19 model and calculates the style loss. It is worth noting that the VGG-19 model used in the embodiment of the present invention is a pre-trained model, so it does not significantly occupy training resources. The calculation function of style loss can be simply expressed as the following formula (11):

[0144]

[0145] Among them, vgg19(x) represents the feature vector extracted by the pre-trained vgg-19 model of the input image x.

[0146] S29. Calculate the overall loss function based on the adversarial loss function, cycle consistency loss function, identity consistency loss function, and style loss function. Train the model to be trained based on the overall loss function to obtain a trained image cross-modal conversion generator.

[0147] In a feasible implementation, the overall loss function can be expressed as the following formula (12):

[0148]

[0149] Among them, α represents the weight of the cycle consistency loss function, β represents the weight of the identity consistency loss function, and γ represents the weight of the style loss function.

[0150] S3. Acquire an image to be processed, where the image to be processed is a visible light image or an infrared image.

[0151] S4. Input the image to be processed into the trained image cross-modal conversion generator to obtain the modal conversion image corresponding to the image to be processed.

[0152] In a feasible implementation, the trained image cross-modal conversion generator includes a trained visible light-infrared generator and an infrared-visible light generator. The structures and principles of the two can refer to the initial visible light-infrared generator and the initial infrared-visible light generator in the above step S2, and are not described here. If the input image to be processed is a visible light image, the image to be processed is input into the visible light-infrared generator, and the generated infrared image is the modal conversion image corresponding to the image to be processed. If the input image to be processed is an infrared image, the image to be processed is input into the infrared-visible light generator, and the generated visible light image is the modal conversion image corresponding to the image to be processed.

[0153] The embodiment of the present invention proposes a multi-scale densely linked generative adversarial network based on an attention mechanism, and applies it to the generation of overhead visible light vehicle images to infrared vehicle images. It improves the loss calculation function of the original CycleGAN network and introduces a style loss function through weighted superposition, thereby establishing a connection between the generated image and the unpaired real image. The improved network designed based on the CycleGAN network can learn features between different domains without relying on paired data for training. In order to further extract the features of vehicle targets in the image, the embodiment of the present invention also improves the structure of the generator network, introduces an attention mechanism, performs dense links through the Unet++ structure, and uses void convolution to improve the discriminator PatchGAN network, thereby enhancing the discriminator's perception of multiple scales. In addition, by introducing a style loss function for calculation, the limitation of unpaired pixels that make the two unable to be compared is broken, and low-cost, high-quality infrared images and visible light-infrared sample data can be obtained.

[0154] Figure 10 This is a block diagram of a device for attention-based visible light-infrared image cross-modal conversion according to an exemplary embodiment. The device is used in an attention-based visible light-infrared image cross-modal conversion method. Figure 10 , the device comprises:

[0155] A first acquisition submodule 1010 is configured to acquire a training sample dataset and an initial image cross-modal conversion generator to be trained, wherein the image cross-modal conversion generator is configured to convert a visible light image into an infrared image;

[0156] A training submodule 1020 is used to train the initial image cross-modal conversion generator based on the training sample data set to obtain a trained image cross-modal conversion generator;

[0157] An acquisition submodule 1030 is configured to acquire an image to be processed, where the image to be processed is a visible light image or an infrared image;

[0158] The conversion submodule 1040 is configured to input the image to be processed into a trained image cross-modal conversion generator to obtain a modality conversion image corresponding to the image to be processed.

[0159] Figure 11 is a structural diagram of an attention-based visible light-infrared image cross-modal conversion device provided by an embodiment of the present invention, such as Figure 11 As shown, the attention-based visible light-infrared image cross-modal conversion device may include the above Figure 10The attention-based visible light-infrared image cross-modal conversion device shown. Optionally, the attention-based visible light-infrared image cross-modal conversion device 1110 may include a first processor 2001.

[0160] Optionally, the attention-based visible light-infrared image cross-modal conversion device 1110 may further include a memory 2002 and a transceiver 2003 .

[0161] The first processor 2001, the memory 2002 and the transceiver 2003 may be connected via a communication bus.

[0162] The following combination Figure 11 The components of the attention-based visible light-infrared image cross-modal conversion device 1110 are described in detail:

[0163] The first processor 2001 is the control center of the attention-based visible light-infrared image cross-modal conversion device 1110, and can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 is one or more central processing units (CPUs), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs) or one or more field programmable gate arrays (FPGAs).

[0164] Optionally, the first processor 2001 can perform various functions of the attention-based visible light-infrared image cross-modal conversion device 1110 by running or executing a software program stored in the memory 2002 and calling data stored in the memory 2002.

[0165] In a specific implementation, as an embodiment, the first processor 2001 may include one or more CPUs, such as Figure 11 CPU0 and CPU1 are shown in FIG.

[0166] In a specific implementation, as an embodiment, the attention-based visible light-infrared image cross-modal conversion device 1110 may also include multiple processors, such as Figure 111 and 2. The first processor 2001 and the second processor 2004 are shown in FIG. Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor herein can refer to one or more devices, circuits, and / or processing cores for processing data (e.g., computer program instructions).

[0167] The memory 2002 is used to store the software program for executing the solution of the present invention, and is controlled by the first processor 2001 for execution. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0168] Alternatively, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, an optical disc storage (including a compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 2002 may be integrated with the first processor 2001 or exist independently and access the first processor 2001 through the interface circuit ( Figure 11 (not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0169] The transceiver 2003 is used to communicate with a network device or a terminal device.

[0170] Optionally, the transceiver 2003 may include a receiver and a transmitter ( Figure 11 (not shown separately in the figure). The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0171] Optionally, the transceiver 2003 may be integrated with the first processor 2001 or may exist independently and be transmitted via the interface circuit ( Figure 11(not shown) is coupled to the first processor 2001, which is not specifically limited in this embodiment of the present invention.

[0172] It should be noted that Figure 11 The structure of the attention-based visible light-infrared image cross-modal conversion device 1110 shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0173] In addition, the technical effects of the attention-based visible light-infrared image cross-modal conversion device 1110 can refer to the technical effects of the attention-based visible light-infrared image cross-modal conversion method described in the above method embodiment, and will not be repeated here.

[0174] It should be understood that the first processor 2001 in the embodiment of the present invention may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0175] It should also be understood that the memory in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0176] The above embodiments can be implemented in whole or in part through software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired method (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains a collection of one or more available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, or magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0177] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0178] In this disclosure, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, "at least one of a, b, or c" can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0179] It should be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0180] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0181] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0182] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of the device or unit, which can be electrical, mechanical or other forms.

[0183] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0184] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0185] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0186] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for cross-modal conversion of visible light to infrared images based on attention, characterized in that: The method comprises: S1. Obtain a training sample dataset and an initial image cross-modal conversion generator to be trained, where the image cross-modal conversion generator is used to convert visible light images and infrared images; S2. Train the initial image cross-modal conversion generator based on the training sample dataset to obtain a trained image cross-modal conversion generator; S3. Acquire an image to be processed, where the image to be processed is a visible light image or an infrared image; S4. Input the image to be processed into a trained image cross-modal conversion generator to obtain a modality conversion image corresponding to the image to be processed.

2. The attention-based visible-to-infrared image cross-modal conversion method according to claim 1, characterized in that: The training sample data set includes visible light sample images and infrared sample images that do not correspond to the visible light sample images; the initial image cross-modal conversion generator includes an initial visible light-infrared generator and an initial infrared-visible light generator; The step S2 of training the initial image cross-modal conversion generator based on the training sample data set to obtain a trained image cross-modal conversion generator includes: S21, constructing a model to be trained, wherein the model to be trained includes an initial visible light-infrared generator, an infrared discriminator, a visible light discriminator, an initial infrared-visible light generator, an infrared feature extractor, and a visible light feature extractor; S22. Input the visible light sample image into an initial visible light-infrared generator to obtain an infrared predicted image corresponding to the visible light sample image; input the infrared sample image into an initial infrared-visible light generator to obtain a visible light predicted image corresponding to the infrared sample image; S23, inputting the infrared prediction image and the infrared sample image into the infrared discriminator to obtain an infrared discrimination result, inputting the visible light prediction image and the visible light sample image into the visible light discriminator to obtain a visible light discrimination result, and calculating the adversarial loss function based on the infrared discrimination result and the visible light discrimination result; S24, inputting the infrared prediction image into an initial infrared-visible light generator to obtain a visible light reconstructed image, and inputting the visible light prediction image into an initial visible light-infrared generator to obtain an infrared reconstructed image; S25. Calculate a cycle consistency loss function based on the visible light reconstructed image, the visible light sample image, the infrared reconstructed image, and the infrared sample image; S26, inputting the visible light sample image into an initial infrared-visible light generator to obtain a visible light feature-preserving image, and inputting the infrared sample image into an initial visible light-infrared generator to obtain an infrared feature-preserving image; S27. Calculate an identity consistency loss function based on the visible light feature-preserving image, the visible light sample image, the infrared feature-preserving image, and the infrared sample image; S28. Input the infrared prediction image and the infrared sample image into an infrared feature extractor to obtain infrared features, input the visible light prediction image and the visible light sample image into a visible light feature extractor to obtain visible light features, and calculate a style loss function based on the infrared features and the visible light features. S29. Calculate the overall loss function based on the adversarial loss function, cycle consistency loss function, identity consistency loss function, and style loss function. Train the model to be trained based on the overall loss function to obtain a trained image cross-modal conversion generator.

3. The attention-based visible-to-infrared image cross-modal conversion method according to claim 2, characterized in that: The initial visible light-infrared generator includes a four-layer recovery structure; S22 inputs the visible light sample image into the initial visible light-infrared generator to obtain an infrared prediction image corresponding to the visible light sample image, including: Input the visible light sample image into the first-layer restoration structure to obtain the first-layer restoration result; According to the intermediate amount of the first-layer restoration structure, the second-layer restoration structure and the jump connection operation, the second-layer restoration result is obtained; According to the intermediate amount of the first layer restoration structure, the intermediate amount of the second layer restoration structure, the third layer restoration structure and the jump connection operation, the third layer restoration result is obtained; According to the intermediate amount of the first layer restoration structure, the intermediate amount of the second layer restoration structure, the intermediate amount of the third layer restoration structure, the fourth layer restoration structure and the jump connection operation, the fourth layer restoration result is obtained; The first layer restoration results, the second layer restoration results, the third layer restoration results and the fourth layer restoration results are input into the averaging module, and the output of the averaging module is used as the infrared prediction image corresponding to the visible light sample image.

4. The attention-based visible-to-infrared image cross-modal conversion method according to claim 3, characterized in that: The first-layer restoration structure includes 2 encoders, 2 decoders and 1 upsampling module; The second-layer restoration structure consists of 2 encoders, 4 decoders, and 3 upsampling modules; The third-layer restoration structure consists of 2 encoders, 6 decoders, and 5 upsampling modules; The fourth-layer restoration structure includes 2 encoders, 8 decoders, and 7 upsampling modules.

5. The attention-based visible-to-infrared image cross-modal conversion method according to claim 4, characterized in that: Each encoder includes a first input unit, a convolution unit, a first instance normalization unit, a leaky rectified linear unit, and a first output unit; Each decoder includes a second input unit, a deconvolution unit, a second instance normalization unit, a linear rectification function, and a second output unit; Each upsampling module includes a third input unit, an upsampling unit, a 1*1 convolution unit, a CBAM module and a third output unit.

6. The attention-based visible-to-infrared image cross-modal conversion method according to claim 5, characterized in that: The CBAM module includes a channel attention CAM unit and a spatial attention SAM unit; The expression of CBAM is defined as the following formula (1): in, Represents element-by-element multiplication, M C () indicates the channel attention weighted operation, M S () indicates the spatial attention weighted operation, F represents the input feature map, and F' represents the channel feature map after attention weighting; The channel attention CAM unit includes: a first global maximum pooling subunit, a first global average pooling subunit, a shared fully connected layer, and a first Sigmoid activation function; The channel attention weighted operation includes: Input the input feature map of size H*W*C into the first global maximum pooling subunit and the first global average pooling subunit respectively, and obtain the first global maximum eigenvector and the first global average eigenvector respectively. The first global maximum eigenvector and the first global average eigenvector are input into the shared fully connected layer to obtain the attention weight vector of each channel. The attention weight vectors of each channel are fused, and the final fused channel attention weight vector is input into the Sigmoid function for activation. The activated vector is multiplied by the input feature map to obtain the attention-weighted channel feature map. The channel feature vector M of the attention-weighted channel feature map is C It is defined as the following formula (2): Among them, σ represents the sigmoid activation function, f MLP is a function learned by sharing a fully connected layer, W1 represents, W0 represents, F represents the input feature map, F max represents the global maximum eigenvector, F acg represents the global average eigenvector; The spatial attention SAM unit includes: a second global maximum pooling subunit, a second global average pooling subunit, a convolution subunit and a second Sigmoid activation function; The spatial attention weighted operation includes: The channel feature map after attention weighting is input into the second global maximum pooling subunit and the second global average pooling subunit respectively, and the second global maximum feature vector and the second global average feature vector are spliced along the channel dimension to obtain feature maps with different scale context information. According to the feature maps with different scale context information and the convolution subunit, a spatial attention weight vector is formed, and the spatial attention weight vector is input into the sigmoid function for activation to obtain the attention-weighted spatial feature map. The spatial attention vector M of the spatial feature map is S It is defined as the following formula (3): Among them, σ represents the sigmoid activation function, f s is the convolution function with a size of 7×7, represents the second global maximum eigenvector, represents the second global average eigenvector.

7. The attention-based visible-to-infrared image cross-modal conversion method according to claim 2, characterized in that: The infrared discriminator and the visible light discriminator are both A-Triple-P discriminators, each of which includes a PatchGAN module and a dilated spatial pyramid convolution attention ASPP module; wherein: The PatchGAN module includes the first convolution block, the second convolution block, the third convolution block, the fourth convolution block and the fifth convolution block; The ASPP module includes the first branch, the second branch, the third branch, the fourth branch, the fifth branch, a 1*1 convolution layer, a linear rectification function, and a dropout layer; The step S23 of inputting the infrared prediction image and the infrared sample image into the infrared discriminator to obtain the infrared discrimination result includes: Input the infrared prediction image into the infrared discriminator to obtain the infrared first discrimination result, and input the infrared sample image into the infrared discriminator to obtain the infrared second discrimination result; The step of inputting the infrared prediction image into the infrared discriminator to obtain the first infrared discrimination result includes: The infrared prediction image is input into the first convolution block, the output result of the first convolution block is input into the second convolution block, the output result of the second convolution block is input into the third convolution block, the output result of the third convolution block is input into the fourth convolution block, the output result of the fourth convolution block is input into the first branch, the second branch, the third branch, the fourth branch and the fifth branch respectively, the output result of the first branch, the output result of the second branch, the output result of the third branch, the output result of the fourth branch and the output result of the fifth branch are connected, the connection result is input into the 1*1 convolution layer, the output result of the 1*1 convolution layer is input into the linear rectification function, the output result of the linear rectification function is input into the drop layer, the output result of the drop layer is added to the output result of the fourth convolution block, and the addition result is input into the fifth convolution block to obtain the first infrared discrimination result.

8. The attention-based visible-to-infrared image cross-modal conversion method according to claim 7, characterized in that: The first convolution block includes a convolutional layer with a channel number of 3->64, a kernel size of 4, a stride of 2, and a padding of 1, and a leaky linear rectification function with a slope of 0.2; The second convolutional block includes a convolutional layer with a channel number of 64->128, a kernel size of 4, a stride of 2, and a padding of 1, a leaky linear rectification function with a slope of 0.2, and an instance normalization layer; The third convolutional block includes a convolutional layer with a channel number of 128->256, a kernel size of 4, a stride of 2, and a padding of 1, a leaky linear rectification function with a slope of 0.2, and an instance normalization layer; The fourth convolutional block includes a convolutional layer with a channel number of 256->512, a kernel size of 4, a stride of 1, and a padding of 1, a leaky linear rectification function with a slope of 0.2, and an instance normalization layer; The fifth convolutional block includes a convolutional layer with a channel number of 512->1, a kernel size of 4, a stride of 1, and a padding of 1; The first branch includes a 1*1 convolutional layer and a linear rectification function; The second branch includes a dilated convolution with a kernel size of 3 and a dilation rate of 1, and a linear rectification function; The third branch includes a dilated convolution with a kernel size of 3 and a dilation rate of 3, and a linear rectification function; The fourth branch includes a dilated convolution with a kernel size of 3 and a dilation rate of 6, and a linear rectification function; The fifth branch includes average pooling, a 1*1 convolution layer, a linear rectification function and an upsampling unit.

9. A device for attention-based visible light-infrared image cross-modal conversion, wherein the device is used to implement the attention-based visible light-infrared image cross-modal conversion method according to any one of claims 1 to 8, characterized in that: The device comprises: A first acquisition submodule is used to acquire a training sample data set and an initial image cross-modal conversion generator to be trained, where the image cross-modal conversion generator is used to convert visible light images and infrared images; A training submodule is used to train the initial image cross-modal conversion generator based on the training sample data set to obtain a trained image cross-modal conversion generator; An acquisition submodule is used to acquire an image to be processed, where the image to be processed is a visible light image or an infrared image; The conversion submodule is used to input the image to be processed into a trained image cross-modal conversion generator to obtain a modality conversion image corresponding to the image to be processed.

10. A device for cross-modal conversion of visible light to infrared images based on attention, characterized in that: The attention-based visible light-infrared image cross-modal conversion device includes: processor; A memory having computer-readable instructions stored thereon, wherein when the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 8 is implemented.

Images