A visible light to infrared method, system, device and storage medium based on semantic constraints and physical models

Abstract

The application discloses a visible light to infrared method and system based on semantic constraints and a physical model, wherein a visible light image and a real infrared image are collected simultaneously; a conversion infrared model is constructed, the conversion infrared model comprising a semantic segmentation model, a generator, a first discriminator, a second discriminator and a loss function; visible light data is input into the semantic segmentation model to obtain a semantic segmentation image with semantic labels; corresponding temperature values are extracted from a temperature distribution database according to the semantic labels, and the temperature values of the visible light image are mapped to obtain mapped gray scale values of the visible light image; the visible light image and the mapped gray scale values are input into the generator, and a generated infrared image is output; paired visible light images and real infrared images are input into the first discriminator, paired visible light images and generated infrared images are input into the second discriminator, and loss discrimination is performed by using the loss function to optimize the generator, and a trained conversion infrared model is obtained. The application realizes high-fidelity infrared image conversion.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing image spatiotemporal fusion technology, and particularly relates to a visible light to infrared conversion method, system, device and storage medium based on semantic constraints and physical models. Background Technology

[0002] Infrared imagery plays a crucial role in nighttime surveillance, navigation, and target tracking, but its high cost puts pressure on budgets. Drones, with their low cost, ease of operation, and high-resolution visible light imaging capabilities, have advantages in target recognition. Converting the visible light information from drone recognition data into mid-wave infrared images can improve target feature recognition capabilities.

[0003] UAV image processing employs photoelectric imaging technology to generate and analyze images covering different spectral bands. Visible light and infrared images are complementary in spectral range and performance characteristics. Infrared images can penetrate obstacles such as clouds, fog, and smoke, reflecting the thermal radiation characteristics of targets and possessing all-weather, all-time imaging capabilities, thus compensating for the shortcomings of visible light images and being widely used in fields such as surveillance and navigation.

[0004] However, converting visible light to infrared images from UAV imagery faces numerous challenges. Traditional methods often neglect the physical properties of infrared radiation, resulting in insufficient realism and accuracy in the generated infrared images. For example, traditional methods frequently produce distorted infrared images when dealing with lighting variations and complex backgrounds in complex environments.

[0005] Therefore, the above problems urgently need to be solved. Summary of the Invention

[0006] Purpose of the invention: The purpose of this invention is to provide a visible light to infrared conversion method, system, device and storage medium based on semantic constraints and physical models, so as to achieve high-fidelity infrared image conversion.

[0007] Technical solution: To achieve the above objectives, the visible light to infrared conversion method of the present invention includes the following steps:

[0008] Simultaneously acquire visible light images and infrared images, and perform preprocessing to obtain paired visible light images and real infrared images;

[0009] An infrared conversion model is constructed, which includes a semantic segmentation model, a generator, a first discriminator, a second discriminator, and a loss function. Visible light data is input into the semantic segmentation model to obtain a semantically segmented image with semantic labels. Based on the semantic labels, the corresponding temperature values ​​are extracted from the temperature distribution database. The temperature values ​​of the visible light image are mapped to the corresponding regions of the infrared image to obtain the mapped grayscale values ​​of the visible light image.

[0010] The generator includes multiple encoders and decoders. A visible light image and mapped grayscale values ​​are input into the generator. The image is processed by a first encoder to output a first feature map. The first feature map undergoes max pooling to output a pooled first feature map. The pooled first feature map is then processed by a second encoder to output a second feature map. The second feature map undergoes max pooling to output a pooled second feature map. The pooled second feature map is then processed by a third encoder to output a third feature map. The third feature map undergoes max pooling to output a pooled third feature map. The pooled third feature map is then processed by a fourth encoder. The fourth feature map is output; the fourth feature map is sent to the fifth encoder for processing and outputs the fifth feature map; the upsampled fifth feature map and the fourth feature map are tensor concatenated and then input into the first decoder; the upsampled first decoder output features are tensor concatenated with the third feature map and then input into the second decoder; the upsampled second decoder output features are tensor concatenated with the second feature map and then input into the third decoder; the upsampled third decoder output features are tensor concatenated with the first feature map and then input into the fourth decoder; the upsampled fourth decoder output features are added to the visible light image and the mapped grayscale value and then output to generate the infrared image.

[0011] By inputting paired visible light images and real infrared images into the first discriminator, and inputting paired visible light images and generated infrared images into the second discriminator, and using a loss function to perform loss discrimination, the generator is optimized to obtain the trained infrared conversion model.

[0012] Optionally, the mapping of the temperature value of the visible light image to the corresponding region of the infrared image to obtain the mapped grayscale value of the visible light image is specifically performed using a temperature-to-grayscale calibration function. The expression for the temperature-to-grayscale calibration function is as follows:

[0013] ,

[0014] in, and The measurement range of the infrared image sensor; The first temperature value. This represents the mapped grayscale value of the visible light image.

[0015] Optionally, the semantic segmentation model includes a pixel decoder and a Transformer decoder. The pixel decoder has 6 layers, each containing a convolutional layer, an attention block, and a DropPath residual. The Transformer decoder has 9 layers, each containing an attention block and a feedforward network block. The feedforward network block consists of a convolutional layer, a ReLU nonlinear activation function, and a residual.

[0016] Optionally, the encoder includes three parallel coding branches: a first coding branch, a second coding branch, and a third coding branch. In the first coding branch, the input features are processed by convolution and LeRU activation functions to obtain a first feature. The first feature is then processed by convolution and LeRU activation functions to obtain a second feature. The first and second features are added together to obtain a third feature. The third feature is then processed by convolution and LeRU activation functions to obtain a fourth feature. The third and fourth features are then added together to obtain the first coding branch feature. In the second coding branch, the input features are processed by two layers of convolution and LeRU activation functions to obtain a second coding branch feature. In the third coding branch, the input features are processed by convolution and LeRU activation functions to obtain a fifth feature. The fifth feature is then processed by convolution and LeRU activation functions to obtain a sixth feature. The fifth and sixth features are concatenated using tensors to obtain a seventh feature. The seventh feature is then processed by convolution and LeRU activation functions to obtain an eighth feature. The fifth, seventh, and eighth features are concatenated using tensors to obtain the third coding branch feature. The first, second, and third coding branch features are concatenated using tensors and then subjected to input convolution to obtain the output feature.

[0017] Optionally, the decoder includes four parallel decoding branches: a first decoding branch, a second decoding branch, a third decoding branch, and a fourth decoding branch. Input features are input into the decoder and processed by convolution and LeRU activation functions to obtain first input features, second input features, third input features, and fourth input features. In the first decoding branch, the first input features are inverted by pixel padding, processed by convolution and LeRU activation functions, and then multiplied with the inverted first input features to obtain a first intermediate feature. The first intermediate feature is added to the inverted first input features to obtain a first transition feature. The first transition feature is processed by convolution and LeRU activation functions and then added to the first transition feature to obtain the first decoding branch feature. In the second decoding branch, the second input feature is multiplied by the first intermediate feature, processed by convolution and LeRU activation functions to obtain a second intermediate feature. The second intermediate feature is added to the second input feature to obtain a second transition feature. The second transition feature is processed by convolution and LeRU activation functions. The second decoding branch feature is obtained by adding the second transition feature after processing with the U activation function; in the third decoding branch, the third input feature is multiplied with the second intermediate feature and then processed by convolution and LeRU activation function to obtain the third intermediate feature. The third intermediate feature is added with the third input feature to obtain the third transition feature. The third transition feature is then processed by convolution and LeRU activation function and added with the third transition feature to obtain the third decoding branch feature; in the fourth decoding branch, the fourth input feature is multiplied with the third intermediate feature and then processed by convolution and LeRU activation function to obtain the fourth intermediate feature. The fourth intermediate feature is added with the fourth input feature to obtain the fourth transition feature. The fourth transition feature is then processed by convolution and LeRU activation function and added with the fourth transition feature to obtain the fourth decoding branch feature; the first decoding branch feature, the second decoding branch feature, the third decoding branch feature, and the fourth decoding branch feature are concatenated into a tensor and then processed by convolution and LeRU activation function to obtain the output feature.

[0018] Optionally, both the first discriminator and the second discriminator include a splicing layer, a first convolutional module, a second convolutional module, a third convolutional module, a fourth convolutional module, and a fifth convolutional module. The first convolutional module includes a convolutional layer and an activation function. The second, third, and fourth convolutional modules each include a convolutional layer, a normalization layer, and an activation function. The fifth convolutional module includes a convolutional layer.

[0019] Optionally, the loss function includes L1 loss, GAN loss, and gradient loss.

[0020] The L1 loss function is expressed as:

[0021] ,

[0022] in, express and Pixel-level data were taken from real infrared images and generated infrared images, respectively. Indicates the expected value;

[0023] The GAN loss function is expressed as follows:

[0024] ,

[0025] in, and These represent the generator and discriminator, respectively. This indicates that the probability of identifying the visible light image and the true infrared image as real is relatively high. This indicates that the probability of the expected visible light image and the generated infrared image being true is low; Visible light image, For true infrared images, To generate an infrared image;

[0026] The gradient loss function is expressed as:

[0027] ,

[0028] in, This indicates gradient calculation.

[0029] The visible light to infrared conversion system of the present invention includes:

[0030] The data acquisition module is used to simultaneously acquire visible light images and infrared images, and perform preprocessing to obtain paired visible light images and real infrared images;

[0031] The infrared conversion model construction module is used to build an infrared conversion model, which includes a semantic segmentation model, a generator, a first discriminator, a second discriminator, and a loss function. Visible light data is input into the semantic segmentation model to obtain a semantically segmented image with semantic labels. Based on the semantic labels, the corresponding temperature values ​​are extracted from the temperature distribution database. The temperature values ​​of the visible light image are mapped to the corresponding regions of the infrared image to obtain the mapped grayscale values ​​of the visible light image.

[0032] The generator includes multiple encoders and decoders. A visible light image and mapped grayscale values ​​are input into the generator. The image is processed by a first encoder to output a first feature map. The first feature map undergoes max pooling to output a pooled first feature map. The pooled first feature map is then processed by a second encoder to output a second feature map. The second feature map undergoes max pooling to output a pooled second feature map. The pooled second feature map is then processed by a third encoder to output a third feature map. The third feature map undergoes max pooling to output a pooled third feature map. The pooled third feature map is then processed by a fourth encoder. The fourth feature map is output; the fourth feature map is sent to the fifth encoder for processing and outputs the fifth feature map; the upsampled fifth feature map and the fourth feature map are tensor concatenated and then input into the first decoder; the upsampled first decoder output features are tensor concatenated with the third feature map and then input into the second decoder; the upsampled second decoder output features are tensor concatenated with the second feature map and then input into the third decoder; the upsampled third decoder output features are tensor concatenated with the first feature map and then input into the fourth decoder; the upsampled fourth decoder output features are added to the visible light image and the mapped grayscale value and then output to generate the infrared image.

[0033] By inputting paired visible light images and real infrared images into the first discriminator, and inputting paired visible light images and generated infrared images into the second discriminator, and using a loss function to perform loss discrimination, the generator is optimized to obtain the trained infrared conversion model.

[0034] The electronic device of the present invention includes a processor and a storage medium;

[0035] The storage medium is used to store instructions;

[0036] The processor is configured to operate according to the instructions to perform the steps of the method described above.

[0037] The present invention discloses a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the method described above.

[0038] Beneficial Effects: Compared with existing technologies, this invention has the following significant advantages: The physical characteristics of infrared radiation and the generative capabilities of deep neural networks are utilized to construct a specialized deep learning model. This model learns the features of visible light images from data and ensures, through physical constraints, that the generated infrared images conform to the physical laws of infrared radiation, thereby achieving high-fidelity infrared image conversion. Furthermore, this invention utilizes semantic tags mapped to physical temperature, and then converts the temperature to infrared grayscale through a calibration function, forming pixel-level physically interpretable priors. This adds information from the fundamental principles of infrared images, making the generated infrared images closer to real infrared images. The encoder in this invention employs a three-way parallel structure, which not only expands the perception domain and deepens the network but also effectively conveys image information through local residual connections, ensuring... The smooth transmission of image information and the connection of outputs from different layers enable the model to reuse features at deeper levels, which further enhances the model's ability to capture image features and helps the model capture features from different levels to achieve the goal of multi-scale feature extraction. The decoder of this invention weights the feature spaces at different scales, enabling the network to better focus on spatial information at different scales in the image. At the same time, by integrating features at different scales, the network can better focus on important information in the image, making the generated infrared image more consistent with human visual perception. In addition to L1 loss and GAN loss, the loss function proposed in this invention also incorporates gradient loss. Through gradient calculation, edge detail features can be better constrained, resulting in higher clarity of object edges in the generated image. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of the structure of the infrared transfer model of the present invention;

[0040] Figure 2 This is a schematic diagram of the generator structure in this invention;

[0041] Figure 3 This is a schematic diagram of the encoder structure in this invention;

[0042] Figure 4 This is a schematic diagram of the decoder structure in this invention;

[0043] Figure 5 This is a schematic diagram of the discriminator in this invention;

[0044] Figure 6 This is a schematic diagram of the semantic segmentation model in this invention. Detailed Implementation

[0045] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0046] Example 1: A visible light to infrared conversion method based on semantic constraints and physical models according to the present invention includes the following steps:

[0047] S1. The drone is equipped with a visible light camera and an infrared thermal imager, which simultaneously capture visible light and infrared images to ensure spatiotemporal alignment.

[0048] This is used to preprocess the acquired visible light images to obtain preprocessed image data. The preprocessing includes normalization, size unification, noise filtering, image registration, and image enhancement. The image enhancement includes using rotation, scaling, and color perturbation data enhancement techniques to improve the robustness of the model.

[0049] S2. Construct an infrared conversion model, such as Figure 1 As shown, the infrared conversion model includes a semantic segmentation model, a generator, a discriminator, and a loss function. Visible light data is input into the semantic segmentation model to obtain a semantically segmented image with semantic labels. Based on the semantic labels, the corresponding first temperature value is extracted from a temperature distribution database. The first temperature value of the visible light image is determined using a temperature-to-grayscale calibration function. Mapping to the corresponding area of ​​the infrared image generates the mapped grayscale value of the visible light image;

[0050] Temperature to grayscale calibration function:

[0051] ,

[0052] in, and This refers to the measurement range of the infrared thermal imager. The first temperature value. This is the grayscale value of the predicted infrared image from the visible light image.

[0053] like Figure 6 As shown, the semantic segmentation model includes a pixel decoder and a Transformer decoder. The pixel decoder has 6 layers, each containing an 8-head attention block with a stride of 1, padding size of 1, and 4 sampling points, as well as a DropPath residual. The Transformer decoder has 9 layers, each containing a 32-dimensional 8-head attention block and a feedforward network block. Each feedforward network block consists of a 1×1 convolution, a ReLU nonlinear activation function, and a residual.

[0054] like Figure 2 As shown, the generator includes an encoder and a decoder. The preprocessed visible light image is input into the encoder and decoded by the decoder to obtain the generated infrared image.

[0055] The preprocessed visible light image and mapped grayscale values ​​are input into the generator. After feature extraction and compression by the first encoder, a first feature map is output. This first feature map undergoes max pooling to further compress its spatial size and expand its feature channels, resulting in a pooled first feature map. The pooled first feature map is then fed into the second encoder for deeper feature extraction, outputting a second feature map with richer semantic information but smaller spatial size. This second feature map undergoes max pooling to output a pooled second feature map. The pooled second feature map is then fed into the third encoder for deeper feature extraction, outputting a third feature map with richer semantic information but smaller spatial size. This third feature map undergoes max pooling to output a pooled third feature map. The pooled third feature map... The feature map is fed into the fourth encoder for deeper feature extraction, outputting a fourth feature map with richer semantic information but smaller spatial size. The fourth feature map is fed into the fifth encoder for deeper feature extraction, outputting a fifth feature map. The upsampled fifth feature map and the fourth feature map are concatenated by tensors and then input into the first decoder. The output features of the upsampled first decoder are concatenated by tensors with the third feature map and then input into the second decoder. The output features of the upsampled second decoder are concatenated by tensors with the second feature map and then input into the third decoder. The output features of the upsampled third decoder are concatenated by tensors with the first feature map and then input into the fourth decoder. The output features of the upsampled fourth decoder are added to the visible light image and the mapped grayscale value to generate an infrared image.

[0056] like Figure 3 As shown, the encoder includes three parallel coding branches: a first coding branch, a second coding branch, and a third coding branch. In the first coding branch, the input features are processed by a 1×1 convolution and a LeRU activation function to obtain the first feature. The first feature is processed by a 3×3 convolution and a LeRU activation function to obtain the second feature. The first feature and the second feature are added together to obtain the third feature. The third feature is processed by a 3×3 convolution and a LeRU activation function to obtain the fourth feature. The third feature and the fourth feature are added together to obtain the first coding branch feature. In the second coding branch, the input features are processed by two layers of 1×1 convolution and a LeRU activation function to obtain the second coding branch feature.

[0057] In the third coding branch, the input features are processed by 1×1 convolution and LeRU activation function to obtain the fifth feature. The fifth feature is processed by 3×3 convolution and LeRU activation function to obtain the sixth feature. The fifth and sixth features are concatenated by tensor to obtain the seventh feature. The seventh feature is processed by 3×3 convolution and LeRU activation function to obtain the eighth feature. The fifth, seventh and eighth features are concatenated by tensor to obtain the features of the third coding branch.

[0058] The first, second, and third coding branch features are concatenated into tensors and then input into a 1×1 convolution to obtain the output features.

[0059] The encoder consists of multiple convolutional blocks, residual blocks, and activation functions. It fully extracts image details and multi-scale features through three parallel branch structures. Each parallel branch consists of a 1×1 convolution, a 3×3 convolution, residual dense connections, and the LeRU activation function. The 1×1 convolutional kernel is used for channel adjustment, which enhances the model's width while maintaining parameter efficiency. The multi-branch structure expands the receptive field and deepens the feature mapping depth. The internal dense connection strategy establishes fusion channels between features in different layers, enabling deeper feature reuse.

[0060] like Figure 4 As shown, the decoder includes four parallel decoding branches: a first decoding branch, a second decoding branch, a third decoding branch, and a fourth decoding branch. In the input feature encoder, the first, second, third, and fourth input features are obtained after processing with a 3×3 convolution and a LeRU activation function. In the first decoding branch, the first input feature is inverted by one pixel, then processed with a 3×3 convolution and a LeRU activation function, and multiplied with the inverted first input feature to obtain the first intermediate feature. The first intermediate feature is added to the inverted first input feature to obtain the first transition feature. The first transition feature is then processed with a 3×3 convolution and a LeRU activation function and added to the first transition feature to obtain the first decoding branch feature. In the second decoding branch, the second input feature is multiplied by the first intermediate feature, then processed with a 3×3 convolution and a LeRU activation function to obtain the second intermediate feature. The second intermediate feature is added to the second input feature to obtain the second transition feature. The second transition feature is then processed with a 3×3 convolution and a LeRU activation function and added to the second transition feature to obtain the second... Decoding branch features: In the third decoding branch, the third input feature is multiplied by the second intermediate feature, then processed by a 3×3 convolution and LeRU activation function to obtain the third intermediate feature. The third intermediate feature is added to the third input feature to obtain the third transition feature. The third transition feature is processed by a 3×3 convolution and LeRU activation function and then added to the third transition feature to obtain the third decoding branch feature. In the fourth decoding branch, the fourth input feature is multiplied by the third intermediate feature, then processed by a 3×3 convolution and LeRU activation function to obtain the fourth intermediate feature. The fourth intermediate feature is added to the fourth input feature to obtain the fourth transition feature. The fourth transition feature is processed by a 3×3 convolution and LeRU activation function and then added to the fourth transition feature to obtain the fourth decoding branch feature. The first decoding branch feature, the second decoding branch feature, the third decoding branch feature, and the fourth decoding branch feature are concatenated into a tensor and then processed by a 1×1 convolution and LeRU activation function to obtain the output feature.

[0061] The decoder first uses convolution to separate four sets of features, then inversely pads the first feature with one pixel, and generates a spatial attention map using a 3×3 convolution. The attention map is weighted and multiplied with the features, and then the information is integrated using a residual with a convolution kernel size of 3×3 and a stride of 1 to obtain the first layer of feature output. Then, the first layer output and the second set of features are added together and passed through a residual block to obtain the second layer of feature output. The last two sets of features are then constructed in the same way to finally obtain an output containing four layers of features. The output features can guide the network to focus on salient regions in the image.

[0062] like Figure 5 As shown, both the first discriminator and the second discriminator include a splicing layer, a first convolutional module, a second convolutional module, a third convolutional module, a fourth convolutional module, and a fifth convolutional module. The first convolutional module includes a convolutional layer and an activation function. The second, third, and fourth convolutional modules each include a convolutional layer, a normalization layer, and an activation function. The fifth convolutional module includes a convolutional layer.

[0063] By inputting paired visible light images and real infrared images into the first discriminator, and inputting paired visible light images and generated infrared images into the second discriminator, and using a loss function for loss discrimination, the discriminator learns to distinguish between "paired real images" and "paired fake images", and then feeds back to optimize the generator, thus obtaining the trained infrared conversion model.

[0064] The discriminator comprises five convolutional modules with a receptive field of 70×70. It discriminates information within the receptive field of the generated image. The discriminator network outputs a matrix, where the average value of each point typically describes the discrimination result. This approach allows for image discrimination at a more detailed level, considering both local and global information.

[0065] Loss functions include L1 loss, GAN loss, and gradient loss.

[0066] The L1 loss function is expressed as:

[0067] ,

[0068] in, express and Pixel-level data were taken from real infrared images and generated infrared images, respectively. Indicates the expected value;

[0069] The GAN loss function is expressed as:

[0070] ,

[0071] in, and These represent the generator and discriminator, respectively. This indicates that the probability of identifying the visible light image and the true infrared image as real is relatively high. This indicates that the probability of the expected visible light image and the generated infrared image being true is low; Visible light image, For true infrared images, To generate an infrared image;

[0072] The gradient loss function is expressed as:

[0073] ,

[0074] in, This indicates gradient calculation.

[0075] The loss function of this invention quantizes targets such as pixel fidelity, structural consistency, and texture clarity into scalar signals, calculates the differences, and drives the generator and discriminator to iterate through backpropagation, thereby guiding the network to converge.

[0076] The visible light image to be converted is input into the trained infrared conversion model to generate an infrared image.

[0077] Example 2: The visible light to infrared conversion system of the present invention includes:

[0078] The data acquisition module is used to simultaneously acquire visible light images and infrared images, and perform preprocessing to obtain paired visible light images and real infrared images;

[0079] The infrared conversion model construction module is used to build an infrared conversion model, which includes a semantic segmentation model, a generator, a first discriminator, a second discriminator, and a loss function. Visible light data is input into the semantic segmentation model to obtain a semantically segmented image with semantic labels. Based on the semantic labels, the corresponding temperature values ​​are extracted from the temperature distribution database. The temperature values ​​of the visible light image are mapped to the corresponding regions of the infrared image to obtain the mapped grayscale values ​​of the visible light image.

[0080] The generator includes multiple encoders and decoders. A visible light image and mapped grayscale values ​​are input into the generator. The image is processed by a first encoder to output a first feature map. The first feature map undergoes max pooling to output a pooled first feature map. The pooled first feature map is then processed by a second encoder to output a second feature map. The second feature map undergoes max pooling to output a pooled second feature map. The pooled second feature map is then processed by a third encoder to output a third feature map. The third feature map undergoes max pooling to output a pooled third feature map. The pooled third feature map is then processed by a fourth encoder. The fourth feature map is output; the fourth feature map is sent to the fifth encoder for processing and outputs the fifth feature map; the upsampled fifth feature map and the fourth feature map are tensor concatenated and then input into the first decoder; the upsampled first decoder output features are tensor concatenated with the third feature map and then input into the second decoder; the upsampled second decoder output features are tensor concatenated with the second feature map and then input into the third decoder; the upsampled third decoder output features are tensor concatenated with the first feature map and then input into the fourth decoder; the upsampled fourth decoder output features are added to the visible light image and the mapped grayscale value and then output to generate the infrared image.

[0081] By inputting paired visible light images and real infrared images into the first discriminator, and inputting paired visible light images and generated infrared images into the second discriminator, and using a loss function to perform loss discrimination, the generator is optimized to obtain the trained infrared conversion model.

[0082] Example 3: An electronic device according to the present invention includes a processor and a storage medium;

[0083] The storage medium is used to store instructions;

[0084] The processor is configured to operate according to the instructions to perform the steps of the method described above.

[0085] Example 4: A computer-readable storage medium according to the present invention stores a computer program thereon, characterized in that the program, when executed by a processor, implements the steps of the method described above.

[0086] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made under the concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the patent protection scope of the present invention.

Claims

1. A method for converting visible light to infrared based on semantic constraints and physical models, the method comprising: obtaining a semantic constraint and a physical model; and converting the visible light to infrared based on the semantic constraint and the physical model. Includes the following steps: Simultaneously acquire visible light images and infrared images, and perform preprocessing to obtain paired visible light images and real infrared images; An infrared conversion model is constructed, which includes a semantic segmentation model, a generator, a first discriminator, a second discriminator, and a loss function. Visible light data is input into the semantic segmentation model to obtain a semantically segmented image with semantic labels. Based on the semantic labels, the corresponding temperature values ​​are extracted from the temperature distribution database. The temperature values ​​of the visible light image are mapped to the corresponding regions of the infrared image to obtain the mapped grayscale values ​​of the visible light image. The generator includes multiple encoders and decoders. A visible light image and mapped grayscale values ​​are input into the generator, processed by the first encoder, and a first feature map is output. The first feature map is subjected to max pooling to output the pooled first feature map; the pooled first feature map is then fed into the second encoder for processing to output the second feature map. The second feature map is max-pooled to output the pooled second feature map; the pooled second feature map is fed into the third encoder for processing to output the third feature map; the third feature map is max-pooled to output the pooled third feature map; the pooled third feature map is fed into the fourth encoder for processing to output the fourth feature map; the fourth feature map is fed into the fifth encoder for processing to output the fifth feature map. The upsampled fifth feature map is concatenated with the fourth feature map using a tensor and then input into the first decoder. The upsampled first decoder output feature is concatenated with the third feature map using a tensor and then input into the second decoder. The upsampled second decoder output feature is concatenated with the second feature map using a tensor and then input into the third decoder. The upsampled third decoder output feature is concatenated with the first feature map using a tensor and then input into the fourth decoder. The upsampled fourth decoder output feature is added to the visible light image and the mapped grayscale value to generate the infrared image. By inputting paired visible light images and real infrared images into the first discriminator, and inputting paired visible light images and generated infrared images into the second discriminator, and using loss functions to perform loss discrimination, the generator is optimized to obtain the trained infrared conversion model; the loss functions include L1 loss, GAN loss and gradient loss. The GAN loss function is expressed as follows: ， wherein, denotes the generator, denotes the first discriminator, denotes the second discriminator, denotes that the probability that the visible image and the real infrared image are true is higher, denotes that the probability that the visible image and the generated infrared image are true is lower; is a visible image, is a real infrared image, is a generated infrared image; The gradient loss function is expressed as: ， wherein denotes the gradient calculation.

2. The visible light to infrared conversion method according to claim 1, characterized in that, The process of mapping the temperature value of the visible light image to the corresponding region of the infrared image to obtain the mapped grayscale value of the visible light image is specifically performed using a temperature-to-grayscale calibration function. The expression for the temperature-to-grayscale calibration function is as follows: ， in, and The measurement range of the infrared image sensor; The first temperature value. This represents the mapped grayscale value of the visible light image.

3. The visible light to infrared conversion method according to claim 1, characterized in that, The semantic segmentation model includes a pixel decoder and a Transformer decoder. The pixel decoder has 6 layers, each containing a convolutional layer, an attention block, and a DropPath residual. The Transformer decoder has 9 layers, each containing an attention block and a feedforward network block. The feedforward network block consists of a convolutional layer, a ReLU nonlinear activation function, and a residual.

4. The visible light to infrared conversion method according to claim 1, characterized in that, The encoder includes three parallel coding branches: a first coding branch, a second coding branch, and a third coding branch. In the first coding branch, the input features are processed by convolution and LeRU activation function to obtain the first feature. The first feature is processed by convolution and LeRU activation function to obtain the second feature. The first feature and the second feature are added together to obtain the third feature. The third feature is processed by convolution and LeRU activation function to obtain the fourth feature. The third feature and the fourth feature are added together to obtain the first coding branch feature. In the second coding branch, the input features are processed by two layers of convolution and LeRU activation function to obtain the second coding branch features; In the third coding branch, the input features are processed by convolution and LeRU activation function to obtain the fifth feature. The fifth feature is processed by convolution and LeRU activation function to obtain the sixth feature. The fifth and sixth features are concatenated by tensor to obtain the seventh feature. The seventh feature is processed by convolution and LeRU activation function to obtain the eighth feature. The fifth, seventh and eighth features are concatenated by tensor to obtain the third coding branch features. The first, second, and third coding branch features are concatenated into tensors and then input into a convolutional layer to obtain the output features.

5. The visible light to infrared conversion method according to claim 1, characterized in that, The decoder includes four parallel decoding branches: a first decoding branch, a second decoding branch, a third decoding branch, and a fourth decoding branch. Input features are input into the decoder and processed by convolution and LeRU activation functions to obtain first input features, second input features, third input features, and fourth input features. In the first decoding branch, the first input features are inverted by pixels, processed by convolution and LeRU activation functions, and then multiplied with the inverted first input features to obtain a first intermediate feature. The first intermediate feature is added to the inverted first input features to obtain a first transition feature. The first transition feature is processed by convolution and LeRU activation functions and then added to the first transition feature to obtain the first decoding branch feature. In the second decoding branch, the second input feature is multiplied by the first intermediate feature and then processed by convolution and LeRU activation function to obtain the second intermediate feature. The second intermediate feature is added to the second input feature to obtain the second transition feature. The second transition feature is processed by convolution and LeRU activation function and then added to the second transition feature to obtain the second decoding branch feature. In the third decoding branch, the third input feature is multiplied by the second intermediate feature, then processed by convolution and LeRU activation function to obtain the third intermediate feature. The third intermediate feature is added to the third input feature to obtain the third transition feature. The third transition feature is processed by convolution and LeRU activation function and then added to the third transition feature to obtain the third decoding branch feature. In the fourth decoding branch, the fourth input feature is multiplied by the third intermediate feature, then processed by convolution and LeRU activation function to obtain the fourth intermediate feature. The fourth intermediate feature is added to the fourth input feature to obtain the fourth transition feature. The fourth transition feature is processed by convolution and LeRU activation function and then added to the fourth transition feature to obtain the fourth decoding branch feature. The features from the first, second, third, and fourth decoding branches are concatenated into tensors and then processed by convolution and LeRU activation functions to obtain the output features.

6. The visible light to infrared conversion method according to claim 1, characterized in that, Both the first discriminator and the second discriminator include a splicing layer, a first convolutional module, a second convolutional module, a third convolutional module, a fourth convolutional module, and a fifth convolutional module. The first convolutional module includes a convolutional layer and an activation function. The second, third, and fourth convolutional modules each include a convolutional layer, a normalization layer, and an activation function. The fifth convolutional module includes a convolutional layer.

7. The visible light to infrared conversion method according to claim 1, characterized in that, The L1 loss function is expressed as: ， in, express and Pixel-level data were taken from real infrared images and generated infrared images, respectively. This represents the expected value.

8. A visible light to infrared conversion system based on semantic constraints and a physical model, characterized in that, include: The data acquisition module is used to simultaneously acquire visible light images and infrared images, and perform preprocessing to obtain paired visible light images and real infrared images; The infrared conversion model construction module is used to build an infrared conversion model, which includes a semantic segmentation model, a generator, a first discriminator, a second discriminator, and a loss function. Visible light data is input into the semantic segmentation model to obtain a semantically segmented image with semantic labels. Based on the semantic labels, the corresponding temperature values ​​are extracted from the temperature distribution database. The temperature values ​​of the visible light image are mapped to the corresponding regions of the infrared image to obtain the mapped grayscale values ​​of the visible light image. The generator includes multiple encoders and decoders. A visible light image and mapped grayscale values ​​are input into the generator, processed by the first encoder, and a first feature map is output. The first feature map is subjected to max pooling to output the pooled first feature map; the pooled first feature map is then fed into the second encoder for processing to output the second feature map. The second feature map is max-pooled to output the pooled second feature map; the pooled second feature map is fed into the third encoder for processing to output the third feature map; the third feature map is max-pooled to output the pooled third feature map; the pooled third feature map is fed into the fourth encoder for processing to output the fourth feature map; the fourth feature map is fed into the fifth encoder for processing to output the fifth feature map. The upsampled fifth feature map is concatenated with the fourth feature map using a tensor and then input into the first decoder. The upsampled first decoder output feature is concatenated with the third feature map using a tensor and then input into the second decoder. The upsampled second decoder output feature is concatenated with the second feature map using a tensor and then input into the third decoder. The upsampled third decoder output feature is concatenated with the first feature map using a tensor and then input into the fourth decoder. The upsampled fourth decoder output feature is added to the visible light image and the mapped grayscale value to generate the infrared image. By inputting paired visible light images and real infrared images into the first discriminator, and paired visible light images and generated infrared images into the second discriminator, and using a loss function for loss discrimination to optimize the generator, a trained infrared-to-digital conversion model is obtained; the loss function includes L1 loss, GAN loss, and gradient loss; the GAN loss function is expressed as: ， in, Represents a generator. This represents the first discriminator. This represents the second discriminator. This indicates that the probability of identifying the visible light image and the true infrared image as real is relatively high. This indicates that the probability of the expected visible light image and the generated infrared image being true is low; Visible light image, For true infrared images, To generate an infrared image; The gradient loss function is expressed as: ， in, This indicates gradient calculation.

9. An electronic device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1 to 7.

Images