High-fidelity infrared remote sensing image generation method, device and equipment

Abstract

The invention provides a high-fidelity infrared remote sensing image generation method, device and equipment. The method comprises the steps of obtaining a to-be-processed visible light remote sensing image; performing image cutting processing on the to-be-processed visible light remote sensing image to obtain a visible light remote sensing cut image; and inputting the visible light remote sensing cut image into a pre-trained infrared image generation network to generate a corresponding high-fidelity infrared remote sensing image. In a training process of a pre-training infrared image generation network, a visible light remote sensing image sample is used as input data, and radiation spectrum characteristic mapping data is used as prior information to be embedded into the design of the pre-training infrared image generation network. And a specific loss function is designed to guide the learning direction of the pre-trained infrared image generation network, so that the generalization ability and reliability of a deep learning model can be enhanced by utilizing the interpretability of a physical model, and the fidelity and reliability of a high-fidelity infrared remote sensing image finally generated by the pre-trained infrared image generation network are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared remote sensing image generation, and in particular to a method, device and equipment for generating high-fidelity infrared remote sensing images. Background Art

[0002] Infrared remote sensing imaging technology has shown wide application potential in a variety of fields such as aerospace, environmental monitoring, urban planning, infrared guidance and disaster management. These fields have a strong demand for high-fidelity infrared remote sensing image information data to support the development of remote sensing target detection, image interpretation and evaluation, and computer vision technologies. However, compared with the easy accessibility of visible light remote sensing data, the acquisition of infrared remote sensing images is not only difficult but also costly, and the amount of measured data is difficult to fully meet the needs of infrared radiation characteristic measurement and algorithm research of high-value targets. Therefore, developing a technical method that can accurately generate high-fidelity infrared remote sensing images from visible light remote sensing images is of great significance to promote the development of related technologies.

[0003] At present, the generation methods of infrared remote sensing images are mainly divided into two categories: physical modeling simulation generation and deep learning intelligent generation. The physical modeling simulation method is based on radiation transfer theory and physical laws. Although it has good physical interpretability and credibility, it is difficult to fully capture the nonlinear and heterogeneous characteristics of the radiation field when dealing with complex scenes, resulting in deviations between the simulation results and the actual infrared images. In addition, the working efficiency of this method is low and the professional knowledge of the operator is high. The deep learning generation method, especially the generative adversarial network, can significantly improve the simulation efficiency when dealing with complex scenes with its powerful nonlinear mapping ability and automatic feature learning ability. However, due to the lack of high-quality paired data, the deep learning model is easily affected by noise and overfitting, resulting in reduced reliability of the generated results. At the same time, the black box characteristics of the deep learning method also limit its interpretability, further increasing the difficulty of generating high-quality infrared remote sensing images.

[0004] In summary, although the existing technologies have achieved the generation of infrared remote sensing images from two perspectives, namely, remote sensing imaging mechanism and deep learning network, they have not overcome the contradictions of poor realism and low simulation efficiency of physical modeling simulation methods, and poor interpretability and generalization of deep learning generation methods. Therefore, the generated high-fidelity infrared remote sensing images still need to be improved in terms of realism and reliability. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a method, device and equipment for generating high-fidelity infrared remote sensing images.

[0006] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0007] In a first aspect, the present invention provides a method for generating a high-fidelity infrared remote sensing image, comprising:

[0008] Acquire visible light remote sensing images to be processed;

[0009] Performing image cropping processing on the visible light remote sensing image to be processed to obtain a visible light remote sensing cropped image;

[0010] The visible light remote sensing cropped image is input into a pre-trained infrared image generation network to generate a corresponding high-fidelity infrared remote sensing image; wherein the pre-trained infrared image generation network is composed of a pre-trained image generation module and a pre-trained image discrimination module; the pre-trained infrared image generation network uses visible light remote sensing image samples as input data, and uses radiation spectrum characteristic mapping data and target infrared remote sensing images as prior information, and is trained in combination with a preset loss function; the radiation spectrum characteristic mapping data is the spectral characteristic mapping relationship between the visible light remote sensing image samples and the corresponding infrared remote sensing image samples; the preset loss function is used to characterize the matching relationship between the infrared remote sensing image generation data and the target infrared remote sensing image and the radiation spectrum characteristic mapping data.

[0011] Optionally, performing image cropping processing on the visible light remote sensing image to be processed to obtain a visible light remote sensing cropped image includes:

[0012] The visible light remote sensing image to be processed is cropped according to a preset cropping size to obtain a visible light remote sensing cropped image.

[0013] Optionally, the pre-trained image generation module adopts the U-Net architecture in Pix2Pix GAN, wherein the U-Net architecture in Pix2Pix GAN is provided with a fast Fourier transform process in the feature calculation process of the corresponding channel attention and spatial attention;

[0014] The pre-trained image discrimination module adopts the discrimination module in Pix2Pix GAN, and the discrimination module in Pix2Pix GAN is provided with a deformable convolution module and a multi-scale feature fusion discrimination module; the deformable convolution module is provided between the preset convolution layers of the discrimination module in Pix2Pix GAN;

[0015] The multi-scale feature fusion discrimination module is used to perform multi-scale discrimination on infrared remote sensing image generation data and target infrared remote sensing images.

[0016] Optionally, the generation process of the pre-trained infrared image generation network includes:

[0017] Acquire a paired data set; the paired data set includes a plurality of sample pairs of visible light remote sensing image samples and corresponding infrared remote sensing image samples;

[0018] Based on the inversion calculation, the spectral reflectance of each pixel of the paired data set is obtained;

[0019] Based on the preset ground object classification method, the material of the visible light remote sensing image samples is classified to obtain the infrared remote sensing image labeled samples;

[0020] Perform matching processing on the infrared remote sensing image marked samples and spectral reflectance to obtain the material spectral reflectance matching data;

[0021] Based on the material spectral reflectance matching data and the preset quantitative characterization model, the radiation spectral characteristic mapping data is obtained;

[0022] The initial infrared image generation network is trained based on the radiation spectrum characteristic mapping data and the paired data set to obtain a pre-trained infrared image generation network; the initial infrared image generation network has the same structure as the pre-trained infrared image generation network.

[0023] Optionally, the initial infrared image generation network is trained based on the radiation spectrum characteristic mapping data and the paired data set to obtain a pre-trained infrared image generation network, including:

[0024] Iteratively train the initial infrared image generation network based on the radiation spectrum characteristic mapping data, the paired data set, and the preset loss function;

[0025] The initial infrared image generation network that meets the preset stop condition is used as the pre-trained infrared image generation network;

[0026] The preset stop condition includes: the number of iterations meets the preset iteration threshold or the value of the preset loss function continues to converge.

[0027] Optionally, the preset loss function is expressed as:

[0028] Loss = c1L GD +c2Loss IR +c3Loss GP +c4Loss FSM ;

[0029] Among them, Loss represents the preset loss function, L GD Represents the basic loss function, Loss IR Indicates the loss of radiation spectrum characteristics, Loss GP Represents the gradient penalty loss, Loss FSM represents the feature space matching loss, c1 represents the first weight value, c2 represents the second weight value, c3 represents the third weight value, and c4 represents the fourth weight value, where the basic loss function is the sum of the adversarial loss and the conditional loss.

[0030] Alternatively, the radiation spectral characteristic loss is expressed as:

[0031]

[0032] Where i represents the i-th paired data set, N represents the total number of paired data sets, and I mapping Represents the radiation spectrum characteristic mapping data, I input Represents visible light remote sensing image samples, G(I input ) represents the infrared remote sensing image generation data, and |·| represents the absolute value.

[0033] Optionally, the preset quantitative representation model is expressed as:

[0034]

[0035] Among them, I mapping (k,λ p ,λ q ,ρ k ,m,n) means that in the paired dataset with pixel size (m,n), the spectral reflectance is ρ k The material k is from the visible light band λ p To infrared bandλ q The radiation spectrum characteristic mapping data of the paired data set; m represents the number of horizontal pixels in the paired data set, and n represents the number of vertical pixels in the paired data set; Indicates that material k is in the infrared band λ q The corresponding material spectral reflectance matching data, is the material k in the visible light band λ p The corresponding material spectral reflectance matching data; Indicates that the pixel size is (m k ,n k ) in the paired data set, the spectral reflectance is ρ k The material k in the infrared band λ q Total radiation at Indicates that the pixel size is (m k ,n k ) in the paired data set, the spectral reflectance is ρ k The material k in the visible light band λ p Total radiation at m k represents the number of horizontal pixels of material k in the paired dataset, n k represents the number of vertical pixels of material k in the paired dataset, The spectral reflectance is ρ k The material k in the infrared band λ q The total radiation at The spectral reflectance is ρ k The material k in the visible light band λp The total radiation at In the infrared band λ q The material spectral reflectance matching data is The total radiation corresponding to the k materials is Indicates the visible light band λ p The material spectral reflectance matching data is The total radiation corresponding to the k materials.

[0036] In a second aspect, the present invention provides a high-fidelity infrared remote sensing image generation device, the high-fidelity infrared remote sensing image generation device comprising: an acquisition unit, a cropping unit and an image generation unit;

[0037] The acquisition unit is used to: acquire the visible light remote sensing image to be processed;

[0038] The cropping unit is used to: perform image cropping processing on the visible light remote sensing image to be processed to obtain a visible light remote sensing cropped image;

[0039] The image generation unit is used to: input the visible light remote sensing cropped image into a pre-trained infrared image generation network to generate a corresponding high-fidelity infrared remote sensing image; wherein the pre-trained infrared image generation network is composed of a pre-trained image generation module and a pre-trained image discrimination module; the pre-trained infrared image generation network uses visible light remote sensing image samples as input data and uses radiation spectrum characteristic mapping data as prior information, and is trained in combination with a preset loss function; the radiation spectrum characteristic mapping data is a spectral characteristic mapping relationship between a visible light remote sensing image sample and a corresponding infrared remote sensing image sample; the preset loss function is used to characterize the matching relationship between the infrared remote sensing image generation data and the target infrared remote sensing image and the radiation spectrum characteristic mapping data.

[0040] In a third aspect, the present invention provides a high-fidelity infrared remote sensing image generation device, comprising: a processor, a storage medium and a bus, the storage medium storing machine-readable instructions executable by the processor, when the high-fidelity infrared remote sensing image generation device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the high-fidelity infrared remote sensing image generation method as described in the first aspect above.

[0041] The present invention provides a method, device and equipment for generating high-fidelity infrared remote sensing images. A method for generating high-fidelity infrared remote sensing images includes: obtaining a visible light remote sensing image to be processed; performing image cropping processing on the visible light remote sensing image to be processed to obtain a visible light remote sensing cropped image; inputting the visible light remote sensing cropped image into a pre-trained infrared image generation network to generate a corresponding high-fidelity infrared remote sensing image; wherein the pre-trained infrared image generation network is composed of a pre-trained image generation module and a pre-trained image discrimination module; the pre-trained infrared image generation network uses visible light remote sensing image samples as input data, and uses radiation spectrum characteristic mapping data and target infrared remote sensing images as prior information, and is trained in combination with a preset loss function; the radiation spectrum characteristic mapping data is a spectral characteristic mapping relationship between a visible light remote sensing image sample and a corresponding infrared remote sensing image sample; the preset loss function is used to characterize the matching relationship between the infrared remote sensing image generation data and the target infrared remote sensing image and the radiation spectrum characteristic mapping data. In the present invention, during the training process of the pre-trained infrared image generation network, visible light remote sensing image samples are used as input data, and radiation spectrum characteristic mapping data and target infrared remote sensing images are embedded in the design of the pre-trained infrared image generation network as prior information, and a specific loss function is designed to guide the learning direction of the pre-trained infrared image generation network. It is possible to utilize the interpretability of the physical model to enhance the generalization ability and reliability of the deep learning model, and ultimately improve the fidelity and reliability of the high-fidelity infrared remote sensing images generated by the pre-trained infrared image generation network.

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of a process for generating a high-fidelity infrared remote sensing image provided by an embodiment of the present invention;

[0044] Figure 2 The schematic diagram of the structure of the pre-trained image generation module is exemplarily shown;

[0045] Figure 3 The schematic diagram of the structure of the pre-trained image discrimination module with the deformable convolution module added is shown exemplarily;

[0046] Figure 4 The architecture diagram of the pre-trained infrared image generation network after adding the multi-scale feature fusion discrimination module is shown exemplarily;

[0047] Figure 5 A schematic structural diagram of a high-fidelity infrared remote sensing image generation device provided by an embodiment of the present invention;

[0048] Figure 6A schematic structural diagram of a high-fidelity infrared remote sensing image generation device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0050] In order to improve the fidelity and reliability of a high-fidelity infrared remote sensing image ultimately generated by a pre-trained infrared image generation network, an embodiment of the present invention provides a high-fidelity infrared remote sensing image generation method. Figure 1 A flowchart of a method for generating a high-fidelity infrared remote sensing image is provided in an embodiment of the present invention, such as Figure 1 As shown, including:

[0051] S101, obtaining a visible light remote sensing image to be processed.

[0052] The visible light remote sensing image to be processed refers to an image acquired from a ground or aerial platform (such as a satellite, a drone, etc.) that can reflect the situation of sunlight reflected from the ground surface. The visible light remote sensing image to be processed usually contains rich ground object information, such as vegetation, water bodies, and buildings. In the embodiment of the present invention, this image is crucial for generating high-quality infrared images.

[0053] S102, performing image cropping processing on the visible light remote sensing image to be processed to obtain a visible light remote sensing cropped image.

[0054] It is understandable that the directly acquired visible light remote sensing images to be processed usually have very high resolution and wide scene coverage. If the entire visible light remote sensing image to be processed is directly processed, it may require a lot of computing resources and time. By cropping the image, the processing range can be reduced, thereby reducing the computational complexity and improving the processing efficiency. In addition, by cropping, it can be ensured that the image input into the pre-trained infrared image generation network meets the input specifications of the model, avoiding the problem of size mismatch during the processing of the pre-trained infrared image generation network.

[0055] Optionally, S102 may specifically include:

[0056] The visible light remote sensing image to be processed is cropped according to a preset cropping size to obtain a visible light remote sensing cropped image.

[0057] It should be noted that the above preset cropping size satisfies the size information input by the pre-trained infrared image generation network model.

[0058] S103, inputting the visible light remote sensing cropped image into a pre-trained infrared image generation network to generate a corresponding high-fidelity infrared remote sensing image.

[0059] Among them, the pre-trained infrared image generation network is composed of a pre-trained image generation module and a pre-trained image discrimination module; the pre-trained infrared image generation network takes visible light remote sensing image samples as input data, and uses radiation spectrum characteristic mapping data and target infrared remote sensing images as prior information, and is trained together with a preset loss function; the radiation spectrum characteristic mapping data is the spectral characteristic mapping relationship between visible light remote sensing image samples and corresponding infrared remote sensing image samples; the preset loss function is used to characterize the matching relationship between the infrared remote sensing image generation data and the target infrared remote sensing image and the radiation spectrum characteristic mapping data.

[0060] An embodiment of the present invention provides a high-fidelity infrared remote sensing image generation method, comprising: in the training process of a pre-trained infrared image generation network, by taking visible light remote sensing image samples as input data, and at the same time taking radiation spectrum characteristic mapping data and target infrared remote sensing images as prior information and embedding them into the design of the pre-trained infrared image generation network, and designing a specific loss function to guide the learning direction of the pre-trained infrared image generation network, it is possible to utilize the interpretability of the physical model to enhance the generalization ability and reliability of the deep learning model, and ultimately improve the fidelity and reliability of the high-fidelity infrared remote sensing images generated by the pre-trained infrared image generation network.

[0061] Optionally, the pre-trained image generation module adopts the U-Net architecture in Pix2Pix GAN, wherein the U-Net architecture in Pix2Pix GAN is provided with a fast Fourier transform process in the feature calculation process of the corresponding channel attention and spatial attention;

[0062] The pre-trained image discrimination module adopts the discrimination module in Pix2Pix GAN, and the discrimination module in Pix2Pix GAN is provided with a deformable convolution module and a multi-scale feature fusion discrimination module; the deformable convolution module is provided between the preset convolution layers of the discrimination module in Pix2Pix GAN;

[0063] The multi-scale feature fusion discrimination module is used to perform multi-scale discrimination on infrared remote sensing image generation data and target infrared remote sensing images.

[0064] Figure 2 The schematic diagram of the structure of the pre-trained image generation module is shown as an example. Figure 2 As shown, the pre-trained image generation module provided in the embodiment of the present invention adopts the U-Net architecture, and a fast Fourier channel spatial attention mechanism is set on the U-Net architecture in the existing Pix2Pix GAN, that is, a fast Fourier transform process is set in the feature calculation process of the corresponding channel attention and spatial attention, such as Figure 2As shown in the FFCSAM module, after being processed by the pre-trained image generation module, a 3*256*256-dimensional feature map is finally obtained. Figure 3 The structural diagram of the pre-trained image discrimination module with the deformable convolution module added is shown as an example. Figure 3 As shown, the embodiment of the present invention adds a deformable convolution to the discrimination module in the existing Pix2Pix GAN, such as Figure 3 As shown in the green part, Figure 3 The discriminant module in is also equipped with a feature map module, a confidence map module, downsampling and fusion convolution processing. Figure 3 The adding position of the deformable convolution is only exemplary, and the specific adding position can be flexibly set according to the requirements, which is not limited in the embodiment of the present invention. In addition, the feature map module and the confidence map module are both implemented through different convolution layers. Figure 4 The following is an exemplary diagram of the architecture of the pre-trained infrared image generation network after adding a multi-scale feature fusion discrimination module. Figure 4 As shown in the figure, the pre-trained infrared image generation network performs comparative discrimination at three scales, that is, the input visible light remote sensing image samples are first downsampled to three dimensions, and the generated infrared remote sensing image generation data and the target infrared remote sensing image (real data) are correspondingly downsampled to three dimensions, and a corresponding comparison is performed in each dimension. The comparators in the three dimensions are called discriminator 1, discriminator 2 and discriminator 3 respectively.

[0065] It should be noted that by setting the fast Fourier transform processing in the feature calculation process of the channel attention and spatial attention corresponding to the U-Net architecture, the learning ability of the pre-trained infrared image generation network for image features can be enhanced. This helps the pre-trained infrared image generation network to extract richer information from the input visible light remote sensing image, providing strong support for the generation of high-quality infrared remote sensing images.

[0066] The discriminant module of Pix2Pix GAN is itself a powerful image discrimination tool, which can effectively distinguish between real images and generated images. By setting up a deformable convolution module, the discriminant module can more flexibly capture the features in visible light remote sensing images, thereby improving the accuracy of discrimination. The introduction of the multi-scale feature fusion discriminant module enables the discriminant module to perform multi-scale discrimination on infrared remote sensing image generation data and target infrared remote sensing images. This design helps the pre-trained infrared image generation network to better adapt to image inputs of different scales and enhance the robustness of the model.

[0067] In addition, the setting of the deformable convolution module enables the discriminant module to process image data more efficiently and reduce computational redundancy. At the same time, the multi-scale feature fusion discriminant module also helps the pre-trained infrared image generation network to learn the key features of visible light remote sensing image samples more quickly, thereby accelerating the training process.

[0068] Optionally, the generation process of the pre-trained infrared image generation network includes:

[0069] Acquire a paired data set; the paired data set includes a plurality of sample pairs of visible light remote sensing image samples and corresponding infrared remote sensing image samples;

[0070] Based on the inversion calculation, the spectral reflectance of the paired data set at each pixel point is obtained;

[0071] Based on the preset ground object classification method, the visible light remote sensing image samples are classified by material to obtain infrared remote sensing image labeled samples;

[0072] Perform matching processing on the infrared remote sensing image marked samples and spectral reflectance to obtain the material spectral reflectance matching data;

[0073] Based on the material spectral reflectance matching data and the preset quantitative characterization model, the radiation spectral characteristic mapping data is obtained;

[0074] The initial infrared image generation network is trained based on the radiation spectrum characteristic mapping data and the paired data set to obtain a pre-trained infrared image generation network; the initial infrared image generation network has the same structure as the pre-trained infrared image generation network.

[0075] In the embodiment of the present invention, Landsat8 data can be used for experiments. In addition, the paired data set can also be subjected to pre-processing operations such as cropping, rotation, and mirroring to meet the needs of subsequent network training. The size of the image in the paired data set of the present invention is generally set to 256*256, and the paired data set has a total of 6000 pairs.

[0076] The simulation process of infrared remote sensing images requires radiation calculation or mapping changes that conform to the characteristics of different landforms. Therefore, the visible light remote sensing image samples can be classified and marked with serial numbers. This can be done in the following two ways. The preset landform classification method can be:

[0077] (1) Manually supervised / semi-supervised classification, which can be mainly completed using Photoshop and ENVI software;

[0078] (2) Deep learning intelligent classification can be completed using classification networks and label data.

[0079] In the embodiment of the present invention, the reflectivity information corresponding to the infrared remote sensing image in multiple spectral bands can be obtained by inversion calculation; specifically, each satellite has its own corresponding inversion calculation formula, which will not be described in detail in the embodiment of the present invention.

[0080] In addition, after the spectral reflectance is obtained, it can be compared with the spectral reflectance database information to obtain the spectral reflectance curve with the highest matching degree with each material as the spectral reflectance of the material.

[0081] Optionally, the initial infrared image generation network is trained based on the radiation spectrum characteristic mapping data and the paired data set to obtain a pre-trained infrared image generation network, including:

[0082] Iteratively train the initial infrared image generation network based on the radiation spectrum characteristic mapping data, the paired data set, and the preset loss function;

[0083] The initial infrared image generation network that meets the preset stop condition is used as the pre-trained infrared image generation network;

[0084] The preset stop condition includes: the number of iterations meets the preset iteration threshold or the value of the preset loss function continues to converge.

[0085] In the embodiment of the present invention, based on the deep understanding of the scene radiation mechanism and the design of the network architecture, the effective combination of data-driven and physical knowledge is realized through the loss of radiation spectrum characteristics, and the initial infrared image generation network is guided to follow the real radiation transmission law in the learning process, and the initial infrared image generation network is helped to better understand the radiation transmission characteristics between visible light and infrared, thereby enhancing the training effect and generalization ability of the initial infrared image generation network. At the same time, relying on the powerful learning ability of the initial infrared image generation network, the accurate characterization of the random distribution, multipath scattering and nonlinear changes of the complex radiation field between visible light and infrared remote sensing images and the expression of image texture features are improved, thereby improving the accuracy and detail expression of feature mapping. Specifically, the loss function formed by the radiation spectrum characteristic mapping relationship (radiation spectrum characteristic mapping data) of visible light remote sensing to infrared remote sensing is used as the entry point, and quantitative coupling is introduced into the network model, and finally a preset loss function is constructed.

[0086] The default loss function is expressed as:

[0087] Loss = c1L GD +c2Loss IR +c3Loss GP +c4Loss FSM ;

[0088] Among them, Loss represents the preset loss function, L GD Represents the basic loss function, LossIR Indicates the loss of radiation spectrum characteristics, Loss GP Represents the gradient penalty loss, Loss FSM represents the feature space matching loss, c1 represents the first weight value, c2 represents the second weight value, c3 represents the third weight value, and c4 represents the fourth weight value, where the basic loss function is the sum of the adversarial loss and the conditional loss.

[0089] Specifically, during the implementation of this embodiment, a preset loss function including a basic loss function, a radiation spectrum characteristic loss, a gradient penalty loss, and a feature space matching loss can be used as an overall loss function in the training process of the pre-trained infrared image generation network.

[0090] Alternatively, the radiation spectral characteristic loss is expressed as:

[0091]

[0092] Where i represents the i-th paired data set, N represents the total number of paired data sets, and I mapping Represents the radiation spectrum characteristic mapping data, I input Represents visible light remote sensing image samples, G(I input ) represents the infrared remote sensing image generation data, and |·| represents the absolute value.

[0093] Optionally, the preset quantitative representation model is expressed as:

[0094]

[0095] Among them, I mapping (k,λ p ,λ q ,ρ k ,m,n) means that in the paired dataset with pixel size (m,n), the spectral reflectance is ρ k The material k is from the visible light band λ p To infrared bandλ q The radiation spectrum characteristic mapping data of the paired data set; m represents the number of horizontal pixels in the paired data set, and n represents the number of vertical pixels in the paired data set; Indicates that material k is in the infrared band λ q The corresponding material spectral reflectance matching data, is the material k in the visible light band λ p The corresponding material spectral reflectance matching data; Indicates that the pixel size is (m k ,n k ) in the paired data set, the spectral reflectance is ρ k The material k in the infrared band λ q Total radiation at Indicates that the pixel size is (m k ,n k ) in the paired data set, the spectral reflectance is ρ k The material k in the visible light band λ p Total radiation at m k represents the number of horizontal pixels of material k in the paired dataset, n k represents the number of vertical pixels of material k in the paired dataset, The spectral reflectance is ρ k The material k in the infrared band λ q The total radiation at The spectral reflectance is ρ k The material k in the visible light band λ p The total radiation at In the infrared band λ q The material spectral reflectance matching data is The total radiation corresponding to the k materials is Indicates the visible light band λ p The material spectral reflectance matching data is The total radiation corresponding to the k materials.

[0096] It should be noted that, in this embodiment, the calculation of total radiation can be based on the scene radiation energy transmission law, including reflected radiation, self-radiation and reflectivity information of ground material (the above-mentioned preset quantitative characterization model). Solar radiation, sky background radiation, atmospheric path radiation and atmospheric transmittance involved in reflected radiation can be calculated using the Modtran model. Self-radiation can be calculated according to Planck's formula.

[0097] In order to verify the effectiveness of the high-fidelity infrared remote sensing image generation method provided by the embodiment of the present invention, the embodiment of the present invention also conducted a simulation experiment as follows:

[0098] Experimental setup: All experiments in this invention were completed on the Windows operating system, using an Intel(R) Core(TM) i7-10700F CPU, an NVIDIA GeForce RTX 4070 GPU, 16GB video memory, and Python programming languages, CUDA Version: 11.1, Cu DNN Version: 8.9.7. The deep learning framework used to build the network was pytorch Version: 2.3.1, with an L1 distance loss coefficient of 100, a physical loss coefficient of 100, a perceptual loss coefficient of 10, and a gradient penalty loss coefficient of 10. The dataset used in the experiment first cropped the image to a size of 256*256 in the image preprocessing stage, set the batch size to 1, used the Adam optimizer, and used an initial learning rate of 0.0002 for the first 100 rounds, and then linearly decayed from 0.0002 to 0 for the next 100 rounds.

[0099] Determine the evaluation index: The embodiment of the present invention uses two indicators, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM), to evaluate and analyze the generated results. PSNR can measure the pixel-level difference between the generated image and the real target image. The larger the calculated value, the closer the pixel value of the generated image is to the real target image. SSIM is more consistent with the human eye's judgment of image quality in measuring image quality. It can measure the structural or brightness difference between the generated image and the real target image. The higher the value, the more similar the structure between the two images.

[0100] Specifically, a series of ablation experiments were conducted with the original Pix2PixGAN with supervised learning strategy as the benchmark. The experimental results for the short-wave high-fidelity infrared remote sensing image generation task are shown in Table 1. Experiment 1: The supervised image transfer model Pix2PixGAN was trained with default parameters and 1800 sets of images, and the index values ​​PSNR and SSIM were calculated; Experiment 2: Based on Experiment 1, a method based on deformable convolution + multi-scale feature fusion discrimination was adopted, and the evaluation index results were calculated; Experiment 3: Based on Experiment 2, a channel attention + spatial attention mechanism was designed, and the evaluation index results were calculated; Experiment 4: Based on Experiment 3, radiation spectral characteristic mapping data was introduced, and the preset loss function after design improvement was adopted, that is, the high-fidelity infrared remote sensing image generation method of coupling scene radiation mechanism and generative adversarial network proposed in the present invention was calculated, and the evaluation index results were calculated.

[0101] Table 1 Comparative experimental results for the shortwave high-fidelity infrared remote sensing image generation task

[0102]

[0103]

[0104] From the experimental results in Table 1 above, it can be seen that the high-fidelity infrared remote sensing images obtained by the method of the present invention have the best results under both SSIM and PSNR evaluation indicators, that is, the infrared remote sensing images generated by the method of the present invention have the highest fidelity and the best texture details. At the same time, the method has high accuracy and strong adaptability in the generation of high-fidelity infrared remote sensing images in short, medium and long wave bands. It proves that the method of the present invention can effectively improve the accuracy, confidence and generalization ability of the basic model.

[0105] In summary, the high-fidelity infrared remote sensing image generation method proposed in the present invention has the following technical innovations:

[0106] 1. Based on the law of scene radiation energy transmission, using the material spectral reflectance characteristics and the spectral dependence of the radiation field energy, a quantitative representation model of the radiation spectral characteristic mapping relationship between visible light remote sensing and infrared remote sensing was constructed, and it was quantitatively coupled with the final preset loss function. By using the interpretability of physical modeling and the powerful feature extraction ability of the network model, the accuracy and generalization of the pre-trained infrared image generation network were improved at the same time;

[0107] 2. Based on the Pix2PixGAN model, the design sets the fast Fourier transform process in the feature calculation process of channel attention and spatial attention. Starting from the two aspects of channel attention and spatial attention, it saves parameters and computing power. At the same time, it can extract the frequency domain features of the learning image, promote the pre-trained image generation module to learn the comprehensive texture features of the image more efficiently, and improve the generator's ability to extract features and perform transformations;

[0108] 3. Based on the Pix2PixGAN model, by designing a deformable convolution module and a multi-scale feature fusion discrimination module, the adaptability and generalization ability of the pre-trained image discrimination module to unknown changes are improved, thereby capturing the details and features of visible light remote sensing images at different scales, and improving the accuracy of the pre-trained image discrimination module;

[0109] 4. A preset loss function including adversarial loss, conditional loss, radiation spectrum characteristic loss, gradient penalty loss and feature space matching loss was designed, which effectively promoted the convergence and balance of the pre-trained infrared image generation network and improved the stability and generalization ability of the pre-trained infrared image generation network.

[0110] The method provided in the embodiment of the present invention can be applied to an electronic device. Specifically, the electronic device can be: a desktop computer, a portable computer, an intelligent mobile terminal, a server, etc., which is not limited in the embodiment of the present invention.

[0111] Based on the same inventive concept, an embodiment of the present invention also provides a high-fidelity infrared remote sensing image generation device. Figure 5 A schematic diagram of the structure of a high-fidelity infrared remote sensing image generation device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, it includes: an acquisition unit 501, a cropping unit 502 and an image generation unit 503;

[0112] The acquisition unit 501 is used to: acquire a visible light remote sensing image to be processed;

[0113] The cropping unit 502 is used to: perform image cropping processing on the visible light remote sensing image to be processed to obtain a visible light remote sensing cropped image;

[0114] The image generation unit 503 is used to: input the visible light remote sensing cropped image into a pre-trained infrared image generation network to generate a corresponding high-fidelity infrared remote sensing image; wherein the pre-trained infrared image generation network is composed of a pre-trained image generation module and a pre-trained image discrimination module; the pre-trained infrared image generation network uses visible light remote sensing image samples as input data and uses radiation spectrum characteristic mapping data as prior information, and is trained in combination with a preset loss function; the radiation spectrum characteristic mapping data is the spectral characteristic mapping relationship between the visible light remote sensing image samples and the corresponding infrared remote sensing image samples; the preset loss function is used to characterize the matching relationship between the infrared remote sensing image generation data and the target infrared remote sensing image and the radiation spectrum characteristic mapping data.

[0115] Figure 6 A structural diagram of a high-fidelity infrared remote sensing image generation device provided by an embodiment of the present invention includes: a processor 610, a storage medium 620 and a bus 630. The storage medium 620 stores machine-readable instructions executable by the processor 610. When the high-fidelity infrared remote sensing image generation device is running, the processor 610 communicates with the storage medium 620 through the bus 630, and the processor 610 executes the machine-readable instructions to perform the steps of the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0116] The storage medium may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk storage. Optionally, the storage medium may also be at least one storage device located away from the aforementioned processor.

[0117] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0118] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Instead, they are merely examples of devices and methods consistent with some aspects of the present invention.

[0119] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0120] Although the present invention is described herein in conjunction with various embodiments, in the process of implementing the claimed invention, those skilled in the art can understand and implement other changes of the above disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the term "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and the meaning of "multiple" is two or more, unless otherwise clearly and specifically limited. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0121] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A method for generating high-fidelity infrared remote sensing images, characterized in that: include: Acquire visible light remote sensing images to be processed; Performing image cropping processing on the visible light remote sensing image to be processed to obtain a visible light remote sensing cropped image; Input the visible light remote sensing cropped image into a pre-trained infrared image generation network to generate a corresponding high-fidelity infrared remote sensing image; wherein the pre-trained infrared image generation network is composed of a pre-trained image generation module and a pre-trained image discrimination module; the pre-trained infrared image generation network uses visible light remote sensing image samples as input data, and uses radiation spectrum characteristic mapping data and target infrared remote sensing images as prior information, and is trained in combination with a preset loss function; The radiation spectrum characteristic mapping data is the spectral characteristic mapping relationship between the visible light remote sensing image samples and the corresponding infrared remote sensing image samples; the preset loss function is used to characterize the matching relationship between the infrared remote sensing image generation data and the target infrared remote sensing image and the radiation spectrum characteristic mapping data.

2. The method for generating high-fidelity infrared remote sensing images according to claim 1, characterized in that: The step of performing image cropping processing on the visible light remote sensing image to be processed to obtain a visible light remote sensing cropped image includes: The visible light remote sensing image to be processed is cropped according to a preset cropping size to obtain the visible light remote sensing cropped image.

3. The method for generating high-fidelity infrared remote sensing images according to claim 1, characterized in that: The pre-trained image generation module adopts the U-Net architecture in Pix2Pix GAN, wherein the U-Net architecture in Pix2Pix GAN is provided with a fast Fourier transform process in the feature calculation process of the corresponding channel attention and spatial attention; The pre-trained image discrimination module adopts the discrimination module in Pix2Pix GAN, and the discrimination module in Pix2Pix GAN is provided with a deformable convolution module and a multi-scale feature fusion discrimination module; The deformable convolution module is arranged between the preset convolution layers of the discriminant module in the Pix2Pix GAN; The multi-scale feature fusion discrimination module is used to perform multi-scale discrimination on the infrared remote sensing image generation data and the target infrared remote sensing image.

4. The method for generating high-fidelity infrared remote sensing images according to claim 1, characterized in that: The generation process of the pre-trained infrared image generation network includes: Acquire a paired data set; the paired data set includes a plurality of sample pairs of visible light remote sensing image samples and corresponding infrared remote sensing image samples; Obtaining the spectral reflectance of each pixel of the paired data set based on inversion calculation; Based on a preset object classification method, the material of the visible light remote sensing image samples is classified to obtain infrared remote sensing image labeled samples; Performing matching processing on the infrared remote sensing image marked sample and the spectral reflectance to obtain material spectral reflectance matching data; Based on the material spectral reflectance matching data and a preset quantitative characterization model, the radiation spectral characteristic mapping data is obtained; The initial infrared image generation network is trained based on the radiation spectrum characteristic mapping data and the paired data set to obtain the pre-trained infrared image generation network; the initial infrared image generation network has the same structure as the pre-trained infrared image generation network.

5. The method for generating high-fidelity infrared remote sensing images according to claim 1, characterized in that: The step of training the initial infrared image generation network based on the radiation spectrum characteristic mapping data and the paired data set to obtain the pre-trained infrared image generation network comprises: Iteratively training the initial infrared image generation network based on the radiation spectrum characteristic mapping data, the paired data set and a preset loss function; Using the initial infrared image generation network that meets the preset stop condition as the pre-trained infrared image generation network; The preset stop condition includes: the number of iterations meets a preset iteration threshold or the value of a preset loss function continues to converge.

6. The method for generating high-fidelity infrared remote sensing images according to claim 5, characterized in that: The preset loss function is expressed as: Loss=c1L GD +c2Loss IR +c3Loss GP +c4Loss FSM ; Among them, Loss represents the preset loss function, L GD Represents the basic loss function, Loss IR Indicates the loss of radiation spectrum characteristics, Loss GP Represents the gradient penalty loss, Loss FSM represents the feature space matching loss, c1 represents the first weight value, c2 represents the second weight value, c3 represents the third weight value, and c4 represents the fourth weight value, wherein the basic loss function is the sum of the adversarial loss and the conditional loss.

7. The method for generating high-fidelity infrared remote sensing images according to claim 6, characterized in that: The radiation spectrum characteristic loss is expressed as: Where i represents the i-th paired data set, N represents the total number of paired data sets, and I mapping Represents the radiation spectrum characteristic mapping data, I input Represents visible light remote sensing image samples, G(I input ) represents the infrared remote sensing image generation data, and |·| represents taking the absolute value.

8. The method for generating high-fidelity infrared remote sensing images according to claim 4, characterized in that: The preset quantitative characterization model is expressed as: Among them, I mapping (k,λ p ,λ q ,ρ k ,m,n) means that in the paired dataset with pixel size (m,n), the spectral reflectance is ρ k The material k is from the visible light band λ p To infrared bandλ q The radiation spectrum characteristic mapping data of the paired data set; m represents the number of horizontal pixels in the paired data set, and n represents the number of vertical pixels in the paired data set; Indicates that material k is in the infrared band λ q The corresponding material spectral reflectance matching data, is the material k in the visible light band λ p The corresponding material spectral reflectance matching data; Indicates that the pixel size is (m k ,n k ) in the paired data set, the spectral reflectance is ρ k The material k in the infrared band λ q Total radiation at Indicates that the pixel size is (m k ,n k ) in the paired data set, the spectral reflectance is ρ k The material k in the visible light band λ p Total radiation at m k represents the number of horizontal pixels of material k in the paired dataset, n k represents the number of vertical pixels of material k in the paired dataset, The spectral reflectance is ρ k The material k in the infrared band λ q The total radiation at The spectral reflectance is ρ k The material k in the visible light band λ p The total radiation at In the infrared band λ q The material spectral reflectance matching data is The total radiation corresponding to the k materials is Indicates the visible light band λ p The material spectral reflectance matching data is The total radiation corresponding to the k materials.

9. A high-fidelity infrared remote sensing image generation device, characterized in that: The high-fidelity infrared remote sensing image generation device comprises: an acquisition unit, a cropping unit and an image generation unit; The acquisition unit is used to: acquire the visible light remote sensing image to be processed; The cropping unit is used to: perform image cropping processing on the visible light remote sensing image to be processed to obtain a visible light remote sensing cropped image; The image generation unit is used to: input the visible light remote sensing cropped image into a pre-trained infrared image generation network to generate a corresponding high-fidelity infrared remote sensing image; wherein the pre-trained infrared image generation network is composed of a pre-trained image generation module and a pre-trained image discrimination module; the pre-trained infrared image generation network uses visible light remote sensing image samples as input data, and uses radiation spectrum characteristic mapping data and target infrared remote sensing images as prior information, and is trained in combination with a preset loss function; the radiation spectrum characteristic mapping data is a spectral characteristic mapping relationship between the visible light remote sensing image samples and the corresponding infrared remote sensing image samples; the preset loss function is used to characterize the matching relationship between the infrared remote sensing image generation data and the target infrared remote sensing image and the radiation spectrum characteristic mapping data.

10. A high-fidelity infrared remote sensing image generation device, characterized in that: include: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the high-fidelity infrared remote sensing image generation device is running, the processor and the storage medium communicate via the bus, and the processor executes the machine-readable instructions to perform the steps of the high-fidelity infrared remote sensing image generation method as described in any one of claims 1-8.

Images