A method for synthesizing images based on multi-band features

Abstract

This invention discloses a method for synthesizing images based on multi-band features, relating to the field of image processing, to improve the performance of image fusion. The image synthesis network model used in this invention generates predicted second-band and predicted first-band images using a first generation sub-model based on mutually matching real first-band and real second-band images, respectively. Then, a second generation sub-model is used to generate a synthesized image fused with the real first-band and real second-band images. Each generation sub-model incorporates a parameter-free attention module to guide learning. For each synthesized image, an adversarial discrimination sub-model is used to perform adversarial discrimination based on the real first-band and real second-band images. This invention improves the efficiency of feature learning with minimal computational resources and enhances the clarity of the synthesized images through bilateral multi-adversarial training.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method for synthesizing images based on multi-band features. Background Technology

[0002] Single-band images (such as RGB) are easily affected by factors such as lighting conditions and climate change, making it difficult to clearly display the important features of the target in the acquired environmental images, resulting in a loss of detail. Images of different bands highlight and miss different details. Consider using images of multiple bands to synthesize a new image to highlight target features and avoid missing details, which can improve the imaging quality of the target.

[0003] The key to image synthesis is extracting details and salient target regions from images of different spectral bands and preserving both in the synthesized image. However, how to fully retain details from different spectral bands, balance the contributions of different band features to the synthesized image, and avoid detail loss are crucial issues to consider in image synthesis. Before the widespread application of deep learning in image processing, the main method was to achieve image synthesis through mathematical transformations, analyzing in the spatial or transform domains, and manually designing fusion rules; this approach is generally referred to as traditional synthesis methods.

[0004] As research progresses, the limitations of traditional fusion methods are becoming increasingly apparent. On the one hand, to ensure the operability of subsequent feature fusion, traditional synthesis methods use the same transformations to extract features from different source images. This approach does not consider the differences in features between source images, resulting in poor feature representation. On the other hand, the transformation strategies of traditional synthesis methods rely too heavily on manual design and cannot adapt to increasingly complex synthesis scenarios, leading to limited synthesis performance.

[0005] The rapid development of deep learning has, to some extent, compensated for the limitations of traditional methods: First, deep learning-based image synthesis methods can design dual-branch fusion frameworks to achieve differentiated feature extraction and extract more targeted features; second, deep learning-based image synthesis methods can integrate features using networks. Therefore, using deep learning for image synthesis is gradually becoming a development trend in image synthesis.

[0006] Furthermore, in machine learning, attention mechanisms can be introduced to make the model focus on learning key features. However, in current technologies, introducing attention mechanisms requires configuring a large number of parameters to guide the model in learning the corresponding features. On the one hand, this consumes a lot of computing resources, and on the other hand, it also requires a high degree of accuracy in configuring the parameters; otherwise, unexpected features will be learned. Summary of the Invention

[0007] The purpose of this invention is to provide a method for synthesizing images based on multi-band features, addressing all or part of the aforementioned problems, and to improve the performance of multi-band image synthesis by leveraging deep learning technology.

[0008] The technical solution adopted in this invention is as follows:

[0009] A method for synthesizing images based on multi-band features is proposed. This method utilizes a trained image synthesis network model to fuse a second-band image and a first-band image to be fused. The image synthesis network model is trained using a training set.

[0010] The image synthesis network model is based on the input of a matching real first-band image and a real second-band image:

[0011] A predicted second-band image is generated from the real first-band image using a first generation sub-model.

[0012] A predicted first-band image is generated from the real second-band image using a second generation sub-model;

[0013] A first synthetic image is generated from the real first-band image and the predicted second-band image using a third generation sub-model.

[0014] A second synthetic image is generated from the real second-band image and the predicted first-band image using a fourth generative sub-model; each generative sub-model is guided by a parameterless attention module for learning.

[0015] For each synthesized image, adversarial identification of first-band features is performed based on the real first-band image, and adversarial identification of second-band features is performed based on the real second-band image.

[0016] Furthermore, for the first synthesized image, a first discriminator sub-model is used to perform adversarial discrimination of the first band features based on the real first band image, and a second discriminator sub-model is used to perform adversarial discrimination of the second band features based on the real second band image and the predicted second band image.

[0017] For the second synthesized image, a third discriminator model is used to perform adversarial discrimination of second-band features based on the real second-band image, and a fourth discriminator model is used to perform adversarial discrimination of first-band features based on the real first-band image and the predicted first-band image.

[0018] Furthermore, the image synthesis network model stitches the real first band image with the predicted second band image and then uses the third generation sub-model to generate the first synthesized image; and stitches the real second band image with the predicted first band image and then uses the fourth generation sub-model to generate the second synthesized image.

[0019] Furthermore, both the first and second generative sub-models contain 9 convolutional blocks, 4 downsampling layers, 4 upsampling layers, one convolutional layer, and a Tanh activation function; each convolutional block contains a convolutional layer, a parameterless attention module, a normalization layer, a dropout layer, and a Leaky ReLU activation function; each downsampling layer contains a convolutional layer, a parameterless attention module, a normalization layer, and a Leaky ReLU activation function; and each upsampling layer contains an F.interpolate interpolation function and a convolutional layer.

[0020] Furthermore, both the third and fourth generative sub-models adopt a five-layer sequential structure. The first four layers each contain a transposed convolutional layer, a parameterless attention module, a normalization layer, and a Leaky ReLU activation function; the fifth layer contains a convolutional layer and a Tanh activation function.

[0021] Furthermore, each discriminator sub-model contains four convolutional layers, three normalization layers, and one fully connected layer, with each convolutional layer followed by a Leaky ReLU activation function.

[0022] Furthermore, the energy function of each neuron in the parameterless attention module is:

[0023]

[0024] in,

[0025]

[0026] In the formula, t represents the t-th target neuron, t∈[1,N], N=H×W is the total number of neurons in a channel, and x i Let w represent the i-th non-target neuron, i∈[1,N-1], w t and b t These are the weights and biases for the linear transformation of the t-th target neuron, where a1 is a set constant, and u... t and These are the mean and variance of the t-th target neuron in the corresponding channel.

[0027] Furthermore, when training the image synthesis network model using the training set, the discrimination sub-model and the generation sub-model are trained alternately, wherein,

[0028] The loss for training the first generator sub-model includes the adversarial loss between the first generator sub-model and the second discriminator sub-model, as well as the structural similarity loss between the input real first-band image and the generated predicted second-band image;

[0029] The loss for training the second generating sub-model includes the adversarial loss between the second generating sub-model and the fourth discriminator sub-model, as well as the structural similarity loss between the input real second-band image and the generated predicted first-band image;

[0030] The loss for training the third generator sub-model includes the adversarial loss between the third generator sub-model and the first and second discriminator sub-models, respectively, as well as the content loss of the generated first synthetic image relative to the real second-band image and the real first-band image.

[0031] The loss for training the fourth generator sub-model includes the adversarial loss between the fourth generator sub-model and the third and fourth discriminator sub-models, respectively, as well as the content loss of the generated second synthetic image relative to the real first-band image and the real second-band image.

[0032] The loss for training the first discriminator sub-model includes the discrimination probability of the real first band image and the discrimination probability of the first synthetic image.

[0033] The loss for training the second discriminator sub-model includes the discrimination probability of the real second-band image, the discrimination probability of the first synthetic image, and the discrimination probability of the predicted second-band image.

[0034] The loss for training the third discriminator sub-model includes the discrimination probability of the real second-band image and the discrimination probability of the second synthetic image.

[0035] The loss for training the fourth discriminator sub-model includes the discrimination probability of the real first-band image, the discrimination probability of the second synthetic image, and the discrimination probability of the predicted first-band image.

[0036] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0037] The image fusion method of the second and first bands in this invention utilizes a machine learning model for image fusion, resulting in high fusion efficiency. During the training phase, a parameter-free attention module is introduced into each generative sub-model, enabling automatic learning of key image features with minimal computational resources, thus improving learning efficiency. The image fusion model of this invention employs a bilateral generative adversarial network architecture, and multi-adversarial training enhances the accuracy of image fusion. Attached Figure Description

[0038] The present invention will be described by way of example and with reference to the accompanying drawings, wherein:

[0039] Figure 1 This is a network structure diagram of an image synthesis network model in one embodiment.

[0040] Figure 2 This is a subnetwork structure diagram of the first generated submodel in one embodiment.

[0041] Figure 3 yes Figure 2 A network structure diagram of a convolutional block in one embodiment.

[0042] Figure 4 yes Figure 2 The network structure diagram of the downsampling layer in one embodiment is shown.

[0043] Figure 5 yes Figure 2 The network structure diagram of the upsampling layer in one embodiment is shown.

[0044] Figure 6 This is a subnetwork structure diagram of the third generative sub-model in one embodiment.

[0045] Figure 7 yes Figure 6 The network structure diagram of the first layer network of the third generating sub-model in the embodiment.

[0046] Figure 8 This is a subnetwork structure diagram of the identification submodel in one embodiment. Detailed Implementation

[0047] All features disclosed in this specification, or steps in all methods or processes disclosed herein, may be combined in any way, except for mutually exclusive features and / or steps.

[0048] Any feature disclosed in this specification (including any appended claims and abstract) may be replaced by other equivalent or similar features, unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is merely one example of a series of equivalent or similar features.

[0049] A method for synthesizing images based on multi-band features is proposed. This method utilizes a trained image synthesis network model to synthesize a higher-resolution image based on the second-band features of a second-band image and the first-band features of a first-band image. The image synthesis network model adopts a generative adversarial network architecture and is trained using a training set.

[0050] Methods for obtaining the training set include:

[0051] 1) Acquire a large number of paired second-band and first-band images, with the paired second-band and first-band images matched to each other. Matching means that the second-band and first-band images are acquired within the same field of view.

[0052] 2) Divide the collected dataset into training and testing sets. For example, divide the dataset into training and testing sets in an 8:2 ratio.

[0053] 3) Preprocess all images in the dataset. Preprocessing includes operations such as image alignment, normalization, and cropping. Specifically, the matching second-band images are aligned with the first-band images, then the images are cropped to the required size and normalized.

[0054] After obtaining the training and test sets, the image synthesis network model can be trained using the training set and its performance tested using the test set.

[0055] In some embodiments, the image synthesis network model generates predicted second-band and predicted first-band images respectively using a generative sub-model based on the input, which are mutually matched real first-band and real second-band images. That is, it generates the predicted second-band image from the real first-band image and the predicted first-band image from the real second-band image. Then, it uses another generative sub-model to generate a synthesized image that is fused with the real first-band and real second-band images. Specifically, the image synthesis network model uses a first generative sub-model to generate the predicted second-band image from the real first-band image, a second generative sub-model to generate the predicted first-band image from the real second-band image; a third generative sub-model to generate the first synthesized image from the real first-band image and the predicted second-band image, and a fourth generative sub-model to generate the second synthesized image from the real second-band image and the predicted first-band image. Each generative sub-model introduces a parameterless attention module to guide learning. The parameterless attention module guides the generative sub-model to learn the weights of image features according to a configured energy function.

[0056] In some embodiments, the energy function of each neuron in the parameterless attention module designed in each generative sub-model is:

[0057]

[0058] in,

[0059]

[0060] In the formula, t represents the t-th target neuron, t∈[1,N], N=H×W is the total number of neurons in one channel, H is the height of the feature map, W is the width of the feature map, and x i Let w represent the i-th non-target neuron, i∈[1,N-1], w t and b tThese are the weights and biases for the linear transformation of the t-th target neuron, where a1 is a set constant, and u... t and It represents the mean and variance of the t-th target neuron in the corresponding channel, calculated by removing the target neuron t and the mean and variance of all other non-target neurons.

[0061] Based on the defined energy function of the t-th target neuron, it can be seen that other neurons in the same channel, when used as target neurons, also satisfy the same distribution. Therefore, all neurons in the same channel can reuse the same mean and variance, significantly reducing the overhead of repeatedly calculating the mean and variance of each neuron. According to the energy function of the t-th target neuron, the minimum energy at each neuron is expressed as:

[0062]

[0063] in, x here i This represents the i-th neuron among all neurons in the same channel. and These represent the sample mean and variance in that channel, respectively. (Through...) This indicates the importance of each neuron. The lower the value, the greater the difference between the target neuron t and other non-target neurons in the surrounding area, and the more important it is in visual processing, resulting in a higher learning weight.

[0064] Finally, the method for refining the parameterless attention module is as follows:

[0065]

[0066] In the formula, X represents the input feature map, ⊙ represents the XOR operation, and E is... In summary, across all channels and spatial dimensions (i.e., all neuron locations in each channel, with one neuron per location), This is the result of extracting the parameterless attention module.

[0067] For each synthesized image, the image synthesis network model performs adversarial discrimination based on both real first-band and real second-band images using discriminator sub-models. Therefore, for each synthesized image, two discriminator sub-models are designed, one for adversarial discrimination based on the real first-band image and the other for adversarial discrimination based on the real second-band image. Thus, the image synthesis network model contains four discriminator sub-models.

[0068] In some embodiments, for each synthetic image, adversarial discrimination based on first-band features is performed based on a real first-band image, and adversarial discrimination based on second-band features is performed based on a real second-band image. The so-called first-band features are the content features of the first-band components in the synthetic image; similarly, the second-band features are the content features of the second-band components in the synthetic image.

[0069] As mentioned earlier, the image synthesis network model contains four discriminator sub-models. In some embodiments, for the first synthesized image, the first discriminator sub-model performs adversarial discrimination of the first band features based on the real first band image, and the second discriminator sub-model performs adversarial discrimination of the second band features based on the real second band image and the predicted second band image; for the second synthesized image, the third discriminator sub-model performs adversarial discrimination of the second band features based on the real second band image, and the fourth discriminator sub-model performs adversarial discrimination of the first band features based on the real first band image and the predicted first band image.

[0070] like Figure 1 As shown, taking the first band as the visible light band and the second band as the infrared band as an example, in some embodiments, the image synthesis network model is designed to include four generative sub-models: gen_vis2ir, gen_ir2vis, gen_fuse1, and gen_fuse2. gen_vis2ir is the first generative sub-model, and gen_ir2vis is the second generative sub-model. gen_vis2ir is used to generate a predicted second band image (pseudo-IR) based on the input real first band image (RBG image), and gen_ir2vis is used to generate a predicted first band image (pseudo-RBG) based on the input real second band image (IR image). The expression for the first generative sub-model is G1(I1, z), where I... V Let I represent the input real first-band image, z represent the input random noise, and the expression for the second generator sub-model is G1(I2,z), where I... R This represents the actual second-band image. gen_fuse1 is the third generation sub-model, and gen_fuse2 is the fourth generation sub-model. gen_fuse1 is used to generate the first composite image based on the actual first-band image and the predicted second-band image, and gen_fuse2 is used to generate the second composite image based on the actual second-band image and the predicted first-band image. In this embodiment, the composite images are all represented as IF.

[0071] like Figure 1As shown, in the image synthesis network model, the real first-band image and the predicted second-band image are stitched together, and the synthesized image is generated using the third generation sub-model. Similarly, the real second-band image and the predicted first-band image are stitched together, and the synthesized image is generated using the fourth generation sub-model. The expression for the ratio of the third generation sub-model to the fourth generation sub-model is G2(I G , z), where I G This indicates a stitched image.

[0072] The image synthesis network model also includes four discriminator sub-models: dis1_vis2ir, dis1_ir2vis, dis2_vis2ir, and dis2_ir2vis. dis1_vis2ir is the first discriminator sub-model, and dis1_ir2vis is the third discriminator sub-model. dis1_vis2ir is used for adversarial discrimination against the first synthesized image (with its first-band features) based on a real first-band image, and dis1_ir2vis is used for adversarial discrimination against the first synthesized image (with its second-band features) based on a real second-band image. The expression for the first discriminator sub-model / third discriminator sub-model is D1(I F dis2_vis2ir is the second discriminator sub-model, and dis2_ir2vis is the fourth discriminator sub-model. dis2_vis2ir is used for adversarial discrimination against the second band features of the first synthetic image based on the real second band image and the predicted second band image. dis2_ir2vis is used for adversarial discrimination against the first band features of the second synthetic image based on the real first band image and the predicted first band image. The expression for the second discriminator sub-model / fourth discriminator sub-model is D2(I F ).

[0073] The image synthesis network model receives pairs of real first-band and real second-band images in batches from the training set. The real first-band image is used by the first generator sub-model gen_vis2ir to generate a predicted second-band image. The real first-band image and the generated predicted second-band image are then concatenated and fed into the third generator sub-model gen_fuse1 to generate the first synthesized image. The first synthesized image and the real first-band image are fed into the first discriminator sub-model dis1_vis2ir to determine the similarity between the first synthesized image and the real first-band image. The first synthesized image, the generated predicted second-band image, and the real second-band image are fed into the second discriminator sub-model dis2_vis2ir to determine the similarity among the three. Similarly, the input real second-band image is used to generate a predicted first-band image through the second generating sub-model gen_ir2vis. Then, the real second-band image and the generated predicted first-band image are concatenated and fed into the fourth generating sub-model gen_fuse2 to generate a second composite image. The second composite image and the real second-band image are fed into the third discriminator sub-model dis1_ir2vis to determine the similarity between the second composite image and the real second-band image. Finally, the second composite image, the generated predicted first-band image, and the real first-band image are fed into the fourth discriminator sub-model dis2_ir2vis to determine the similarity among the three.

[0074] In some embodiments, the sub-network structure of the first generative sub-model gen_vis2ir includes 9 convolutional blocks, 4 downsampling layers, 4 upsampling layers, one convolutional layer, and a Tanh activation function; each convolutional block includes a convolutional layer, a parameterless attention module, a normalization layer, a dropout layer, and a Leaky ReLU activation function; each downsampling layer includes a convolutional layer, a parameterless attention module, a normalization layer, and a Leaky ReLU activation function; and each upsampling layer includes an F.interpolate interpolation function and a convolutional layer. Figure 2 As shown, the hierarchical order in the sub-network structure is as follows: eight convolutional blocks are connected sequentially, followed by alternating layers of four downsampling and four upsampling layers, then connected to one more convolutional block, and finally a convolutional layer and a Tanh activation function. The network structure of the convolutional block is as follows. Figure 3 As shown, the network structure of the downsampling layer is as follows: Figure 4 As shown, the network structure of the upsampling layer is as follows: Figure 5 As shown, the subnetwork structure of the second generator sub-model gen_ir2vis is the same as that of the first generator sub-model gen_vis2ir.

[0075] In some embodiments, the third generative sub-model gen_fuse1 adopts a five-layer sequential structure. The first four layers each contain a transposed convolutional layer, a parameter-free attention module, a normalization layer, and a Leaky ReLU activation function; the fifth layer contains a convolutional layer and a Tanh activation function. The sub-network structure of the third generative sub-model gen_fuse1 is as follows: Figure 6 As shown, the first layer network structure of the third generative sub-model gen_fuse1 is as follows: Figure 7 As shown. The subnetwork structure of the fourth generator sub-model gen_fuse2 is the same as that of the third generator sub-model gen_fuse1.

[0076] Each discriminator sub-model has the same sub-network structure. In some embodiments, the discriminator sub-model comprises four convolutional layers, three normalization layers, and one fully connected layer, with each convolutional layer followed by a Leaky ReLU activation function. The sub-network structure of the discriminator sub-model is as follows: Figure 8 As shown.

[0077] The preceding text introduced examples of partial sub-network structures for each generator sub-model and each discriminator sub-model in an image synthesis network model. When training an image synthesis network model using a training set, the discriminator sub-model and each generator sub-model are usually trained alternately. That is, all discriminator sub-models are trained first, then all generator sub-models are trained, then all discriminator sub-models are trained again, and so on, until the model converges, that is, until the loss converges.

[0078] In some embodiments, the loss for training the first generative sub-model gen_vis2ir includes the adversarial loss between the first generative sub-model gen_vis2ir and the second discriminative sub-model dis2_vis2ir, and the structural similarity loss between the input real first-band image and the generated predicted second-band image. Since the image synthesis network structure is a bilaterally symmetric generative adversarial network, containing prediction and fusion processes for the real first-band image and the real second-band image respectively, the loss functions of the generative sub-model and the discriminative sub-model in both networks are the same. For the first generative sub-model gen_vis2ir, its loss function is defined as:

[0079]

[0080] In the formula, the first term represents the structural similarity loss between the input real visible image and the predicted second-band image, and the second term represents the adversarial loss between the first generator sub-model gen_vis2ir and the second discriminator sub-model dis2_vis2ir. K represents the batch size, and n is the sample index. This indicates that the first generator sub-model gen_vis2ir is based on the nth real first-band image. The generated predicted second-band image, The second discriminant submodel dis2_vis2ir represents the probability of discriminating the predicted second-band image, and ||·||1 represents the L1 norm.

[0081] Similarly, the loss for training the second generator sub-model gen_ir2vis includes the adversarial loss between the second generator sub-model gen_ir2vis and the fourth discriminator sub-model dis2_ir2vis, as well as the structural similarity loss between the input real second-band image and the generated predicted first-band image.

[0082] The loss function of gen_ir2vis is:

[0083]

[0084] In the formula, This indicates that the second generator sub-model gen_ir2vis is based on the nth real second-band image. The generated predicted first band image, This indicates that the fourth discriminator sub-model, dis2_ir2vis, predicts the probability of discriminating the light image.

[0085] The loss for training the third generator sub-model gen_fuse1 includes the adversarial loss between the third generator sub-model gen_fuse1 and the first discriminator sub-model dis1_vis2ir and the second discriminator sub-model dis2_vis2ir, respectively, as well as the content loss of the generated first synthetic image relative to the real second-band image and the real first-band image.

[0086] For the third generator sub-model gen_fuse1, its loss function is defined as:

[0087]

[0088] In the formula, ε, a2, ψ, c1, and c2 are constants, and H×W represents the number of neurons in one channel. This represents the first synthesized image corresponding to the nth sample. This represents the discrimination probability of the first discriminator sub-model dis1_vis2ir for the first synthesized image. This represents the discrimination probability of the second discriminator sub-model dis2_vis2ir against the first synthesized image. To represent the change, ||·|| F This indicates the calculation of the F-norm.

[0089] Similarly, the loss for training the fourth generator sub-model includes the adversarial loss between the fourth generator sub-model gen_fuse2 and the third discriminator sub-model dis1_ir2vis and the fourth discriminator sub-model dis2_ir2vis, respectively, as well as the content loss of the generated second synthetic image relative to the real first band image and the real second band image.

[0090] For the fourth generator sub-model gen_fuse2, its loss function is defined as:

[0091]

[0092] For the fourth generator sub-model gen_fuse2, here... This represents the second synthesized image corresponding to the nth sample.

[0093] The loss for training the first discriminator sub-model dis1_vis2ir includes the discrimination probability of the real first band image and the discrimination probability of the first synthetic image.

[0094] Taking the discriminative sub-model dis1_vis2ir as an example, its loss function is defined as:

[0095]

[0096] Wherein, the first term represents the classification result (i.e., the discrimination probability) of the first synthetic image, and the second term represents the classification result of the real first band image. In the formula, This represents the probability of the first discriminator sub-model, dis1_vis2ir, discriminating the nth real first-band image. This represents the discrimination probability of the first discriminator sub-model dis1_vis2ir for the nth first synthetic image, where a3 and a4 are both set constants.

[0097] Similarly, the loss for training the third discriminator sub-model dis1_ir2vis includes the discrimination probability of the real second-band image and the discrimination probability of the second synthetic image.

[0098] For the third discriminant sub-model, dis1_ir2vis, its loss function is defined as:

[0099]

[0100] This represents the discrimination probability of the third discriminator sub-model dis1_ir2vis for the nth real second-band image. For the third discriminator sub-model dis1_ir2vis, here... This represents the discrimination probability of the third discriminator sub-model dis1_ir2vis for the nth second synthetic image.

[0101] The loss for training the second discriminator sub-model dis2_vis2ir includes the discrimination probability of the real second-band image, the discrimination probability of the first synthetic image, and the discrimination probability of the predicted second-band image.

[0102] Taking the second discriminator sub-model, dis2_vis2ir, as an example, its loss function is defined as:

[0103]

[0104] Wherein, the first term represents the classification result of the real second-band image, the second term represents the classification result of the first synthetic image, and the third term represents the classification result of the predicted second-band image. In the formula, This represents the discrimination probability of the second discriminator sub-model dis2_vis2ir for the nth real second-band image. This represents the discrimination probability of the second discriminator sub-model dis2_vis2ir for the nth first synthetic image. This represents the discrimination probability of the second discriminator sub-model dis2_vis2ir for the nth predicted second-band image, i.e., the discrimination probability of the second discriminator sub-model dis2_vis2ir for... The discrimination probability is defined by a5, a6, and a7, which are all set constants.

[0105] Similarly, the loss for training the fourth discriminator sub-model dis2_ir2vis includes the discrimination probability of the real first band image, the discrimination probability of the second synthetic image, and the discrimination probability of the predicted first band image.

[0106] For the fourth discriminant sub-model, dis2_ir2vis, its loss function is defined as:

[0107]

[0108] In the formula, This represents the discrimination probability of the fourth discrimination sub-model dis2_ir2vis for the nth real first-band image. This represents the discrimination probability of the fourth discrimination sub-model dis2_ir2vis for the nth second synthetic image. This represents the discrimination probability of the fourth discrimination sub-model dis2_ir2vis for the nth predicted first band image, i.e., the discrimination probability of the fourth discrimination sub-model dis2_ir2vis for... The probability of identification.

[0109] This invention is not limited to the specific embodiments described above. The invention extends to any new feature or combination disclosed in this specification, as well as any new method or process step or combination disclosed herein.

Claims

1. A method for synthesizing images based on multi-band features, wherein the method utilizes a trained image synthesis network model to fuse a second-band image and a first-band image to be fused; characterized in that, The image synthesis network model is trained using a training set, wherein... The image synthesis network model, based on the input matching real first-band images and real second-band images, uses a first generation sub-model to generate a predicted second-band image from the real first-band images, a second generation sub-model to generate a predicted first-band image from the real second-band images, a third generation sub-model to generate a first synthesized image from the real first-band images and the predicted second-band images, and a fourth generation sub-model to generate a second synthesized image from the real second-band images and the predicted first-band images. Each generation sub-model is guided by a parameterless attention module. For each synthesized image, adversarial identification of first-band features is performed based on the real first-band image, and adversarial identification of second-band features is performed based on the real second-band image.

2. The method for synthesizing images based on multi-band features as described in claim 1, characterized in that, For the first synthesized image, a first discrimination sub-model is used to perform adversarial discrimination of the first band features based on the real first band image, and a second discrimination sub-model is used to perform adversarial discrimination of the second band features based on the real second band image and the predicted second band image. For the second synthesized image, a third discriminator model is used to perform adversarial discrimination of second-band features based on the real second-band image, and a fourth discriminator model is used to perform adversarial discrimination of first-band features based on the real first-band image and the predicted first-band image.

3. The method for synthesizing images based on multi-band features as described in claim 2, characterized in that, The image synthesis network model stitches the real first band image with the predicted second band image and then uses the third generation sub-model to generate a first synthesized image. After stitching the real second band image with the predicted first band image, it uses the fourth generation sub-model to generate a second synthesized image.

4. The method for synthesizing images based on multi-band features as described in claim 1, characterized in that, Both the first and second generator sub-models contain 9 convolutional blocks, 4 downsampling layers, 4 upsampling layers, one convolutional layer, and a Tanh activation function. Each convolutional block contains a convolutional layer, a parameterless attention module, a normalization layer, a dropout layer, and a Leaky ReLU activation function. Each downsampling layer contains a convolutional layer, a parameterless attention module, a normalization layer, and a Leaky ReLU activation function. Each upsampling layer contains an F.interpolate interpolation function and a convolutional layer.

5. The method for synthesizing images based on multi-band features as described in claim 1, characterized in that, Both the third and fourth generator sub-models adopt a five-layer sequential structure. The first four layers each contain a transposed convolutional layer, a parameter-free attention module, a normalization layer, and a Leaky ReLU activation function; the fifth layer contains a convolutional layer and a Tanh activation function.

6. The method for synthesizing images based on multi-band features as described in claim 2, characterized in that, Each discriminant sub-model contains four convolutional layers, three normalization layers, and one fully connected layer. Each convolutional layer is followed by a Leaky ReLU activation function.

7. The method for synthesizing images based on multi-band features as described in claim 1 or 4, characterized in that, The energy function of each neuron in the parameterless attention module is: in, In the formula, t represents the t-th target neuron, t∈[1,N], N=H×W is the total number of neurons in a channel, and x i Let w represent the i-th non-target neuron, i∈[1,N-1], w t and b t These are the weights and biases for the linear transformation of the t-th target neuron, where a1 is a set constant, and u... t and These are the mean and variance of the t-th target neuron in the corresponding channel.

8. The method for synthesizing images based on multi-band features as described in claim 2, characterized in that, When training the image synthesis network model using the training set, the discrimination sub-model and the generation sub-model are trained alternately, wherein, The loss for training the first generator sub-model includes the adversarial loss between the first generator sub-model and the second discriminator sub-model, as well as the structural similarity loss between the input real first-band image and the generated predicted second-band image; The loss for training the second generating sub-model includes the adversarial loss between the second generating sub-model and the fourth discriminator sub-model, as well as the structural similarity loss between the input real second-band image and the generated predicted first-band image; The loss for training the third generator sub-model includes the adversarial loss between the third generator sub-model and the first and second discriminator sub-models, respectively, as well as the content loss of the generated first synthetic image relative to the real second-band image and the real first-band image. The loss for training the fourth generator sub-model includes the adversarial loss between the fourth generator sub-model and the third and fourth discriminator sub-models, respectively, as well as the content loss of the generated second synthetic image relative to the real first-band image and the real second-band image. The loss for training the first discriminator sub-model includes the discrimination probability of the real first band image and the discrimination probability of the first synthetic image. The loss for training the second discriminator sub-model includes the discrimination probability of the real second-band image, the discrimination probability of the first synthetic image, and the discrimination probability of the predicted second-band image. The loss for training the third discriminator sub-model includes the discrimination probability of the real second-band image and the discrimination probability of the second synthetic image. The loss for training the fourth discriminator sub-model includes the discrimination probability of the real first-band image, the discrimination probability of the second synthetic image, and the discrimination probability of the predicted first-band image.

Images