Method for synthesizing image based on multiband features

Abstract

The invention discloses a method for synthesizing an image based on multiband features, relates to the field of image processing, and is used for improving the performance of image fusion. An image synthesis network model used in the invention generates a predicted second wave band image and a predicted first wave band image by using a first generation sub-model according to an input real first wave band image and a real second wave band image which are matched with each other; respectively utilizing a second generation sub-model to generate a composite image fused with the real first waveband image and the real second waveband image; each generation sub-model introduces a parameter-free attention module to guide learning; and carrying out confrontation identification on each composite image based on the real first waveband image and the real second waveband image by using an identification sub-model. According to the method, the learning efficiency of the features is improved by only needing a small number of operation resources, and the definition of the synthetic image is improved by adopting bilateral multi-adversarial training.

Description

Technical Field

[0001] The invention relates to the field of image processing, and in particular to a method for synthesizing images based on multi-band features. Background Art

[0002] Single-band images (such as RGB) are easily affected by factors such as lighting conditions and climate change, which makes it impossible to clearly display the important features of the target in the collected environmental images, resulting in missing details. Images of different bands highlight and miss different details. Considering using images of multiple bands to synthesize new images, highlighting the target features and avoiding missing details can improve the imaging quality of the target.

[0003] The key to image synthesis is how to extract the details and salient target areas of images of different bands and keep them in the synthesized image. However, how to fully retain the details of images of different bands, balance the contribution of different band features to the synthesized image, and avoid detail loss are issues that need to be considered in synthesizing images. Before deep learning was widely used in processing, the main method was to achieve image synthesis through mathematical transformation, analyze in the spatial domain or transform domain, and manually design fusion rules. This method is usually called the traditional synthesis method.

[0004] As scientific research progresses, the limitations of traditional fusion methods become increasingly apparent. On the one hand, in order to ensure the operability of subsequent feature fusion, traditional synthesis methods use the same transformation to extract features from different source images. This method does not take into account the differential characteristics of the source images, and the extracted features have poor expressiveness. On the other hand, the transformation strategy of traditional synthesis methods relies too much on manual design and cannot adapt to increasingly complex synthesis scenes, resulting in limited synthesis performance.

[0005] The rapid development of deep learning has made up for the limitations of traditional methods to a certain extent: first, the image synthesis method based on deep learning can design a dual-branch fusion framework to achieve differentiated feature extraction and extract more targeted features; second, the image synthesis method based on deep learning can use the network to integrate features. Therefore, using deep learning for image synthesis has gradually become a development trend of image synthesis.

[0006] In addition, in machine learning, an attention mechanism can be introduced to make the model focus on learning key features. However, the introduction of the attention mechanism in the prior art requires the configuration of more parameters to guide the model to learn the corresponding features. On the one hand, it consumes more computing resources, and on the other hand, it also has high requirements for the accuracy of the configured parameters, otherwise unexpected features will be learned. Summary of the invention

[0007] The purpose of the present invention is to provide a method for synthesizing graphics based on multi-band features to improve the performance of multi-band image synthesis by means of deep learning technology in order to address all or part of the above-mentioned problems.

[0008] The technical solution adopted by the present invention is as follows:

[0009] A method for synthesizing graphics based on multi-band features, the method uses a trained image synthesis network model to fuse a second band image to be fused and a first band image; the image synthesis network model is trained using a training set, wherein:

[0010] The image synthesis network model is based on the input mutually matching real first band image and real second band image:

[0011] Generate a predicted second band image from the real first band image using a first generation sub-model,

[0012] Generate a predicted first band image from the real second band image using a second generation sub-model;

[0013] generating a first synthetic image from the real first band image and the predicted second band image using a third generation sub-model,

[0014] Using a fourth generation sub-model to generate a second composite image from the real second band image and the predicted first band image; each generation sub-model uses a parameter-free attention module to guide learning;

[0015] For each synthetic image, adversarial identification of first band features is performed based on the real first band image, and adversarial identification of second band features is performed based on the real second band image.

[0016] Further, for the first synthetic image, a first identification sub-model is used to perform adversarial identification of first band features based on the real first band image, and a second identification sub-model is used to perform adversarial identification of second band features based on the real second band image and the predicted second band image;

[0017] For the second synthetic image, the third identification sub-model is used to perform adversarial identification of the second band features based on the real second band image, and the fourth identification sub-model is used to perform adversarial identification of the first band features based on the real first band image and the predicted first band image.

[0018] Furthermore, the image synthesis network model splices the real first band image with the predicted second band image, and then uses the third generation sub-model to generate a first synthetic image, and splices the real second band image with the predicted first band image, and then uses the fourth generation sub-model to generate a second synthetic image.

[0019] Furthermore, the first generation sub-model and the second generation sub-model both include 9 convolution blocks, 4 downsampling layers, 4 upsampling layers, a convolution layer and a Tanh activation function; the convolution block includes a convolution layer, a parameter-free attention module, a normalization layer, a dropout layer and a Leaky ReLU activation function; the downsampling layer includes a convolution layer, a parameter-free attention module, a normalization layer and a Leaky ReLU activation function; the upsampling layer includes an F.interpolate interpolation function and a convolution layer.

[0020] Furthermore, the third generation sub-model and the fourth generation sub-model both adopt a five-layer sequential structure, the first four layers each include a transposed convolution layer, a parameter-free attention module, a normalization layer and a Leaky ReLU activation function; the fifth layer includes a convolution layer and a Tanh activation function.

[0021] Furthermore, each discriminator model contains four convolutional layers, three normalization layers and one fully connected layer, and each convolutional layer is followed by a Leaky ReLU activation function.

[0022] Furthermore, the energy function of each neuron in the parameter-free attention module is:

[0023]

[0024] in,

[0025]

[0026] Where t represents the tth target neuron, t∈[1,N], N=H×W is the total number of neurons on a channel, and x i represents the i-th non-target neuron, i∈[1,N-1], w t and b t is the weight and bias of the linear transformation of the t-th target neuron, a1 is the set constant, u t and is the mean and variance of the t-th target neuron in the corresponding channel.

[0027] Furthermore, when the image synthesis network model is trained using the training set, each identification sub-model and each generation sub-model are trained alternately, wherein:

[0028] The loss of training the first generative sub-model includes the adversarial loss between the first generative sub-model and the second discriminative sub-model, and the structural similarity loss between the input true first band image and the generated predicted second band image;

[0029] The loss of training the second generative sub-model includes the adversarial loss between the second generative sub-model and the fourth discriminative sub-model, and the structural similarity loss between the input real second band image and the generated predicted first band image;

[0030] The loss of training the third generative sub-model comprises adversarial losses between the third generative sub-model and the first discriminative sub-model and the second discriminative sub-model, respectively, and content losses of the generated first synthetic image relative to the real second band image and the real first band image;

[0031] The loss of training the fourth generative sub-model comprises adversarial losses between the fourth generative sub-model and the third discriminative sub-model and the fourth discriminative sub-model, respectively, and content losses of the generated second synthetic image relative to the true first band image and the true second band image;

[0032] The loss of training the first identification sub-model includes the identification probability of the real first band image and the identification probability of the first synthetic image;

[0033] The loss of training the second identification sub-model includes the identification probability of the real second band image, the identification probability of the first synthetic image, and the identification probability of the predicted second band image;

[0034] The loss of training the third identification sub-model includes the identification probability of the real second band image and the identification probability of the second synthetic image;

[0035] The loss of training the fourth discriminator model includes the discrimination probability of the real first band image, the discrimination probability of the second synthetic image, and the discrimination probability of the predicted first band image.

[0036] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0037] The second band and first band image fusion method of the present invention uses a machine learning model to perform image fusion, and the fusion efficiency is high. In the training stage, a parameter-free attention module is introduced into each generation sub-model, so that the key features of the image can be automatically focused on with only a small amount of computing resources, thereby improving the learning efficiency. The image fusion model of the present invention adopts a bilateral generative adversarial network architecture, and multiple adversarial training improves the accuracy of image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present invention will now be described by way of example with reference to the accompanying drawings, in which:

[0039] Figure 1 is a network structure diagram of an image synthesis network model in one embodiment.

[0040] Figure 2 FIG. 4 is a sub-network structure diagram of the first generation sub-model in one embodiment.

[0041] Figure 3 yes Figure 2 A network structure diagram of a convolution block in an embodiment.

[0042] Figure 4 yes Figure 2 A network structure diagram of a downsampling layer in an embodiment.

[0043] Figure 5 yes Figure 2 A network structure diagram of an upsampling layer in an embodiment.

[0044] Figure 6 4 is a sub-network structure diagram of the third generation sub-model in one embodiment.

[0045] Figure 7 yes Figure 6 A network structure diagram of the first layer network of the third generation sub-model in the embodiment.

[0046] Figure 8 FIG. 4 is a subnetwork structure diagram of an identification submodel in one embodiment. DETAILED DESCRIPTION

[0047] All features disclosed in this specification, or steps in all methods or processes disclosed, except mutually exclusive features and / or steps, can be combined in any manner.

[0048] Any feature disclosed in this specification (including any additional claims and abstract), unless otherwise stated, may be replaced by other equivalent or alternative features having similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

[0049] A method for synthesizing images based on multi-band features, the method uses a trained image synthesis network model to synthesize a higher definition synthetic image according to the second band features of the second band image and the first band features of the first band image. The image synthesis network model adopts a generative adversarial network as the architecture and is trained using a training set.

[0050] The methods for obtaining the training set include:

[0051] 1) A large number of pairs of second-band images and first-band images are acquired, and the pairs of second-band images and first-band images are matched with each other. The so-called mutual matching means that the second-band images and first-band images are acquired in the same field of view.

[0052] 2) Divide the collected data set into a training set and a test set. For example, divide the data set into a training set and a test set in a ratio of 8:2.

[0053] 3) Preprocess all images in the data set. Preprocessing includes operations such as image alignment, normalization, and cropping. That is, align the matching second-band image and first-band image, then crop the image size to the required size and perform normalization.

[0054] After obtaining the training set and the test set, the image synthesis network model can be trained using the training set, and the performance of the trained image synthesis network model can be tested using the test set.

[0055] In some embodiments, the image synthesis network model generates a predicted second band image and a predicted first band image using a generative sub-model respectively according to the input mutually matching real first band image and real second band image. That is, the predicted second band image is generated according to the real first band image, and the predicted first band image is generated according to the real second band image. Then another generative sub-model is used to generate a synthetic image fused with the real first band image and the real second band image. Specifically, the image synthesis network model generates a predicted second band image from the real first band image using the first generative sub-model, and generates a predicted first band image from the real second band image using the second generative sub-model; generates a first synthetic image from the real first band image and the predicted second band image using the third generative sub-model, and generates a second synthetic image from the real second band image and the predicted first band image using the fourth generative sub-model. Each generative sub-model introduces a parameter-free attention module to guide learning, and the parameter-free attention module guides the weights learned by the generative sub-model for image features according to the configured energy function.

[0056] In some embodiments, the energy function of each neuron in the parameter-free attention module designed in each generator model is:

[0057]

[0058] in,

[0059]

[0060] Where t represents the tth target neuron, t∈[1,N], N=H×W is the total number of neurons on a channel, H is the height of the feature map, W is the width of the feature map, and x i represents the i-th non-target neuron, i∈[1,N-1], w t and b tis the weight and bias of the linear transformation of the t-th target neuron, a1 is the set constant, u t and is the mean and variance of the t-th target neuron in the corresponding channel, calculated by removing the mean and variance of all other non-target neurons after removing the target neuron t.

[0061] According to the defined energy function of the t-th target neuron, it can be seen that other neurons on the same channel also satisfy the same distribution when they are used as target neurons. Therefore, all neurons on the same channel can reuse the same mean and variance, which greatly reduces the overhead of repeatedly calculating the mean and variance of each neuron. Based on the energy function of the t-th target neuron above, the minimum energy at each neuron is expressed as:

[0062]

[0063] in, Here x i It represents the i-th neuron among all neurons in the same channel. and They represent the sample mean and variance in the channel respectively. represents the importance of each neuron, The lower it is, the greater the difference between the target neuron t and other surrounding non-target neurons, the more important it is in visual processing, and the higher the learning weight.

[0064] Finally, the refinement method of the parameter-free attention module is:

[0065]

[0066] In the formula, X represents the input feature map, ⊙ represents the XOR operation, and E is Aggregated across all channels and spatial dimensions (i.e., all neuron locations in each channel, with one neuron per location), This is the result of refining the parameter-free attention module.

[0067] For each synthetic image, the image synthesis network model uses the discrimination sub-model to perform adversarial discrimination based on the real first band image and the real second band image. For each synthetic image, two discrimination sub-models are designed, which are respectively based on the real first band image for adversarial discrimination and based on the real second band image for adversarial discrimination. The image synthesis network model contains four discrimination sub-models.

[0068] In some embodiments, for each synthetic image, adversarial identification of first band features is performed based on the real first band image, and adversarial identification of second band features is performed based on the real second band image. The so-called first band features are the content features of the first band component in the synthetic image; similarly, the second band features are the content features of the second band component in the synthetic image.

[0069] As mentioned above, the image synthesis network model includes four identification sub-models. In some embodiments, for the first synthetic image, the first identification sub-model is used to perform adversarial identification of the first band feature based on the real first band image, and the second identification sub-model is used to perform adversarial identification of the second band feature based on the real second band image and the predicted second band image; for the second synthetic image, the third identification sub-model is used to perform adversarial identification of the second band feature based on the real second band image, and the fourth identification sub-model is used to perform adversarial identification of the first band feature based on the real first band image and the predicted first band image.

[0070] like Figure 1 As shown, taking the first band as the visible light band and the second band as the infrared band as an example, in some embodiments, the architecture of the image synthesis network model is designed to include four generation sub-models gen_vis2ir, gen_ir2vis, gen_fuse1 and gen_fuse2, wherein gen_vis2ir is the first generation sub-model, gen_ir2vis is the second generation sub-model, gen_vis2ir is used to generate a predicted second band image (fake IR) according to the input real first band image (RBG image), gen_ir2vis is used to generate a predicted first band image (fake RBG) according to the input real second band image (IR image), and the expression of the first generation sub-model is G1(I1, z), wherein I V represents the real first band image of the input, z represents the random noise of the input, and the expression of the second generation submodel is G1(I2,z), I R Represents the real second band image. gen_fuse1 is the third generation sub-model, gen_fuse2 is the fourth generation sub-model, gen_fuse1 is used to generate a first synthetic image according to the real first band image and the predicted second band image, gen_fuse2 is used to generate a second synthetic image according to the real second band image and the predicted first band image, in this embodiment, the synthetic images are all represented as IF.

[0071] like Figure 1As shown, in the image synthesis network model, the third generation sub-model is used to generate a synthetic image after the real first band image is spliced ​​with the predicted second band image, and the fourth generation sub-model is used to generate a synthetic image after the real second band image is spliced ​​with the predicted first band image. The expression of the third generation sub-model / the fourth generation sub-model is G2(I G , z), where I G Represents a stitched image.

[0072] The image synthesis network model also includes four identification sub-models dis1_vis2ir, dis1_ir2vis, dis2_vis2ir and dis2_ir2vis, where dis1_vis2ir is the first identification sub-model, dis1_ir2vis is the third identification sub-model, dis1_vis2ir is used to perform adversarial identification on the first synthetic image (the first band feature) based on the real first band image, and dis1_ir2vis is used to perform adversarial identification on the first synthetic image (the second band feature) based on the real second band image. The expression of the first identification sub-model / the third identification sub-model is D1(I F ). dis2_vis2ir is the second identification sub-model, dis2_ir2vis is the fourth identification sub-model, dis2_vis2ir is used to perform adversarial identification on the first synthetic image (the second band feature) based on the real second band image and the predicted second band image, dis2_ir2vis is used to perform adversarial identification on the second synthetic image (the first band feature) based on the real first band image and the predicted first band image, and the expression of the second identification sub-model / the fourth identification sub-model is D2(I F ).

[0073] Input pairs of real first-band images and real second-band images from the training set in batches into the image synthesis network model, input the real first-band image to generate the predicted second-band image through the first generation sub-model gen_vis2ir, then splice the real first-band image and the generated predicted second-band image and send them to the third generation sub-model gen_fuse1 to generate the first synthetic image, send the first synthetic image and the real first-band image to the first identification sub-model dis1_vis2ir to judge the similarity between the first synthetic image and the real first-band image, and send the first synthetic image, the generated predicted second-band image and the real second-band image to the second identification sub-model dis2_vis2ir to judge the similarity between the three. Similarly, the real second-band image is input and the predicted first-band image is generated through the second generation sub-model gen_ir2vis. Then the real second-band image and the generated predicted first-band image are spliced ​​and sent to the fourth generation sub-model gen_fuse2 to generate a second synthetic image. The second synthetic image and the real second-band image are sent to the third identification sub-model dis1_ir2vis to judge the similarity between the second synthetic image and the real second-band image. The second synthetic image, the generated predicted first-band image and the real first-band image are sent to the fourth identification sub-model dis2_ir2vis to judge the similarity between the three.

[0074] In some embodiments, the subnetwork structure of the first generation submodel gen_vis2ir includes 9 convolution blocks, 4 downsampling layers, 4 upsampling layers, a convolution layer and a Tanh activation function; the convolution block includes a convolution layer, a parameter-free attention module, a normalization layer, a dropout layer and a Leaky ReLU activation function; the downsampling layer includes a convolution layer, a parameter-free attention module, a normalization layer and a Leaky ReLU activation function; the upsampling layer includes an F.interpolate interpolation function and a convolution layer. Figure 2 As shown in the figure, the order of the layers in the sub-network structure is: 8 convolution blocks are connected in sequence, then 4 downsampling layers and 4 upsampling layers are set alternately, then 1 convolution block is connected, and finally a convolution layer and Tanh activation function are set. The network structure of the convolution block is shown in Figure 3 As shown, the network structure of the downsampling layer is as follows Figure 4 As shown, the network structure of the upsampling layer is as follows Figure 5 As shown in FIG. 1 , the subnetwork structure of the second generation submodel gen_ir2vis is the same as that of the first generation submodel gen_vis2ir.

[0075] In some embodiments, the third generation sub-model gen_fuse1 adopts a five-layer sequential structure, the first four layers of the network each include a transposed convolution layer, a parameter-free attention module, a normalization layer and a Leaky ReLU activation function; the fifth layer includes a convolution layer and a Tanh activation function. The sub-network structure of the third generation sub-model gen_fuse1 is as follows Figure 6 As shown, the first-layer network structure of the third generation sub-model gen_fuse1 is as follows Figure 7 The sub-network structure of the fourth generation sub-model gen_fuse2 is the same as that of the third generation sub-model gen_fuse1.

[0076] The sub-network structure of each identification sub-model is the same. In some embodiments, the identification sub-model includes four convolutional layers, three normalization layers, and one fully connected layer, and each convolutional layer is followed by a Leaky ReLU activation function. The sub-network structure of the identification sub-model is as follows: Figure 8 shown.

[0077] The foregoing article introduces an embodiment of a partial sub-network structure of each generator sub-model and each discriminator sub-model in an image synthesis network model. When training an image synthesis network model through a training set, each discriminator sub-model and each generator sub-model are usually trained alternately, that is, all the discriminator sub-models are trained first, then all the generator sub-models are trained, and then all the discriminator sub-models are trained again, and so on, until the model converges, that is, until the loss converges.

[0078] In some embodiments, the loss of training the first generation sub-model gen_vis2ir includes the adversarial loss between the first generation sub-model gen_vis2ir and the second identification sub-model dis2_vis2ir, and the structural similarity loss between the input real first band image and the generated predicted second band image. Since the image synthesis network structure is a bilaterally symmetric generative adversarial network, which includes the prediction and fusion processes of the real first band image and the real second band image respectively, the loss functions of the generation sub-model and the identification sub-model in the two-side network are the same. For the first generation sub-model gen_vis2ir, its loss function is defined as:

[0079]

[0080] Where, the first term represents the structural similarity loss between the input real visible image and the predicted second band image, and the second term represents the adversarial loss between the first generation sub-model gen_vis2ir and the second identification sub-model dis2_vis2ir. K represents the sample batch size batchsize, n is the sample number, Indicates that the first generation sub-model gen_vis2ir is based on the nth real first band image The generated predicted second band image, It represents the identification probability of the second identification sub-model dis2_vis2ir for the predicted second band image, and ||·||1 represents the L1 norm.

[0081] Similarly, the loss of training the second generation sub-model gen_ir2vis includes the adversarial loss between the second generation sub-model gen_ir2vis and the fourth identification sub-model dis2_ir2vis, as well as the structural similarity loss between the input real second band image and the generated predicted first band image.

[0082] The loss function of gen_ir2vis is:

[0083]

[0084] In the formula, Indicates that the second generation sub-model gen_ir2vis is based on the nth real second band image The generated predicted first band image, It indicates that the fourth identification sub-model dis2_ir2vis can predict the identification probability of the light image.

[0085] The loss of training the third generation sub-model gen_fuse1 includes the adversarial loss between the third generation sub-model gen_fuse1 and the first identification sub-model dis1_vis2ir and the second identification sub-model dis2_vis2ir, respectively, and the content loss of the generated first synthetic image relative to the real second band image and the real first band image.

[0086] For the third generation sub-model gen_fuse1, its loss function is defined as:

[0087]

[0088] In the formula, ε, a2, ψ, c1 and c2 are constants, H×W represents the number of neurons on a channel, represents the first synthetic image corresponding to the nth sample, represents the identification probability of the first identification sub-model dis1_vis2ir for the first synthetic image, represents the identification probability of the second identification sub-model dis2_vis2ir for the first synthetic image, represents the amount of change, ||·|| F It means to find the F norm.

[0089] Similarly, the loss of training the fourth generation sub-model includes the adversarial loss between the fourth generation sub-model gen_fuse2 and the third identification sub-model dis1_ir2vis and the fourth identification sub-model dis2_ir2vis, and the content loss of the generated second synthetic image relative to the real first band image and the real second band image.

[0090] For the fourth generation sub-model gen_fuse2, its loss function is defined as:

[0091]

[0092] For the fourth generation sub-model gen_fuse2, here Represents the second synthetic image corresponding to the nth sample.

[0093] The loss of training the first identification sub-model dis1_vis2ir includes the identification probability of the real first band image and the identification probability of the first synthetic image.

[0094] Taking the identification sub-model dis1_vis2ir as an example, its loss function is defined as:

[0095]

[0096] The first term represents the classification result of the first synthetic image (i.e., identification probability), and the second term represents the classification result of the real first band image. It represents the identification probability of the first identification sub-model dis1_vis2ir for the nth real first band image. It represents the identification probability of the first identification sub-model dis1_vis2ir for the nth first synthetic image, and a3 and a4 are both set constants.

[0097] Similarly, the loss of training the third identification sub-model dis1_ir2vis includes the identification probability of the real second band image and the identification probability of the second synthetic image.

[0098] For the third identification sub-model dis1_ir2vis, its loss function is defined as:

[0099]

[0100] Represents the identification probability of the third identification sub-model dis1_ir2vis for the nth real second band image. For the third identification sub-model dis1_ir2vis, here Represents the identification probability of the third identification sub-model dis1_ir2vis for the nth second synthetic image.

[0101] The loss of training the second identification sub-model dis2_vis2ir includes the identification probability of the real second band image, the identification probability of the first synthetic image, and the identification probability of the predicted second band image.

[0102] Taking the second identification sub-model dis2_vis2ir as an example, its loss function is defined as:

[0103]

[0104] The first term represents the classification result of the real second band image, the second term represents the classification result of the first synthetic image, and the third term represents the classification result of the predicted second band image. It represents the identification probability of the second identification sub-model dis2_vis2ir for the nth real second band image. represents the identification probability of the second identification sub-model dis2_vis2ir for the nth first synthetic image, It represents the identification probability of the second identification sub-model dis2_vis2ir for the nth predicted second band image, that is, the second identification sub-model dis2_vis2ir The identification probability, a5, a6 and a7 are all set constants.

[0105] Similarly, the loss of training the fourth identification sub-model dis2_ir2vis includes the identification probability of the real first band image, the identification probability of the second synthetic image, and the identification probability of the predicted first band image.

[0106] For the fourth identification sub-model dis2_ir2vis, its loss function is defined as:

[0107]

[0108] In the formula, It represents the identification probability of the fourth identification sub-model dis2_ir2vis for the nth real first band image. represents the identification probability of the fourth identification sub-model dis2_ir2vis for the nth second synthetic image, It represents the identification probability of the fourth identification sub-model dis2_ir2vis for the nth predicted first band image, that is, the fourth identification sub-model dis2_ir2vis The probability of identification.

[0109] The present invention is not limited to the above-mentioned specific embodiments, but extends to any new features or any new combination disclosed in this specification, as well as any new method or process steps or any new combination disclosed.

Claims

1. A method for synthesizing images based on multi-band features, the method uses a trained image synthesis network model to fuse the second band image to be fused and the first band image; characterized in that: The image synthesis network model is trained using a training set, where: The image synthesis network model generates a predicted second-band image from the real first-band image and the real second-band image that match each other according to the input, using the first generation sub-model, and generates a predicted first-band image from the real second-band image using the second generation sub-model; generates a first synthetic image from the real first-band image and the predicted second-band image using the third generation sub-model, and generates a second synthetic image from the real second-band image and the predicted first-band image using the fourth generation sub-model; each generation sub-model uses a parameter-free attention module to guide learning; For each synthetic image, adversarial identification of first band features is performed based on the real first band image, and adversarial identification of second band features is performed based on the real second band image.

2. The method for synthesizing images based on multi-band features according to claim 1, characterized in that: For the first synthetic image, using a first identification sub-model to perform adversarial identification of first band features based on the real first band image, and using a second identification sub-model to perform adversarial identification of second band features based on the real second band image and the predicted second band image; For the second synthetic image, the third identification sub-model is used to perform adversarial identification of the second band features based on the real second band image, and the fourth identification sub-model is used to perform adversarial identification of the first band features based on the real first band image and the predicted first band image.

3. The method for synthesizing images based on multi-band features as claimed in claim 2, characterized in that: After the image synthesis network model splices the real first band image with the predicted second band image, the third generation sub-model is used to generate a first synthetic image, and after splicing the real second band image with the predicted first band image, the fourth generation sub-model is used to generate a second synthetic image.

4. The method for synthesizing images based on multi-band features as claimed in claim 1, characterized in that: The first generation sub-model and the second generation sub-model both include 9 convolution blocks, 4 downsampling layers, 4 upsampling layers, a convolution layer and a Tanh activation function; the convolution block includes a convolution layer, a parameter-free attention module, a normalization layer, a dropout layer and a Leaky ReLU activation function; the downsampling layer includes a convolution layer, a parameter-free attention module, a normalization layer and a Leaky ReLU activation function; the upsampling layer includes an F.interpolate interpolation function and a convolution layer.

5. The method for synthesizing images based on multi-band features as claimed in claim 1, characterized in that: The third generation sub-model and the fourth generation sub-model both adopt a five-layer sequential structure, the first four layers each include a transposed convolution layer, a parameter-free attention module, a normalization layer and a Leaky ReLU activation function; the fifth layer includes a convolution layer and a Tanh activation function.

6. The method for synthesizing images based on multi-band features as claimed in claim 2, characterized in that: Each discriminator model consists of four convolutional layers, three normalization layers and one fully connected layer, and each convolutional layer is followed by a Leaky ReLU activation function.

7. The method for synthesizing images based on multi-band features according to claim 1 or 4, characterized in that: The energy function of each neuron in the parameter-free attention module is: in, Where t represents the tth target neuron, t∈[1,N], N=H×W is the total number of neurons on a channel, and x i represents the i-th non-target neuron, i∈[1,N-1], w t and b t is the weight and bias of the linear transformation of the t-th target neuron, a1 is the set constant, u t and is the mean and variance of the t-th target neuron in the corresponding channel.

8. The method for synthesizing images based on multi-band features as claimed in claim 2, characterized in that: When the image synthesis network model is trained using the training set, each identification sub-model and each generation sub-model are trained alternately, wherein: The loss of training the first generative sub-model includes the adversarial loss between the first generative sub-model and the second discriminative sub-model, and the structural similarity loss between the input true first band image and the generated predicted second band image; The loss of training the second generative sub-model includes the adversarial loss between the second generative sub-model and the fourth discriminative sub-model, and the structural similarity loss between the input real second band image and the generated predicted first band image; The loss of training the third generative sub-model comprises adversarial losses between the third generative sub-model and the first discriminative sub-model and the second discriminative sub-model, respectively, and content losses of the generated first synthetic image relative to the real second band image and the real first band image; The loss of training the fourth generative sub-model comprises adversarial losses between the fourth generative sub-model and the third discriminative sub-model and the fourth discriminative sub-model, respectively, and content losses of the generated second synthetic image relative to the true first band image and the true second band image; The loss of training the first identification sub-model includes the identification probability of the real first band image and the identification probability of the first synthetic image; The loss of training the second identification sub-model includes the identification probability of the real second band image, the identification probability of the first synthetic image, and the identification probability of the predicted second band image; The loss of training the third identification sub-model includes the identification probability of the real second band image and the identification probability of the second synthetic image; The loss of training the fourth discriminator model includes the discrimination probability of the real first band image, the discrimination probability of the second synthetic image, and the discrimination probability of the predicted first band image.

Images