Unsupervised image fusion method and device for improving feature correspondence, equipment and medium

Abstract

The invention provides an unsupervised image fusion method and device for improving feature correspondence, equipment and a medium, and the method comprises the steps: obtaining a training data set; the training data set is used as the input of an initial adversarial network, the feature fusion image after feature fusion is used as the output of the initial adversarial network, a loss function of a generator and a loss function of a discriminator are used for training the initial adversarial network, an image fusion model is obtained, and the initial adversarial network comprises the generator and the discriminator connected with the generator; the generator comprises a style encoder and a content encoder, and style features of each level obtained by the style encoder and content features obtained by the content encoder are integrated to each level of the generator for fusion; and inputting the to-be-converted data set into the image fusion model for image fusion to obtain a feature fusion image, so as to solve the problems that in a complex image conversion task, complete style representation is difficult to retain stably, and an image which can be matched with input style / content is difficult to generate.

Description

Technical Field

[0001] The present invention relates to the field of image fusion technology, and in particular to an unsupervised image fusion method, device, equipment and medium for improving feature correspondence. Background Art

[0002] Image-to-Image (I2I) techniques are able to adaptively convert images between two different visual domains. Recent advances in I2I have greatly boosted the performance of downstream tasks such as image processing, style transfer, and game design. Early I2I research focused on supervised methods, which heavily rely on manually labeled or paired data. However, due to the labor-intensive nature of creating such datasets, research focus has gradually shifted to unsupervised image translation methods (Unsupervised Image-to-Image Translation Networks, UNIT). UNIT is able to extract content features and style features from images in different visual domains separately without the need for manually labeled or paired data, and fuse them to generate new images. In this paper, style refers to the visual appearance of the main objects in the image, while content refers to other aspects such as facial orientation, body pose, facial features, and approximate shape correspondence.

[0003] Most of the work on UNIT exploits cycle consistency to establish inter-domain mappings, which enables it to show surprising results on very close inter-domain translations (i.e., tasks without drastic shape and appearance changes), such as horse to zebra, summer to winter, and photographs to Monet-style pictures. However, the performance of these models degrades when performing far-domain translations, such as tasks such as bird to dog and bird to car, which is largely attributed to the over-reliance on pixel-level reconstruction losses. Despite the continuous progress of UNIT, there are still challenges in maintaining the stability of the generated images consistent with the input style and content features. Specifically, 1) Reference Figure 1 (a) Incomplete style representation: Unable to generate realistic dogs. Previous work failed to generate realistic dogs corresponding to the style input, indicating that previous models have difficulty in stably retaining complete style representation in complex transformations; 2) Figure 1 (b) shows style mismatch: it is impossible to generate a dog breed that corresponds to the style input. The dog breeds generated by previous work are significantly different from the style input, revealing the instability of previous work in maintaining style correspondence. Figure 1 (c) is a content mismatch: it is impossible to generate a left-facing bird corresponding to the content input. The facial orientation of the bird generated by previous work is different from the facial orientation of the dog in the content input, highlighting the instability of previous work in maintaining content correspondence; 3) Figure 1(d) shows insufficient feature decoupling: a black cat is generated, but the shape of its ears is like the dog’s ears in the content input. The shape of the cat’s ears generated by previous work does not match the style input, but is similar to the shape of the dog’s ears in the content input, which suggests that previous work is instability in fully decoupling style / content features.

[0004] Therefore, there is an urgent need to propose an unsupervised image fusion method that improves feature correspondence in order to solve the technical problems of being difficult to stably retain the complete style representation and generating images that can match the input style / content in complex image conversion tasks. Summary of the invention

[0005] In order to overcome the problems existing in the related art, the present disclosure provides an unsupervised image fusion method, device, equipment and medium for improving feature correspondence, so as to solve the technical problems in the related art that it is difficult to stably retain the complete style representation in complex image conversion tasks and it is difficult to generate images that can match the input style / content.

[0006] One or more embodiments of the present specification provide an unsupervised image fusion method for improving feature correspondence, comprising the following steps:

[0007] Obtain a training data set, wherein the training data set includes a content input image and a style input image;

[0008] The training data set is used as the input of the initial adversarial network, and the feature fusion image after feature fusion is used as the output of the initial adversarial network. The initial adversarial network is trained using the loss function of the generator and the loss function of the discriminator to obtain an image fusion model, wherein the initial adversarial network includes a generator and a discriminator connected to the generator, the generator includes a style encoder and a content encoder, and each level of style features obtained by the style encoder and the content features obtained by the content encoder are integrated into each level of the generator for fusion;

[0009] The data set to be converted is input into the image fusion model for image fusion to obtain a feature fusion image after feature fusion.

[0010] Preferably, the style features of each level acquired by the style encoder and the content features acquired by the content encoder are integrated into each level of the generator for fusion, and the fusion process is as follows:

[0011] The first-level style features and the first-level content features are integrated into the first processing unit of the generator, and the first-level generator features and the first-level hidden state are obtained by fusing them using the residual block of adaptive instance normalization;

[0012] Using DSC, the third-level content features are connected with the first-level generator features and the first-level hidden states in the second processing unit of the generator, and then fused with the second-level style features to obtain the second-level generator features and the second-level hidden states;

[0013] Use DSC to connect the fourth-level content features with the second-level generator features in the third processing unit of the generator and then fuse them with the third-level style features to obtain the third-level generator features;

[0014] fusing the third generator feature with the fourth-level style feature in a fourth processing unit of the generator to obtain a fourth-level generator feature;

[0015] fusing the fourth-level generator feature with the fifth-level style feature in a fifth processing unit of the generator to obtain a fifth-level generator feature;

[0016] The fifth-level generator features and the sixth-level style features are fused in the sixth processing unit of the generator to obtain the sixth-level generator features as the fused image output.

[0017] Preferably, the method further comprises the following steps:

[0018] Inputting the style input image in the training data set into the generator to obtain a reconstructed image;

[0019] The style of the reconstructed image and the style of the style input image are aligned at a given level by enriching the style reconstruction loss function, expressed as:

[0020]

[0021] in, represents the style features of the reconstructed image, Represents the multi-level style features of the style input image, λ rsr represents the weight of the rich style reconstruction loss, Represents the expected value of the rich style reconstruction loss for all style input images y in the style input dataset;

[0022] By enriching the style similarity loss function, the style of the feature fusion image and the multi-level style of the style input image are aligned at each given level, expressed as:

[0023]

[0024] in, represents the style features of the feature fusion image, Represents the multi-level style features of the style input image, λ rss represents the weight of the rich style similarity loss, represents the expected value of the rich style similarity loss for all style input images y in the style input dataset and all content input images x in the content input dataset;

[0025] By enriching the content similarity loss function, the content of the feature fusion image is aligned with the content of the content input image at each level, expressed as:

[0026]

[0027] in, Represents the content features of the feature fusion image, Represents the multi-level content features of the style input image, λ rcs represents the weight of the rich content similarity loss, Represents the expected value of the rich content similarity loss with respect to all style input images y in the style input dataset and all content input images x in the content input dataset;

[0028] The content of the reconstructed image and the content of the style input image are aligned at each given level by enriching the content reconstruction loss function, expressed as:

[0029]

[0030] in, Represents the content features of the reconstructed image, represents the content features of the style input image, λ rcr represents the weight of the rich content reconstruction loss, Represents the expected value of the rich content reconstruction loss with respect to all style input images y in the style input dataset and all content input images x in the content input dataset.

[0031] Preferably, the method further comprises the following steps:

[0032] Before training the image fusion model, a text set describing the image content and a text set describing the style are pre-generated through the LLM model, the text set describing the content and the text set describing the style are encoded into text features through the text encoder in the CLIP model, and the content input image, the style input image, the feature fusion image and the reconstructed image are encoded into image features through the image encoder in the CLIP model;

[0033] Based on the text features and image features, the style input image is calculated by dot product, and the semantic similarity distribution of the style corresponding to the feature fusion image and the reconstructed image is aligned by minimizing the Kullback-Leibler divergence to calculate the style distribution difference loss, which is expressed as:

[0034]

[0035] Among them, S s (k,y) represents the similarity between the kth style description in the given style text list and image y, Represents the kth style description and image in a given style text list The similarity between Represents the kth style description and image in a given style text list The similarity between sdd Represents the weight of the style distribution difference loss.

[0036] Based on the text features and image features, the semantic similarity distribution of the content corresponding to the style input image, the content input image, the feature fusion image and the reconstructed image is calculated by dot product, and the semantic similarity distribution of the content is aligned by calculating the content distribution difference loss, which is expressed as:

[0037]

[0038] Among them, S c (k,y) represents the similarity between the kth content description in the given content text list and image y. Represents the kth content description and image in a given content text list The similarity between Represents the kth content description and image in a given content text list The similarity between c (k,x) represents the similarity between the kth content description in the given content text list and image x, λ cdd Represents the weight of the content distribution difference loss.

[0039] Preferably, the method further comprises the following steps:

[0040] The feature difference between the style input image and the feature fusion image at different VGG layers is calculated through the visual perception similarity loss function, which is expressed as:

[0041]

[0042] in, Representative image After inputting the k-th layer features of the VGG model, represents the k-th layer feature of image y after inputting the VGG model, λ vps Represents the weight of the visually perceptual similarity loss.

[0043] Preferably, the method further comprises the following steps:

[0044] Construct a comprehensive objective loss function:

[0045]

[0046] in, represents the adversarial loss function, where represents the weight of the adversarial loss function, logd(y) is the probability that the discriminator D judges y to be a true image, Judge for D The probability of being a fake image; represents the reconstruction loss function, where λ rec is the weight of the reconstruction loss; represents the mask sparse loss, where λ msk is the weight of the mask sparse loss, m k Represents the mask of the input image x at the kth layer in the generator; represents the discriminator feature distance loss, λ dfd represents the weight of the discriminator distance loss, and F D (y) represent images and y are the feature vectors after entering the discriminator D.

[0047] Preferably, the method further comprises the following steps:

[0048] Before the main training phase, the content encoder is pre-trained using BigGAN with image data generated by the same latent code.

[0049] One or more embodiments of this specification provide an unsupervised image fusion device for improving feature correspondence, including a data preparation module, a model training module, and a fusion module;

[0050] A data preparation module, used to obtain a training data set, wherein the training data set includes a content input image and a style input image;

[0051] A model training module is used to use the training data set as the input of the initial adversarial network, use the feature fusion image after feature fusion as the output of the initial adversarial network, and use the loss function of the generator and the loss function of the discriminator to train the initial adversarial network to obtain an image fusion model, wherein the initial adversarial network includes a generator and a discriminator connected to the generator, the generator includes a style encoder and a content encoder, and each level of style features obtained by the style encoder and the content features obtained by the content encoder are integrated into each level of the generator for fusion;

[0052] The fusion module is used to input the data set to be converted into the image fusion model for image fusion to obtain a feature fusion image after feature fusion.

[0053] One or more embodiments of the present specification provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the unsupervised image fusion method corresponding to the above-mentioned improved features when executing the computer program.

[0054] One or more embodiments of the present specification provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the unsupervised image fusion method for improving feature correspondence as described above.

[0055] The present disclosure provides an unsupervised image fusion method, device, equipment and medium for improving feature correspondence, which has the advantages of obtaining a training data set, wherein the training data set includes a content input image and a style input image, and the performance of the model in image conversion between near domain, far domain and very far domain is guaranteed by training the data set; the training data set is used as the input of an initial adversarial network, and the feature fusion image is used as the output of the initial adversarial network, and the initial adversarial network is trained using the loss function of a generator and the loss function of a discriminator to obtain an image fusion model, wherein the initial adversarial network includes a generator and a discriminator connected to the generator, and the generator includes a style encoder and a content encoder, and each level of style features obtained by the style encoder and the content features obtained by the content encoder are integrated into each level of the generator for fusion, and multi-level style features are obtained through a formulated structure, and these features are subsequently integrated into each level of the generator for fusion, thereby ensuring the retention of more detail representations, improving the overall quality of the generated image, and promoting feature decoupling; the data set to be converted is input into the image fusion model for image fusion, and a feature fusion image after feature fusion is obtained, which can achieve stable unsupervised image conversion in complex image conversion tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0057] Figure 1 defects in the operation of the UNIT provided for one or more embodiments of this specification, Figure 1 (a) is an incomplete style representation. Figure 1 (b) style mismatch, Figure 1 (c) The content does not match. Figure 1 (d) The feature decoupling is insufficient;

[0058] Figure 2 A schematic flow chart of an unsupervised image fusion method for improving feature correspondence provided in one or more embodiments of this specification;

[0059] Figure 3 A schematic diagram of the main training process of the unsupervised image fusion method for improving feature correspondence provided by one or more embodiments of this specification, Figure 3 (a) Schematic diagram of the main training process. Figure 3 (b) Schematic diagram of the process of multi-level style embedding;

[0060] Figure 4 A schematic diagram of a feature fusion process in a generator provided for one or more embodiments of this specification;

[0061] Figure 5 A schematic diagram of the structure of the semantic perception fitting process provided in one or more embodiments of this specification, Figure 5 (a) is a schematic diagram of the overall process of semantic perception fitting. Figure 5 (b) is y and Schematic diagram of the flow of calculating style distribution differences in text;

[0062] Figure 6 A schematic diagram of a process for training a content encoder using content-related images generated by BigGAN for one or more embodiments of this specification;

[0063] Figure 7 A schematic diagram of the structure of an unsupervised image fusion device for improving feature correspondence provided by one or more embodiments of this specification;

[0064] Figure 8 A schematic diagram of the structure of a computer device provided for one or more embodiments of this specification. DETAILED DESCRIPTION

[0065] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the technical solutions in one or more embodiments of this specification will be clearly and completely described below in conjunction with the drawings in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of the present invention.

[0066] The present invention is described in detail below in conjunction with specific implementation methods and the accompanying drawings.

[0067] Method Embodiment

[0068] According to an embodiment of the present invention, an unsupervised image fusion method for improving feature correspondence is provided, such as Figure 2 As shown, it is a schematic diagram of the process of the unsupervised image fusion method for improving feature correspondence provided in this embodiment. The unsupervised image fusion method for improving feature correspondence according to the embodiment of the present invention includes:

[0069] S210, obtain a training data set, wherein the training data set includes content input images and style input images. All images are resized to 256×256 pixels to unify the input format. In order to improve the generalization of the model, we randomly crop the images in the data set (the cropped images retain the main objects intact) and flip them to enhance the data.

[0070] To more comprehensively evaluate the effectiveness of our method, we train and test it on three datasets: Celeb A-HQ, AFHQ, and ImageNet 291 to measure the performance of our method in image translation between near domain, far domain, and far domain.

[0071] CelebA-HQ is a high-quality face image dataset that contains 30,000 high-quality celebrity face images. These images have been further processed and repaired to eliminate some of the low quality and distortion issues in the original Celeb A dataset, thereby providing better data support for image generation and computer vision tasks. We use 28,000 images for training and the remaining 2,000 images for testing. This dataset is provided by the paper Progressive Growing of GANs for Improved Quality, Stability, and Variation.

[0072] The AFHQ dataset is a high-quality animal face image dataset. The dataset contains 15,000 high-quality animal face images, including cats, dogs, and other wild animal images, covering a variety of postures, expressions, and background changes, with good diversity. We use 4,000 cat and dog images for training, and use 500 images from each domain for testing. The dataset is provided by the paper "Star GANv2: Diverse Image Synthesis for Multiple Domains".

[0073] The Image Net 291 dataset is extracted from Image Net through careful screening. By performing object detection on the images in the Image Net dataset and cutting out the object area, smaller objects are filtered out. After that, the remaining images of each category are ranked according to the similarity between their style features and the average style features of the category, and finally the 650 images with the highest similarity are selected. For each domain, we use 600 images for training and the remaining 50 images for testing. This dataset is provided by the paper "Unsupervised Image-to-Image Translation with Generative Prior".

[0074] The Syn Image Net 291 dataset is obtained by using the official Big GAN to generate related images associated by random latent codes for each of the 291 visual domains, including animals and vehicles. After filtering the low-quality images, 655 images were finally obtained for each domain, and the images generated by the same latent code in each domain have content correlation. For each domain, we use 600 images for training and the remaining 50 images for testing. This dataset is provided by the paper "Unsupervised Image-to-Image Translation with Generative Prior".

[0075] S220, using the training data set as the input of the initial adversarial network, using the feature fusion image after feature fusion as the output of the initial adversarial network, using the loss function of the generator and the loss function of the discriminator to train the initial adversarial network to obtain an image fusion model, the initial adversarial network includes a generator and a discriminator connected to the generator, the generator includes a style encoder Es and a content encoder Ec, the generator is mainly composed of multiple processing units, different processing units are not the same, including a conditional residual block with adaptive instance normalization (Ada IRes), an upsampling layer (UP), a multi-channel to RGB (To RGB) module and a dynamic skip connection (DSC) module. Composition of the style encoder: The style encoder architecture consists of Figure 3 (b) shows that it contains 6 convolutional layers, 6 activation function layers, 6 pooling layers, 6 fully connected layers and a multi-layer perceptron (MLP). Content encoder structure: The content encoder architecture is also composed of Figure 3 As shown in (b), it contains 12 convolutional layers, 12 activation function layers and 12 instance normalization layers. Construction of the discriminator: The discriminator consists of two convolutional layers, six residual blocks, six mean normalization layers, nine spectral normalization layers, two activation function layers and one fully connected layer.

[0076] The style encoder E s Each level of style feature E is obtained s (y) and the content encoder E c The content feature E obtained c (x) is integrated into each level of the generator for fusion, such as Figure 3 As shown, it is a schematic diagram of the main training process of the unsupervised image fusion method for improving feature correspondence provided in this embodiment, Figure 3 (a) Schematic diagram of the main training process.

[0077] Specifically, in the main training phase, each iteration involves a content input image x and a style input image y, and the goal is to fuse the content features of x and the style features of y to generate a new image. In order to achieve high-quality image generation and ensure perfect feature correspondence, in addition to requiring a pre-trained content encoder E c Able to completely abandon E c In addition to the style representation in (x), a style encoder E is also required s Completely decoupled style features E s (y). In addition, the generator G is also required to be able to convert E c (x) and E s (y) Perfect fusion to generate feature fusion image To this end, we propose a multi-level style embedding strategy (MSE) to alleviate the difficulty of decoupling and provide more specific style representation.

[0078] Step S230: input the data set to be converted into the image fusion model for image fusion to obtain a feature fusion image after feature fusion.

[0079] The method provided in this embodiment obtains a training data set, wherein the training data set includes a content input image and a style input image, and the performance of the model in image conversion between near domain, far domain and very far domain is guaranteed by training the data set; the training data set is used as the input of the initial adversarial network, the feature fusion image is used as the output of the initial adversarial network, and the initial adversarial network is trained using the loss function of the generator and the loss function of the discriminator to obtain an image fusion model, wherein the initial adversarial network includes a generator G and a discriminator D connected to the generator, and the generator includes a style encoder E c and content encoder E s , the style encoder E c Obtain each level of style features and the content encoder E s The acquired content features are integrated into each level of the generator G for fusion, and E sThe structure is used to obtain multi-level style features, which are then integrated into each level of the generator G for fusion, ensuring the retention of more detail representations, improving the overall quality of the generated image, and promoting feature decoupling; the data set to be converted is input into the image fusion model for image fusion to obtain a feature fused image after feature fusion, which can achieve stable unsupervised image conversion in complex image conversion tasks.

[0080] In one embodiment, the style encoder E s Each level of style feature E is obtained s (y) and the content encoder E c The content feature E obtained c (x) is integrated into each level of the generator G for fusion, such as Figure 3 (a) is a schematic diagram of the main training process of generating feature fusion images using MSE (Lv.0-Lv.5) provided in this embodiment. When x and y are given, we first apply a pooling layer and a fully connected layer after each activation layer to obtain style features E at each level. s (x), then we gradually add E to G s (y) and E c (x) Fusion to generate For brevity, we omit the presentation of the normalization layer.

[0081] The fusion process is as follows:

[0082] The core feature fusion process in MSE is as follows:

[0083]

[0084] Among them, G l and H l Respectively represent input to l th Processing Unit (PU) l ), and Respectively represent l th and (l+1) th The style and content features of the PU are shown in Figure 2. ↑ indicates upsampling, Ada IRes indicates a residual block with adaptive instance normalization, and DSC is a dynamic skip connection block, which helps the model learn the feature correspondence from coarse to fine. It is worth noting that for different PUs, the corresponding feature fusion process is slightly different, especially DSC, which is only used in PU 1 and PU 2 Used in, such as Figure 4 As shown in Figure 2, it is a schematic diagram of the feature fusion process in the generator.

[0085] Specifically, Figure 3 (b) is a schematic diagram of the multi-level style embedding process provided by this embodiment. and first level content features The first processing unit PU integrated into the generator G 0 In the example, the residual block with adaptive instance normalization is used to fuse the first-level generator feature G 1 and the first level hidden state H 1 .

[0086] Use DSC to characterize tertiary content With the first level generator feature G 1 , the first level hidden state H 1 In the second processing unit PU of the generator G 1 After the middle connection and the second level style features Fusion, get the second level generator feature G 2 and the second level hidden state H 2 .

[0087] Use DSC to characterize the fourth level content With the second level generator feature G 2 In the third processing unit PU of the generator G 2 After the middle connection and the third level style features Fusion, get the third-level generator feature G 3 .

[0088] The third generator feature G 3 With the fourth level style characteristics In the fourth processing unit PU of the generator G 3 Fusion, get the fourth level generator feature G 4 .

[0089] The fourth-level generator feature G 4 With the fifth level style characteristics In the fifth processing unit PU of the generator G 4 The fifth-level generator feature G is obtained by fusion 5 .

[0090] The fifth-level generator feature G 5 With sixth-level style features In the sixth processing unit PU of the generator G 5 The sixth-level generator feature G is obtained by fusion 6 Output as the fused image.

[0091] The method provided in this embodiment is to formulate E sThe structure is used to obtain multi-level style features and integrate them into each level of the generator G for fusion, ensuring the retention of more detailed representations, improving the overall quality of the generated image, and promoting feature decoupling.

[0092] Although MSE enhances feature preservation, decoupling, and fusion, the lack of sufficient constraints may lead to feature drift, resulting in style / content mismatch between the generated image and the input. To reduce the mismatch, we design Rich Consistency Constraints (RCC) for each level of style and content features during the main training process. c (x), E s (y)) The reconstructed image obtained It will be used to help G achieve pixel-level restoration of y.

[0093] In one embodiment, the following steps are also included:

[0094] Input the style in the training dataset into image E s (y) Input into the generator G to obtain the reconstructed image

[0095] In order to promote the model to align the style, the style of the reconstructed image and the style of the style input image are aligned at a given level by enriching the style reconstruction loss function, which is expressed as:

[0096]

[0097] in, represents the style features of the reconstructed image, Represents the multi-level style features of the style input image, λ rsr represents the weight of the rich style reconstruction loss, Represents the expected value of the rich style reconstruction loss with respect to all style input images y in the style input dataset.

[0098] Similarly, The style representation of y should be the same as that of y. Based on this, we further propose a rich style similarity loss By enriching the style similarity loss function, the style of the feature fusion image and the multi-level style of the style input image are aligned at each given level, expressed as:

[0099]

[0100] in, represents the style features of the feature fusion image, Represents the multi-level style features of the style input image, λ rss represents the weight of the rich style similarity loss, Represents the expected value of the rich style similarity loss with respect to all style input images y in the style input dataset and all content input images x in the content input dataset.

[0101] Similar to the style consistency constraint, in order to align the content representation and avoid content mismatch, the content of the feature fusion image is aligned with the content of the content input image at each level by enriching the content similarity loss function, which is expressed as:

[0102]

[0103] in, Represents the content features of the feature fusion image, Represents the multi-level content features of the style input image, λ rcs represents the weight of the rich content similarity loss, Represents the expected value of the rich content similarity loss with respect to all style input images y in the style input dataset and all content input images x in the content input dataset.

[0104] In addition, to ensure that the reconstructed image The consistency with the content of the style input image y is achieved by aligning the content of the reconstructed image and the content of the style input image at each given level through the enriched content reconstruction loss function, which is expressed as:

[0105]

[0106] in, Represents the content features of the reconstructed image, represents the content features of the style input image, λ rcr represents the weight of the rich content reconstruction loss, Represents the expected value of the rich content reconstruction loss with respect to all style input images y in the style input dataset and all content input images x in the content input dataset.

[0107] The method provided in this embodiment reduces the feature mismatch phenomenon that occurs when the model performs image conversion tasks by aligning content and style features at various levels during training.

[0108] In one embodiment, in order to fully decouple style and content features and better fuse them for image conversion, we additionally design a flexible semantic perception fitting module (SPF). This module constrains the similarity of semantic distribution between input and output by mapping the image and the descriptive text generated by the large language model into the same latent space, such as Figure 5FIG. 1 is a schematic diagram of the structure of the semantic perception fitting process provided in this embodiment. Figure 5 (a) Schematic diagram of the overall process of semantic-aware fitting.

[0109] Recent large language models (LLMs) have demonstrated excellent knowledge understanding and conversation capabilities, close to human level. Therefore, in some areas where one's own knowledge is limited, one can use their extensive knowledge base to flexibly generate a complete style description list containing multiple styles through conversations with LLMs. sty , such as ["Mallard", "Osprey", "Ring-necked Plover", ..., "Snake Darter"], with a variable length N. Each description is then passed through the pre-trained CLIP text encoder E t Encoded as text feature F t =[E t ("Mallard"),E t ("Osprey"),…,E t ("Snake Pelican")].

[0110] The following steps are also included:

[0111] Before training the image fusion model, a text collection List describing the image content is pre-generated through the LLM model con List of text collections with styles sty , the text set of the content and the text set of the style are encoded into text features through the text encoder in the deep learning model (Contrastive Language-Image Pretraining, CLIP), and the content input image, style input image, feature fusion image and reconstructed image are encoded into image features through the image encoder in the CLIP model.

[0112] Based on the text features and image features, the style input image is calculated by dot product, and the semantic similarity distribution of the style corresponding to the feature fusion image and the reconstructed image is converted into a probability distribution using softmax, and then the KL divergence between the distributions is calculated to minimize the distribution difference, such as Figure 5 (b) shows y and Schematic diagram of the style distribution difference calculation process in text.

[0113] First, we calculate their semantic similarity distribution in terms of style.

[0114] SSD s Take (y) as an example,

[0115]

[0116] Among them, S S (k, y) represents image y and List sty The similarity between the k-th style description (e.g., "snake darter") in Representing List sty The kth feature of t ("Ring-necked Plover"), Represents image y input to image encoder E i The characteristic after that, e represents the base of the natural logarithm. and All by Style E s (y) is composed of and Should be with SSD s (y)Same.

[0117] Style distribution difference loss alignment is calculated by minimizing Kullback-Leibler divergence and With SSD s The semantic similarity distribution of the style of (y) is expressed as:

[0118]

[0119] Among them, S s (k,y) represents the similarity between the kth style description in the given style text list and image y, Represents the kth style description and image in a given style text list The similarity between Represents the kth style description and image in a given style text list The similarity between sdd Represents the weight of the style distribution difference loss.

[0120] In the same way, we calculate the semantic similarity distribution of their contents respectively.

[0121] SSD c Take (x) as an example,

[0122]

[0123] Among them, S c (k, x) means x and List con The similarity between the k-th content description (e.g., “portrait with side view”) in Representing List con The kth feature oft ("Portrait with side view"), Represents image x as input to image encoder E i Theoretically, at the semantic level, should be identical in content to x, The same should be true for y.

[0124] Based on the text features and image features, the semantic similarity distribution of the content corresponding to the style input image, the content input image, the feature fusion image and the reconstructed image is calculated by dot product, and the alignment is performed by calculating the content distribution difference loss With SSD c The semantic similarity distribution of the content of (x) is expressed as:

[0125]

[0126] Among them, S c (k,y) represents the similarity between the kth content description in the given content text list and image y. Represents the kth content description and image in a given content text list The similarity between Represents the kth content description and image in a given content text list The similarity between c (k,x) represents the similarity between the kth content description in the given content text list and image x, λ cdd Represents the weight of the content distribution difference loss.

[0127] The method provided in this embodiment uses the multimodal model CLIP to intuitively minimize the semantic difference between input and output, so that the model can better decouple style and content features and fuse them for high-quality image conversion.

[0128] In one embodiment, to further improve To find the correspondence between the appearance of y and y, we propose Visual Perception Guidance (VPG), which uses the structured perceptual loss of the VGG (Visual Geometry Group) deep convolutional neural network as a more intuitive way to visually match the appearance of two images with large content differences in the UNIT process based on classification features. Given that the pre-trained VGG model is designed for classification, the classification features extracted by VGG and F VGG(y) Should remain consistent across changes in pose, facial orientation, and depth of field.

[0129] The following steps are also included:

[0130] The feature difference between the style input image and the feature fusion image at different VGG layers is calculated through the visual perception similarity loss function, which is expressed as:

[0131]

[0132] in, Representative image After inputting the k-th layer features of the VGG model, represents the k-th layer feature of image y after inputting the VGG model, λ vps Represents the weight of the visually perceptual similarity loss.

[0133] The method provided in this embodiment avoids generating unrealistic details that reduce the visual quality of the image by using the VGG model to reduce the perceptual distance between the generated image and the style input, thereby ensuring high-quality visual output.

[0134] In one embodiment, the following steps are also included:

[0135] Construct a comprehensive objective loss function:

[0136]

[0137] in, represents the adversarial loss function, represents the weight of the adversarial loss function, logD(y) is the probability that the discriminator D judges y to be a true image, and the game between the generator G and the discriminator D is used to enhance the realism of the generated image. represents the reconstruction loss function, λ rec The weights for the reconstruction loss help the model to accurately reconstruct the image by focusing on pixel-level differences. represents the mask sparse loss, λ msk is the weight of the mask sparse loss, m k Represents the mask of the input image x at the kth layer in the generator, which aims to adaptively remove irrelevant content by sparsifying the mask in the DSC process. The mask m k is generated by the attention mechanism of the spatial and channel dimensions in the DSC process, represents the discriminator feature distance loss, λ dfd represents the weight of the discriminator distance loss, and F D (y) represent images The feature vector after y is input into the discriminator D is designed to minimize and the discriminator features extracted from y and F DThe weights in each loss are set as: adv =λ rec =λ msk =1.0,λ rcs =λ vps =0.1,λ rcr =0.5,λ rsr =λ rss =λ dfd =50.0,λ cdd =λ sdd =0.05.

[0138] The method provided in this embodiment improves the feature correspondence between the generated image and the input image by establishing rich feature constraints and introducing multimodal information, thereby realizing a technology for stable unsupervised image conversion.

[0139] In one embodiment, the following steps are also included:

[0140] Before the main training phase, in order to stably extract abstract cross-domain content features, the content encoder is pre-trained using Big GAN with image data generated by the same latent code to extract priors, and then E is fixed. c and transfer it to the main training phase, such as Figure 6 , which is a schematic diagram of the process of training a content encoder using content-related images generated by Big GAN provided in this embodiment.

[0141] Device Embodiment

[0142] According to an embodiment of the present invention, an unsupervised image fusion device for improving feature correspondence is provided, such as Figure 7 As shown, it is a schematic diagram of the structure of the unsupervised image fusion device corresponding to the improved features provided in this embodiment. The unsupervised image fusion device corresponding to the improved features according to the embodiment of the present invention includes a data preparation module, a model training module and a fusion module.

[0143] The data preparation module 71 is used to obtain a training data set, wherein the training data set includes a content input image and a style input image.

[0144] The model training module 72 is used to use the training data set as the input of the initial adversarial network and the feature fusion image as the output of the initial adversarial network, and use the loss function of the generator and the loss function of the discriminator to train the initial adversarial network to obtain an image fusion model. The initial adversarial network includes a generator and a discriminator connected to the generator, and the generator includes a style encoder and a content encoder. The style features of each level obtained by the style encoder and the content features obtained by the content encoder are integrated into each level of the generator for fusion.

[0145] The fusion module 73 is used to input the data set to be converted into the image fusion model for image fusion to obtain a feature fusion image after feature fusion.

[0146] In the apparatus provided by this embodiment, the data preparation module 71 obtains a training data set, wherein the training data set includes a content input image and a style input image, and the performance of the model in image conversion between near domain, far domain and very far domain is guaranteed by training the data set; the model training module 72 uses the training data set as the input of the initial adversarial network and the feature fusion image as the output of the initial adversarial network, and trains the initial adversarial network using the loss function of the generator and the loss function of the discriminator to obtain an image fusion model, wherein the initial adversarial network includes a generator and a discriminator connected to the generator, and the generator G includes a style encoder and a content encoder, and each level of style features obtained by the style encoder and the content features obtained by the content encoder are integrated into each level of the generator for fusion, and the E is formulated. s The structure is used to obtain multi-level style features, which are then integrated into each level of the generator G for fusion, ensuring the retention of more detail representations, improving the overall quality of the generated image, and promoting feature decoupling; the fusion module 73 inputs the data set to be converted into the image fusion model for image fusion, and obtains a feature fused image after feature fusion, which can achieve stable unsupervised image conversion in complex image conversion tasks.

[0147] The embodiment of the present invention is an apparatus embodiment corresponding to the above-mentioned method embodiment. The specific operations of the processing steps of each module can be understood by referring to the description of the method embodiment, which will not be repeated here.

[0148] like Figure 8 As shown, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the unsupervised image fusion method corresponding to the improved features in the above-mentioned embodiment, or, when executed by a processor, implements the unsupervised image fusion method corresponding to the improved features in the above-mentioned embodiment.

[0149] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0150] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and the contents not described in detail in the specification of the present invention belong to the common knowledge of those skilled in the art.

Claims

1. An unsupervised image fusion method for improving feature correspondence, characterized in that: The following steps are involved: Obtain a training data set, wherein the training data set includes a content input image and a style input image; The training data set is used as the input of the initial adversarial network, and the feature fusion image after feature fusion is used as the output of the initial adversarial network. The initial adversarial network is trained using the loss function of the generator and the loss function of the discriminator to obtain an image fusion model, wherein the initial adversarial network includes a generator and a discriminator connected to the generator, the generator includes a style encoder and a content encoder, and each level of style features obtained by the style encoder and the content features obtained by the content encoder are integrated into each level of the generator for fusion; The data set to be converted is input into the image fusion model for image fusion to obtain a feature fusion image after feature fusion.

2. The unsupervised image fusion method according to claim 1, characterized in that: The style features of each level obtained by the style encoder and the content features obtained by the content encoder are integrated into each level of the generator for fusion. The fusion process is as follows: The first-level style features and the first-level content features are integrated into the first processing unit of the generator, and the first-level generator features and the first-level hidden state are obtained by fusing them using the residual block of adaptive instance normalization; Using DSC, the third-level content features are connected with the first-level generator features and the first-level hidden states in the second processing unit of the generator, and then fused with the second-level style features to obtain the second-level generator features and the second-level hidden states; Use DSC to connect the fourth-level content features with the second-level generator features in the third processing unit of the generator and then fuse them with the third-level style features to obtain the third-level generator features; fusing the third generator feature with the fourth-level style feature in a fourth processing unit of the generator to obtain a fourth-level generator feature; fusing the fourth-level generator feature with the fifth-level style feature in a fifth processing unit of the generator to obtain a fifth-level generator feature; The fifth-level generator features and the sixth-level style features are fused in the sixth processing unit of the generator to obtain the sixth-level generator features as the fused image output.

3. The unsupervised image fusion method according to claim 1, characterized in that: The following steps are also included: Inputting the style input image in the training data set into the generator to obtain a reconstructed image; The style of the reconstructed image and the style of the style input image are aligned at a given level by enriching the style reconstruction loss function, expressed as: in, represents the style features of the reconstructed image, Represents the multi-level style features of the style input image, λ rsr represents the weight of the rich style reconstruction loss, Represents the expected value of the rich style reconstruction loss for all style input images y in the style input dataset; By enriching the style similarity loss function, the style of the feature fusion image and the multi-level style of the style input image are aligned at each given level, expressed as: in, represents the style features of the feature fusion image, Represents the multi-level style features of the style input image, λ rss represents the weight of the rich style similarity loss, represents the expected value of the rich style similarity loss for all style input images y in the style input dataset and all content input images x in the content input dataset; By enriching the content similarity loss function, the content of the feature fusion image is aligned with the content of the content input image at each level, expressed as: in, Represents the content features of the feature fusion image, Represents the multi-level content features of the style input image, λ rcs represents the weight of the rich content similarity loss, Represents the expected value of the rich content similarity loss with respect to all style input images y in the style input dataset and all content input images x in the content input dataset; The content of the reconstructed image and the content of the style input image are aligned at each given level by enriching the content reconstruction loss function, expressed as: in, Represents the content features of the reconstructed image, represents the content features of the style input image, λ rcr represents the weight of the rich content reconstruction loss, Represents the expected value of the rich content reconstruction loss with respect to all style input images y in the style input dataset and all content input images x in the content input dataset.

4. The unsupervised image fusion method according to claim 1, characterized in that: The following steps are also included: Before training the image fusion model, a text set describing the image content and a text set describing the style are pre-generated through the LLM model, the text set describing the content and the text set describing the style are encoded into text features through the text encoder in the CLIP model, and the content input image, the style input image, the feature fusion image and the reconstructed image are encoded into image features through the image encoder in the CLIP model; Based on the text features and image features, the style input image is calculated by dot product, and the semantic similarity distribution of the style corresponding to the feature fusion image and the reconstructed image is aligned by minimizing the Kullback-Leibler divergence to calculate the style distribution difference loss, which is expressed as: Among them, S s (k,y) represents the similarity between the kth style description in the given style text list and image y, Represents the kth style description and image in a given style text list The similarity between Represents the kth style description and image in a given style text list The similarity between sdd Represents the weight of the style distribution difference loss. Based on the text features and image features, the semantic similarity distribution of the content corresponding to the style input image, the content input image, the feature fusion image and the reconstructed image is calculated by dot product, and the semantic similarity distribution of the content is aligned by calculating the content distribution difference loss, which is expressed as: Among them, S c (k,y) represents the similarity between the kth content description in the given content text list and image y. Represents the kth content description and image in a given content text list The similarity between Represents the kth content description and image in a given content text list The similarity between c (k,x) represents the similarity between the kth content description in the given content text list and image x, λ cdd Represents the weight of the content distribution difference loss.

5. The unsupervised image fusion method according to claim 1, characterized in that: The following steps are also included: The feature difference between the style input image and the feature fusion image at different VGG layers is calculated through the visual perception similarity loss function, which is expressed as: in, Representative image After inputting the k-th layer features of the VGG model, represents the k-th layer feature of image y after inputting the VGG model, λ vps Represents the weight of the visually perceptual similarity loss.

6. The unsupervised image fusion method according to claim 1, 3, 4 or 5, characterized in that: The following steps are also included: Construct a comprehensive objective loss function: in, represents the adversarial loss function, represents the weight of the adversarial loss function, logD(y) is the probability that the discriminator D judges y to be a true image, Judge for D The probability of being a fake image; represents the reconstruction loss function, λ rec is the weight of the reconstruction loss; represents the mask sparse loss, λ msk is the weight of the mask sparse loss, m k Represents the mask of the input image x at the kth layer in the generator; represents the discriminator feature distance loss, λ dfd represents the weight of the discriminator distance loss, and F D (y) represent images and y are the feature vectors after entering the discriminator D.

7. The unsupervised image fusion method according to claim 1, characterized in that: The following steps are also included: Before the main training phase, the content encoder is pre-trained using BigGAN with image data generated by the same latent code.

8. An unsupervised image fusion device for improving feature correspondence, characterized in that: Includes data preparation module, model training module and fusion module; A data preparation module, used to obtain a training data set, wherein the training data set includes a content input image and a style input image; A model training module is used to use the training data set as the input of the initial adversarial network, use the feature fusion image after feature fusion as the output of the initial adversarial network, and use the loss function of the generator and the loss function of the discriminator to train the initial adversarial network to obtain an image fusion model, wherein the initial adversarial network includes a generator and a discriminator connected to the generator, the generator includes a style encoder and a content encoder, and each level of style features obtained by the style encoder and the content features obtained by the content encoder are integrated into each level of the generator for fusion; The fusion module is used to input the data set to be converted into the image fusion model for image fusion to obtain a feature fusion image after feature fusion.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the unsupervised image fusion method with improved feature correspondence as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the unsupervised image fusion method for improving feature correspondence as claimed in any one of claims 1 to 7 are implemented.

Images