A one-way image conversion method using contrastive learning and data augmentation

Abstract

The application discloses a one-way image conversion method using contrast learning and data enhancement, and comprises the following steps: acquiring training data sets for one-way image conversion from a source domain and a target domain respectively, inputting the training data sets into a pre-constructed conversion network model, and performing data enhancement on the training data sets based on feature interpolation; the training data sets comprise input images and output images; inputting the data-enhanced training data sets into the conversion network model for training, establishing a target function through contrast learning, and generating a trained conversion network model; and inputting an image to be converted into the trained conversion network model, and outputting a converted image. The method can effectively prevent the constructed conversion network model discriminator from overfitting, increase the generalization ability of the network, effectively guarantee the perceptual relationship between samples, and improve the one-way image conversion efficiency.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a one-way image conversion method using contrastive learning and data augmentation. Background Technology

[0002] Image-to-image transformation essentially involves training a model to learn the mapping relationship between one image domain and another. The current mainstream approach uses generative adversarial networks (GANs). The goal is to ensure the output reflects the appearance of the target domain while preserving the structure or content of the specific input. Fundamentally, this is a separation problem: separating the content that needs to be preserved across domains from the appearance that must be changed. Typically, the target appearance is enforced using adversarial losses, while the content is guaranteed using cycle consistency. While effective, the fundamental bijective assumption behind cycle consistency is sometimes too strict. Perfect reconstruction is difficult to achieve, especially when an image from one domain contains additional information compared to another.

[0003] Another approach is to encourage image relationships present in the input to be similarly reflected in the output. For example, images perceived as similar in the input should be similar in the output. Regarding predefined distances, the output and input images share similar content; the vector algorithm between input images is performed using margin-based triplet loss; the distances between input images should be consistent in the output image; and the network should be identical to the geometric transformation. TraVeLGAN, DistanceGAN, and GcGAN implement unidirectional transformations and avoid cycle consistency. However, they rely on relationships across the entire image, or often on a predefined distance function.

[0004] Traditional unsupervised learning attempts to learn a compressed code that can efficiently reconstruct the input. Data imputation, which preserves a subset of the original data to make predictions from another subset, has emerged as a more efficient approach for a similar family of tasks, including denoising, context prediction, colorization, multichannel coding, frame prediction, and multisensory prediction. However, this approach suffers from the same problem—the need for a pre-specified, hand-designed loss function to measure prediction performance.

[0005] While Generative Adversarial Networks (GANs) have demonstrated significant success in image translation tasks in recent years, with well-designed adversarial losses providing effective domain-level supervision and rendering translated images indistinguishable from real samples, GAN-based image translation methods heavily rely on a large number of training examples. For instance, Karras et al. used 70K high-quality face images to train their model. However, collecting such large amounts of image data can be very expensive or impractical. This issue highlights the importance of training GANs with limited data. However, reducing the amount of training data often leads to severe model overfitting. Recent results show that GANs overfit on the training set, resulting in a sharp decline in performance on the test set. Adaptive-based methods use external datasets as an alternative. Even with limited data, these methods can prevent discriminator overfitting, thus achieving effective adversarial supervision. However, increasing data for GANs in this way is not feasible due to pattern leakage problems.

[0006] Therefore, based on existing image conversion technologies, how to avoid the mandatory constraints of cycle consistency and the design loss of sensory similarity in the constructed conversion network model, while avoiding the use of reverse auxiliary generators and discriminators to perform unidirectional image conversion, has become a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0007] In view of the above problems, this invention proposes a one-way image conversion method using contrastive learning and data augmentation that at least solves some of the above technical problems. This method can effectively prevent the discriminator of the constructed conversion network model from overfitting, increase the generalization ability of the network, effectively ensure the perceptual relationship between samples, and effectively improve the efficiency of one-way image conversion.

[0008] This invention provides a one-way image transformation method using contrastive learning and data augmentation, comprising the following steps:

[0009] S1. Obtain training datasets for unidirectional image conversion from the source domain and the target domain respectively, input them into the pre-built conversion network model, and perform data augmentation on the training datasets based on feature interpolation; the training datasets include: input images and output images;

[0010] S2. Input the augmented training dataset into the conversion network model for training, establish the objective function through comparative learning, and generate the trained conversion network model.

[0011] S3. Input the image to be converted into the trained conversion network model and output the converted image.

[0012] Further, in step S1, data augmentation is performed on the training dataset based on feature interpolation, including:

[0013] S11. Interpolate the training dataset at the feature level, wherein the interpolated data is:

[0014] e′=λ·e 1 +(1-λ)·e 2

[0015] In the above formula, e' represents the interpolated data; λ represents the interpolation weight, λ∈[0,1]; e 1 and e 2 Representing two eigenvectors, by using x 1 and x 2 The x is obtained by inputting the encoder in the transformation network model. 1 and x 2 This represents two random samples in the source domain;

[0016] S12. Based on the constraints, the interpolated data is converted into interpolated data that retains specific content information, and the interpolated image is obtained by inputting it into the decoder in the conversion network model. Data augmentation is then performed on the training dataset. The specific content information refers to the shape and position of objects in the training dataset images.

[0017] Furthermore, the constraints in step S12 are as follows:

[0018]

[0019] In the above formula, This represents the content loss; s′=G2(G1(e′)), e′ represents the interpolated data, G1 represents the first half of the generator G of the transformation network model, and G2 represents the second half of the generator G of the transformation network model; t 1 Indicates the x 1 Content information; t 2 Indicates the x 2 Content information; t rand x represents rand Content information, the x rand This indicates that the source domain contains, except for x. 1 and the x 2 Another random sample; φ(s') represents the output of s' after passing through a pre-constructed content extraction network; φ(t 1 ) represents the t 1 The output after passing through the content extraction network; φ(t) 2 ) represents the t 2 The output after passing through the content extraction network; φ(t)rand ) represents the t rand The output after passing through the content extraction network.

[0020] Further, in step S2, the calculated cross-entropy loss of the generated trained transformation network model is:

[0021]

[0022] In the above formula, τ represents the scaling factor; υ represents the query vector; υ + Represents a positive example vector; υ - Represents a negative example vector; This represents the nth negative example vector; n represents the index of the negative example vector; N represents the number of negative example vectors.

[0023] Further, in step S2, the objective function is established through contrastive learning, including:

[0024] S21. Generate a feature map based on the encoder intermediate layer in the conversion network model and output a feature vector;

[0025] S22. Based on the feature vector, match image patches in the input image of the training dataset that are related to the output image to generate a positive excitation signal; and use other image patches in the input image that are not related to the output image as negative samples to generate a negative excitation signal, and construct the PatchNCE loss function:

[0026]

[0027] In the above formula, G represents the generator of the transformation network model; H represents the transformation network model; X represents the source domain; L represents the number of intermediate layers of the transformation network model; S l This represents the number of features in the l-th intermediate layer of the transformation network model; This represents the feature encoding of the output image; This represents the positive excitation signal; This represents the negative excitation signal; represents the expectation for the source domain; x represents the image in the source domain;

[0028] S23. Based on the PatchNCE loss function and adversarial loss, establish the objective function:

[0029]

[0030] In the above formula, λ X λ represents the contrast loss coefficient of the source domain; YThe contrastive loss coefficient of the target domain is represented by G; the generator of the transformation network model is represented by H; the transformation network model is represented by X; the source domain is represented by Y; and the target domain is represented by Y.

[0031] Further, step S21 includes:

[0032] Select a preset number of encoder intermediate layers from the transformation network model and output feature maps;

[0033] The feature map is input into a two-layer MLP network in the transformation network model, and the output feature vector is:

[0034]

[0035] In the above formula, H l This refers to the two-layer MLP network; G represents the output of the l-th layer of the encoder in the transformation network model; L represents the generator of the transformation network model; and L represents the number of intermediate layers in the two-layer MLP network.

[0036] Furthermore, in step S23, the adversarial loss is:

[0037]

[0038] In the above formula, D(y) represents the output of the discriminator of the transformation network model for the target domain; This represents the expectation for the target domain; G represents the expectation for the source domain; D represents the generator of the transformation network model; X represents the discriminator of the transformation network model; Y represents the source domain; and Y represents the target domain.

[0039] The beneficial effects of the above-described technical solutions provided in the embodiments of the present invention include at least the following:

[0040] This invention provides a one-way image conversion method using contrastive learning and data augmentation, comprising the following steps: obtaining training datasets for one-way image conversion from the source and target domains respectively, inputting them into a pre-constructed conversion network model, and performing data augmentation on the training dataset based on feature interpolation; the training dataset includes an input image and an output image; inputting the data-augmented training dataset into the conversion network model for training, establishing an objective function through contrastive learning, and generating a trained conversion network model; inputting the image to be converted into the trained conversion network model, and outputting the converted image. This method effectively prevents overfitting of the discriminator in the constructed conversion network model, increases the network's generalization ability, effectively ensures the perceptual relationship between samples, and improves the efficiency of one-way image conversion.

[0041] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings.

[0042] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0043] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0044] Figure 1 Flowchart of a one-way image conversion method using contrastive learning and data augmentation provided in an embodiment of the present invention;

[0045] Figure 2 A flowchart of a data augmentation method provided in an embodiment of the present invention;

[0046] Figure 3 A flowchart illustrating an application example of the one-way image conversion method provided in this embodiment of the invention;

[0047] Figure 4 A flowchart illustrating the method for calculating the PatchNCE loss function provided in an embodiment of the present invention. Detailed Implementation

[0048] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0049] This invention provides a one-way image transformation method using contrastive learning and data augmentation, referring to... Figure 1 As shown, it includes the following steps:

[0050] S1. Obtain training datasets for unidirectional image transformation from the source domain and target domain respectively, input them into the pre-built transformation network model, and perform data augmentation on the training datasets based on feature interpolation; the training datasets include: input images and output images;

[0051] S2. Input the augmented training dataset into the conversion network model for training. Establish the objective function through comparative learning and generate the trained conversion network model.

[0052] S3. Input the image to be converted into the trained conversion network model, and output the converted image.

[0053] This embodiment provides a one-way image transformation method using contrastive learning and data augmentation, which both enforces the appearance of the target domain (such as the texture and color of objects in the image) and preserves the structure or content of specific inputs in the image across domains (such as the shape and position of objects in the image). A multi-layer neural network is used to compute contrastive loss on image patches, suitable for training with few samples. The InfoNCE contrastive loss is applied to the image-to-image transformation task, avoiding the strong constraints of cycle consistency and the design of perceptual similarity loss. It only needs to learn the mapping in one direction, avoiding the use of a reverse auxiliary generator and discriminator. This facilitates training the generative adversarial network with limited data in image-to-image transformation. A feature interpolation-based data augmentation strategy called ReMix is ​​used. Linear interpolation methods in convex combinations are used to mix source images in the feature space. The generator learns to map the mixed samples to the target space to prevent overfitting. Furthermore, the loss function in the discriminator is improved during the identification of augmented fake samples, using a novel content loss method that preserves the perceptual relationship between samples, improving training and testing speeds, as well as the efficiency of one-way image transformation.

[0054] The method provided in this embodiment will be described in detail below:

[0055] 1) Data augmentation strategies based on feature interpolation:

[0056] First, training datasets are obtained from the source and target domains, and then the datasets are filtered. (Refer to...) Figure 2 As shown, random samples x are obtained from two source domains. 1 and x 2 The goal is to learn a mapping function from the source domain to the target domain. Image transformation transforms an image from one domain to another (from the source domain to the target domain). For example, in the grayscale to color image transformation, the X domain refers to the set of grayscale images, and the Y domain refers to the set of color images. A transformation network model is built, and a generator G is trained. The objectives are twofold: 1) Given the source domain... G(x) should be almost identical to the sample in the target domain Y; 2) G(x) should retain specific content information, and the final optimizer optimizes the adversarial loss. and content loss After training is complete, use generator G to generate samples.

[0057] Specifically, the transformation network model consists of three networks: a generator (which includes an encoder and a decoder), a discriminator, and a two-layer MLP auxiliary network. During the transformation process, the image is first encoded and then decoded to output the target domain image. The MLP auxiliary network primarily maps the encoder output to the same space, ensuring that similar samples are as close as possible in the feature space, while dissimilar samples are as far apart as possible.

[0058] Secondly, the training dataset is enhanced using an interpolation method based on the feature level of the intermediate layers of the established transformation network model. To this end, the generator G is divided into two parts, let G = G2 * G1, where * represents a function composition operation; G represents the generator of the established transformation network model, and G1 and G2 represent the first and second halves of generator G, respectively. The interpolated data, after mixing the intermediate features of G1, is represented as follows:

[0059] e′=λ·e 1 +(1-λ)·e 2

[0060] Among them, e 1 =G1(x 1 ), e 2 =G1(x 2 ), where G1(x) represents the first half of the generator G(x); e 1 and e 2 The eigenvector is x 1 and x 2 The input x is obtained from the encoder in the input transformation network model; here x 1 and x 2 The input e′ is specified as a random sample from two source domains, and λ∈[0,1] is used as the interpolation weight. When λ is 0 or 1, the input e′ is x. 1 Or x 2 The intermediate features are a special case in interpolation methods. For the interpolation method to be effective for training, the interpolated data needs to be transformed to still retain specific content information (the output retains the corresponding proportion of x). 1 and x 2 The result is the content information. However, for the interpolated e', the content loss needs to be calculated. Therefore, it is necessary to know the content target t'. The conventional approach to obtaining t' is to... 1 and t 2 (t 1 and t 2 They represent x respectively 1 and x 2 The content information is interpolated between the two to calculate the content loss. However, for high-modal content information t, simple interpolation may not correspond to e'. Therefore, a new method for calculating content loss is used to avoid predicting the content target t'. Let s′=(G2(G1(e′)), which restricts {t 1 , t 2 Let G2(x) represent the perceptual relationship between s' and s', where s' represents the output. G2(x) represents the latter half of the generator G(x). Without loss of generality, assume e 1 The weight of is greater, and the generated result s' is restricted to satisfy the following:

[0061]

[0062] in, The content loss is represented by s′ and t. 1 The L1 loss output by φ (content extraction network); φ(t) 1 ) represents t 1 The output after the content extraction network; φ(t) 2 ) represents t 2 The output after passing through the content extraction network; φ(s') represents the output of the interpolation feature s' after passing through the content extraction network. The content extraction network is a pre-trained model used to extract the content information of the samples.

[0063] The L1 loss, or L1 distance loss, has the following meaning: for the input vector x = (x1, x2, ..., x...) n ) and y = (y1, y2, ..., y2) n The definition of L1 Distance Loss is as follows:

[0064]

[0065] After interpolation, e' needs to be located away from e in the interpolation space. 1 than e 2 Closer.

[0066] Although the formula above provides supervised constraints to ensure that the generated result retains content information, This term has no upper bound. This means simply moving the result s' away from t. 2 It can also satisfy the constraints, which is undesirable. Therefore, we define e. rand =G1(x rand ), where x rand It is a source domain other than x 1 and x 2 Another random sample, with the following additional constraints:

[0067]

[0068] Where t rand It is x rand Content information, φ(t) rand ) represents t rand The output after passing through the content extraction network.

[0069] Finally, after the interpolated data is transformed into interpolated data that retains specific content information, it is input into the decoder in the transformation network model to obtain the final interpolated image, thus augmenting the training dataset.

[0070] The above are data augmentation techniques used in image transformation. The relative position of the output is constrained based on the perceptual relationship between the inputs. Effective supervision is provided while allowing for diverse generated results. As long as the content constraints are met, the data-augmented interpolated image transformation result can be multimodal.

[0071] 2) Contrastive learning in one-way image transformation:

[0072] The inference process of the constructed transformation network model is divided into two parts: an encoder G. enc and a decoder F dec The input image is sequentially processed by an encoder and a decoder to obtain the final output image.

[0073]

[0074] Adversarial loss in one-way image transformation:

[0075] The training process of the constructed translational network model uses adversarial loss as the loss function for optimization, to encourage the output to be visually similar to the image in the target domain. The minimax loss, i.e., the adversarial loss, is as follows:

[0076]

[0077] Where D(y) represents the output of the discriminator of the transformation network model for the target domain (image); This represents the expectation for the target domain (image); G represents the expectation for the source domain (image); D represents the generator of the transformation network model; X represents the discriminator of the transformation network model; Y represents the X domain (i.e., the source domain); and Y represents the Y domain (i.e., the target domain).

[0078] The generator G minimizes this loss, while the discriminator D maximizes it.

[0079] Maximizing the relative input-output mutual information in one-way image transformation:

[0080] A noisy contrastive estimation framework is used to maximize the mutual information between the input and output images in the constructed transformation network model. The idea behind contrastive learning is to correlate two signals: a "query" signal and its corresponding "positive stimulus" example signal, contrasting them with other points in the dataset (non-positive stimulus signals, called "negative stimulus" example signals). The query signal, positive stimulus signals, and N negative stimulus signals are mapped to corresponding K-dimensional vectors (K-dimensional vectors refer to the output feature vectors of the input and output image patches after passing through the encoder and two MLP layers, i.e., feature vectors in the feature space, where K refers to the dimension of the feature vector). To prevent the expansion and collapse of the mapping space, these K-dimensional vectors are normalized. Specifically, positive and negative stimulus signals (positive and negative example vectors) are distinguished based on the feature embeddings of the signals. Given a query signal, the signal with the closest embedding vector distance to the query signal is the positive stimulus signal, and all other signals are negative stimulus signals. Therefore, it can be modeled as an (N+1) classification problem, where the distance between the query and other examples (including positive and negative examples) is scaled by a factor τ = 0.07 and passed as logit (logistic regression, the direct output of the network, without an activation function) in the constructed transformation network model. The cross-entropy loss is calculated, representing the probability of selecting a positive example instead of a negative example. The positive example vector is the feature vector of the image patch at the same location as the query image patch after passing through an encoder and two MLP layers; the negative example vector is the feature vector of image patches at other locations after passing through an encoder and two MLP layers. The formula for calculating the cross-entropy loss is as follows:

[0081]

[0082] Where τ represents the scaling factor; υ represents the query vector; v + Represents a positive example vector; υ - Represents a negative example vector; This represents the nth negative example vector; n represents the index of the negative example vector; N represents the number of negative example vectors.

[0083] The goal of setting cross-entropy loss is to correlate corresponding input and output data. In this context, the query signal refers to the output signal, while positive and negative excitation signals refer to input signals that are related to and unrelated to the output, respectively. A related example is an image patch in the input feature map that is at the same location as the query image patch, while an unrelated example is an image patch at other locations.

[0084] 3) Establish a multi-layer neural network model (transformation network model), calculate the contrast loss for each image patch, and establish the objective function:

[0085] In unsupervised learning environments, contrastive learning is applied at both the image and patch levels. In the transformation network model constructed in this embodiment, it is noted that not only do the entire input and output images (images before and after transformation) share content information, but corresponding image patches within the images also share content information. (Refer to...) Figure 3 As shown, for example, given an image patch showing the output zebra's leg, it should be more clearly associated with the corresponding leg of the input horse image than with other parts of the horse image. Even at the pixel level, the association between the zebra's body color (black and white) and the horse's body color is more obvious than its association with the background hue of the grass. Therefore, computation is performed at the image patch level, using a multilayer neural network (MLP) to establish the objective function.

[0086] Due to encoder G enc The final output image needs to be calculated, and its features can be easily obtained, so the output of the intermediate layer of the encoder is used. The input image is encoded, intermediate features are obtained, and the inference process of the transformation network model is divided into two parts: an encoder G... enc and a decoder G dec Each feature point on the feature map in each layer represents a small patch of the input image. As the number of layers increases, the feature points represent larger image patches.

[0087] The output feature maps of the Lth intermediate layer of the encoder are selected and fed into a small two-layer MLP network (fully connected multilayer neural network) to obtain feature vectors. in This represents the l-th layer of the encoder network. The relevant features are specified as... (R refers to the real number field), other irrelevant characteristics are: Where C l S represents the number of channels for each feature layer. l This refers to the number of spatial locations in each layer (the number of features in each intermediate layer). Similarly, for the generated image... Pass it through encoder G in the generator enc The intermediate feature maps of layer L are selected and processed through a small two-layer MLP network H. l The feature vector of the generated image is obtained through a fully connected multilayer neural network, as follows:

[0088]

[0089] Among them, H l This represents a two-layer MLP network; G represents the output of the l-th layer of the encoder; G represents the generator; L represents the number of intermediate layers.

[0090] For the generated image, refer to Figure 4 As shown, the intermediate feature map first passes through the encoder in the generator. For this intermediate feature map, a feature point vector at a certain location is selected (if the number of channels is C, then the vector dimension is C). After passing through two layers of MLP network, the feature vector is obtained. It is assumed that after the input image passes through the encoder in the generator, the feature vector at the same location on the feature map is passed through a two-layer MLP network to obtain the feature vector. The signal is a positive excitation signal, and the feature vectors of feature points at different locations on the same feature map are the result of passing through a two-layer MLP network. The signal is a negative excitation signal. For a specific location (feature vector) on a certain intermediate feature map of the generated image after the encoder... ), considering the feature encoding z of the input image at the same location. l for The positive excitation signal, and the feature encoding z at other positions. l 'for The negative excitation signal can be used to calculate the PatchNCE loss.

[0091] The goal is to match related input and output image patches at a specific location within an image. For a given output image, related image patches from its input image can be used as positive samples, while other image patches from its input image can be used as negative samples. For example, the output image of a zebra's leg should be more closely associated with the input image patch of a horse's leg than with other image patches from the same input image (such as other horse parts or background sky and vegetation). (See reference...) Figure 4 As shown, this is named PatchNCE loss:

[0092]

[0093] Where G represents the generator; H represents the established transformation network model; X represents the source domain; L represents the number of intermediate layers in the transformation network model; S l This represents the number of features in the l-th intermediate layer of the transformation network model; This represents the query vector (feature encoding of the output image); Indicates a positive excitation signal; Indicates a negative excitation signal; denoted by , x represents the expectation of the source domain (image); x represents the image in the source domain; X represents the source domain.

[0094] Alternatively, image patches from other images in the dataset (images other than the input image x) can be used. A random negative sample image from the dataset can be encoded into a two-layer MLP network. In this variant, an auxiliary moving average encoder, following MoCo, is used to maintain a large, consistent dictionary of negative examples, thus allowing the external NCE loss to be obtained:

[0095]

[0096] In the above formula, G represents the generator; H represents the established transformation network model; X represents the source domain; L represents the number of intermediate layers in the transformation network model; S l This represents the number of features in the l-th intermediate layer of the transformation network model; This indicates that the source domain image and negative examples are derived from the external dictionary Z. - Expectations; This represents the query vector (feature encoding of the output image); Indicates a positive excitation signal; represents the negative activation vector, from the negative example dictionary; x represents the image in the source domain; X represents the source domain; Z represents a random vector; - This represents an external dictionary used for sampling negative examples.

[0097] Among them, the negative activation vector of the dataset From external dictionary Z - Mid-sample (from the source domain), whose data is encoded using a moving average encoder. and moving average For the specific details of the calculation, please refer to the external NCE loss in MoCo.

[0098] Objective function (network optimization objective):

[0099] The transformation network model optimizes the objective function and updates the parameters of the generator and discriminator, thereby enabling the generator and discriminator to achieve the desired goal. To ensure the generated image is realistic, image patches in the input and output images should share the corresponding responses. Furthermore, PatchNCE loss can be applied to the image from the target domain Y. This is to prevent the generator from making unnecessary changes. This loss is essentially a learnable, target-domain-specific version of identity loss, commonly used in unsupervised image transformation methods. The final objective function is as follows:

[0100]

[0101] In the above formula, λ X λ represents the contrast loss coefficient of the source domain. Y The contrastive loss coefficient for the target domain is represented and needs to be manually adjusted during training; G represents the generator; H represents the established transformation network model; X represents the source domain; and Y represents the target domain.

[0102] The loss function L of generator G G for:

[0103] φ represents the content extraction network;

[0104] The loss function L of discriminator D D for:

[0105]

[0106] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A one-way image conversion method using contrastive learning and data augmentation, characterized in that, The method comprises the following steps: S1, respectively, from the source domain and the target domain, obtain a training data set for one-way image conversion, input a pre-constructed conversion network model, and perform data enhancement on the training data set based on feature interpolation; The training data set comprises: input images and output images; S2, input the data-enhanced training data set into the conversion network model for training, establish a target function through contrastive learning, and generate a trained conversion network model; S3, input an image to be converted into the trained conversion network model, and output a converted image; In the step S1, the data enhancement on the training data set based on feature interpolation comprises: S11, interpolate the training data set at the feature level, and the interpolated data is: e' = λ · e 1 + (1 - λ) · e 2 In the above formula, e' represents the interpolated data; λ represents the interpolation weight, λ ∈ [0, 1]; e 1 and e 2 represent two feature vectors, which are obtained by inputting x 1 and x 2 into an encoder in the conversion network model, wherein x 1 and x 2 represent two random samples in the source domain; S12, according to the constraint restriction, convert the interpolated data into interpolated data that maintains specific content information, input the interpolated data into a decoder in the conversion network model to obtain an interpolated image, and perform data enhancement on the training data set; the specific content information refers to the shape and position of an object in the training data set image; The constraint restriction in the step S12 is: In the above formula, represents the content loss; s' = G2(G1(e')), e' represents the interpolated data, G1 represents the first half of the generator G of the conversion network model, and G2 represents the second half of the generator G of the conversion network model; t 1 represents the content information of the x 1 ; t 2 represents the content information of the x 2 ; t rand represents the content information of the x rand , the x rand represents another random sample in the source domain except the x 1 and the x 2 ; φ(s') represents the output of the s' after the pre-constructed content extraction network; φ(t 1 ) represents the output of the t 1 after the content extraction network; φ(t 2 ) represents the output of the t 2 after the content extraction network; and φ(t rand ) represents the output of the t rand after the content extraction network. In the step S2, the target function is established through contrastive learning, comprising: S21, generate a feature map according to an intermediate layer of an encoder in the conversion network model, and output a feature vector; S22, according to the feature vector, match an image block related to the output image in the input image of the training data set, generate a positive excitation signal; and take other image blocks unrelated to the output image as negative samples, generate a negative excitation signal, and construct a PatchNCE loss function: In the above formula, G represents a generator of the conversion network model; H represents the conversion network model; X represents the source domain; L represents an intermediate layer number of the conversion network model; S l represents a feature number of an lth intermediate layer of the conversion network model; represents a feature encoding of the output image; represents the positive excitation signal; represents the negative excitation signal; represents an expectation for the source domain; x represents an image in the source domain; S23, according to the PatchNCE loss function and the adversarial loss, establish a target function: In the above formula, λ X represents the contrast loss coefficient of the source domain; λ Y represents the contrast loss coefficient of the target domain; G represents the generator of the conversion network model; H represents the conversion network model; X represents the source domain; and Y represents the target domain 2. The one-way image conversion method using contrastive learning and data augmentation of claim 1, wherein, In the step S2, the cross-entropy loss of the generated trained conversion network model is: In the above formula, τ represents a proportional factor; v represents a query vector; v + represents a positive example vector; v - represents a negative example vector; represents an nth negative example vector; n represents an index of the negative example vector; and N represents a number of the negative example vectors.

3. The one-way image conversion method using contrastive learning and data augmentation of claim 1, wherein, The step S21 comprises: Select a preset number of intermediate layers of encoders in the conversion network model to output feature maps; Input the feature maps into two layers of MLP networks in the conversion network model to output feature vectors: In the above formula, H l denotes the two-layer MLP network; denotes the output of the l-th layer network of the encoder in the conversion network model; G denotes the generator of the conversion network model; and L denotes the number of intermediate layers of the two-layer MLP network.

4. The one-way image conversion method using contrastive learning and data augmentation of claim 1, wherein, In the step S23, the adversarial loss is: In the above formula, D(y) represents the output of the discriminator of the conversion network model for the target domain; represents the expectation for the target domain; represents the expectation for the source domain; G represents the generator of the conversion network model; D represents the discriminator of the conversion network model; X represents the source domain; and Y represents the target domain.

Images