Image cross-domain migration method, computer device, readable storage medium and program product

Abstract

This application relates to an image cross-domain transfer method, computer device, readable storage medium, and program product. The method implements a network model, and the model training process includes: obtaining source and target domain images; extracting a first content feature and a first style vector from the source domain image; extracting a second content feature and a second style vector from the target domain image; combining them to obtain a first style transfer image conforming to the style of the source domain image; combining them to obtain a second style transfer image conforming to the style of the target domain image; extracting the content features of the second style transfer image and combining them with the first style vector to obtain a first source domain reconstructed image; extracting the style vector of the first style transfer image and combining it with the first content feature to obtain a second source domain reconstructed image; when the training expectation is met, the training of the image cross-domain transfer network model is completed, and the first style transfer image and / or the second style transfer image are output. This application can achieve source-to-target domain conversion and target-to-source domain conversion in one training cycle.

Description

Technical Field

[0001] This application relates to the fields of computer vision and deep learning, and in particular to an image cross-domain transfer method, computer device, readable storage medium, and program product. Background Technology

[0002] Image cross-domain transfer primarily involves style transfer between two image domains, enabling image editing and generation across different domains. It mainly includes two methods: algorithms for content transfer without deformation, and algorithms focusing on diverse cross-domain image transformations in deformation tasks. Unlike non-deformation tasks, deformation tasks require a certain degree of geometric change before and after the transformation (e.g., converting a cat face image to a dog face image; the transformed image should retain the pose and position of the cat face in the input image while possessing the texture style of the dog face). Compared to non-deformation tasks, defining the content constraint function in deformation tasks is more difficult, mainly because unsupervised algorithms are based on unpaired datasets, meaning that for a source domain image, there is no target domain image with the same pose and position.

[0003] CycleGAN makes the assumption that the output obtained by transforming an input image from the source domain to the target domain and then back from the target domain should be consistent with the input image. Using this assumption, CycleGAN implements a cycle consistency loss, which is widely used in image cross-domain transformation algorithms for deformation tasks, such as AGGAN and U-GAT-IT. However, these algorithms can only generate a single transformation result for the input source domain image, which is insufficient for application requirements. To obtain diverse generation results, DRIT and MUNIT use decoupling to learn the content and style of the image separately. During transformation, they combine different images in the target domain with the input source domain image to obtain diverse transformation results. However, the content and style of the image cannot be accurately defined, thus posing a challenge to decoupling. Therefore, this type of algorithm still suffers from two problems: 1) the transformed image has significant content differences (mainly in pose) compared to the corresponding source domain image; 2) the texture similarity between the transformed image and the sample image is low. Furthermore, they also exhibit varying degrees of "artifacts" in the generated images, further reducing image quality.

[0004] Both DRIT and MUNIT recombine the content and style of the generated image, but the generated image suffers a certain degree of quality degradation compared to the real image, and the content and style obtained are correspondingly distorted. Summary of the Invention

[0005] Therefore, it is necessary to provide an image cross-domain transfer method to address the aforementioned technical problems.

[0006] This application's image cross-domain transfer method performs style transfer on an image between two image domains and is implemented in an image cross-domain transfer network model. The training process of the image cross-domain transfer network model includes:

[0007] Obtain source and target domain images, extract the first content features and first style vector of the source domain image, and extract the second content features and second style vector of the target domain image;

[0008] By combining the second content feature and the first style vector, a first style transfer image that conforms to the style of the source domain image is obtained;

[0009] By combining the first content feature and the second style vector, a second style transfer image that conforms to the style of the target domain image is obtained;

[0010] Extract the content features of the second style transfer image and combine them with the first style vector to obtain the first source domain reconstructed image;

[0011] Extract the style vector from the first style transfer image and combine it with the first content feature to obtain the second source domain reconstructed image;

[0012] When the training expectation is met, the training of the image cross-domain transfer network model is completed. The training expectation includes that the source domain image, the first source domain reconstructed image, and the second source domain reconstructed image satisfy the loss constraint condition.

[0013] Using the trained image cross-domain transfer network model, output the first style transfer image and / or the second style transfer image.

[0014] Optionally, the training process of the image cross-domain transfer network model further includes:

[0015] Extract the content features of the first style transfer image and combine them with the second style vector to obtain the first target domain reconstructed image;

[0016] The style vector of the second style transfer image is extracted and combined with the second content feature to obtain the second target domain reconstructed image;

[0017] The training expectation also includes that the target domain image, the first target domain reconstructed image, and the second target domain reconstructed image satisfy the loss constraint condition.

[0018] Optionally, the loss constraint used is a cross-cyclic consistency loss.

[0019] Optionally, the image cross-domain transfer network model includes a content encoder, and the image cross-domain transfer method further includes: extracting corresponding content features using the content encoder, wherein the content encoder includes multiple convolutional layers, and after passing through a convolutional layer, normalization is performed sequentially using parameterless instance normalization and activation is performed using the ReLU function.

[0020] Optionally, the image cross-domain transfer method further includes: extracting corresponding content features using a content encoder, wherein the content encoder includes a preprocessing layer, a downsampling layer, and a residual layer, and the downsampling layer includes a compression extraction module, wherein the compression extraction module includes:

[0021] The spatial compression-channel extraction module performs global average pooling on the input features and then obtains a weight vector through a nonlinear function, which is used to represent the scaling scale of each channel in the globally observed input features.

[0022] The channel compression-spatial extraction module convolves the input features to obtain a spatial attention map, and then uses the Sigmoid activation function to obtain the scaling scale.

[0023] The output characteristic of the compression extraction module is the larger value output by the spatial compression-channel extraction module and the channel compression-spatial extraction module.

[0024] Optionally, the image cross-domain transfer network model includes a style encoder, and the image cross-domain transfer method further includes: extracting the corresponding style vector using the style encoder, wherein the style encoder sequentially includes a preprocessing layer, a downsampling layer, a pooling layer, and a convolutional layer.

[0025] Optionally, the image cross-domain transfer network model includes a decoder, and the image cross-domain transfer method further includes: using the decoder to combine content features and style vectors, and obtaining the corresponding image through merging and connecting;

[0026] The decoder comprises a residual layer, an upsampling layer, and a convolutional layer in sequence. The upsampling layer includes transposed convolution, layer normalization, and the compression extraction module.

[0027] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the image cross-domain migration method described in this application.

[0028] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image cross-domain migration method described in this application.

[0029] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the image cross-domain migration method described in this application.

[0030] The image cross-domain transfer method of this application has at least the following effects:

[0031] This application comprises two parts: forward transformation and inverse transformation. Forward transformation includes decomposing and combining the source domain image and the target domain image into a first style transfer image and a second style transfer image. Inverse transformation includes decomposing the first style transfer image and the second style transfer image, and then combining them with the decomposition results of the source domain image and the target domain image to generate reconstructed source domain image and target domain image.

[0032] This application inherently achieves bidirectional cross-domain processing with multiple constraints, enabling balanced network training and reducing pattern collapse compared to unidirectional algorithms. By controlling training stability through constraint loss on the reconstructed image, it improves the transfer quality under conditions of varying content structure during cross-domain image transfer.

[0033] The image cross-domain transfer network model trained in this application can simultaneously obtain a first-style transfer image that conforms to the style of the source domain image and a second-style transfer image that conforms to the style of the target domain image. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating an image cross-domain migration method in one embodiment of this application;

[0035] Figure 2 This is a network model diagram of an image cross-domain migration network used to implement an image cross-domain migration method in one embodiment of this application;

[0036] Figure 3 for Figure 2 A schematic diagram of the network structure of the content encoder;

[0037] Figure 4 This is a model diagram of the compression extraction module used in the image cross-domain migration method in one embodiment of this application;

[0038] Figure 5 for Figure 4 The corresponding network architecture diagram;

[0039] Figure 6 for Figure 2 Network architecture diagram of a medium-style encoder;

[0040] Figure 7 for Figure 2 Network architecture diagram of the decoder;

[0041] Figure 8This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0042] Existing technologies recombine the content and style of generated images, but the generated images suffer from a certain degree of quality degradation compared to real images, and the content and style obtained are correspondingly distorted.

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] To solve the above technical problems, see [link to relevant documentation]. Figure 1 and Figure 2 This application provides an image cross-domain transfer method in one embodiment, which performs style transfer on an image between two image domains (a source domain and a target domain whose relative positions can be interchanged). The method is implemented in an image cross-domain transfer network model, and the training process of the image cross-domain transfer network model includes steps S100 to S500. Wherein:

[0045] Step S100: Obtain the source domain (domain A) image and the target domain (domain B) image, and extract the first content feature A from the source domain image. c and the first style vector A s Extract the second content feature B from the target domain image. c Second style vector B s ;

[0046] Step S200, combine the second content feature B c and the first style vector A s Obtain a first style transfer image that conforms to the style of the source domain image. Combining the first content feature A c Second style vector B s Obtain a second style transfer image that conforms to the style of the target domain image.

[0047] Step S300: Extract the second style transfer image. Content features With the first style vector A s Combine to obtain the reconstructed image of the first source domain Extracting the first style transfer image Style vectors With the first content feature A c Combined to obtain the reconstructed image of the second source domain

[0048] Step S400: When the training expectation is met, the training of the image cross-domain transfer network model is completed. The training expectation includes the source domain image and the first source domain reconstructed image. Second source domain reconstructed image All three satisfy the loss constraint conditions;

[0049] Step S500: Using the trained image cross-domain transfer network model, output the first style transfer image. and / or second style transfer images

[0050] In this embodiment, steps S100 and S200 are forward transformations, including decomposing and combining the source domain image and the target domain image into a first style transfer image. Second style transfer image Step S300 is a reverse transformation, which includes transferring the first style image. Second style transfer image After decomposition, the results are recombined with the decomposition results of the source and target domain images to generate reconstructed source and target domain images. Step S400 reveals the training completion criteria. Meeting the constraints during training can be understood as ensuring that the images are essentially identical after training. For example, after training, the source domain image A and the first reconstructed source domain image... Second source domain reconstructed image Based on consistency.

[0051] Although this embodiment is also based on the decoupling concept, the proposed image cross-domain transfer method uses a different combination method than DRIT and MUNIT. The specific difference lies in the "reverse transformation" process provided in step S300. In this embodiment, the content features (style vectors) obtained from the generated image are combined with the style vectors (content features) of the corresponding source domain image to generate the reconstructed image.

[0052] When the reconstructed image meets the constraints, it demonstrates the reliability of the forward and reverse transformations. In subsequent use, only the forward transformation part needs to be used for style transfer (corresponding to step S500).

[0053] This embodiment naturally achieves bidirectional cross-domain processing and has multiple constraints. Compared to unidirectional algorithms, it can balance network training and reduce pattern collapse. This application uses constraint loss on the reconstructed image to control training stability, improving the transfer quality under conditions of variable content structure during cross-domain image transfer.

[0054] This embodiment utilizes a pair of content encoders and style encoders for both the source and target domains to decouple the learning of content features and style vectors in the source and target domain images, respectively. Based on a Siamese network, bidirectional cross-domain transformation is achieved, meaning that a single training iteration can realize both source-to-target and target-to-source domain transformations. In other words, a first style transfer image conforming to the style of the source domain image can be obtained with a single training iteration. It can also obtain a second style transfer image that conforms to the style of the target domain image.

[0055] The image cross-domain transfer method provided in this embodiment can transfer an image from one image domain to another while preserving the content of the source image domain. The image cross-domain transfer network model used in this embodiment is a Siamese network model, which can be used for content and style decoupling learning (separating the content and style of an image) to train the image cross-domain transfer network model. After training, the image cross-domain transfer network model can be used to decouple the content and style of input images from other domains that have not been learned, and then recombine the decoupled content and style to achieve cross-domain image transfer under small sample conditions.

[0056] Furthermore, step S300 also includes extracting the first style transfer image. Content features With the second style vector B s Combine to obtain the first target domain reconstructed image Extracting second style transfer images Style vectors With the second content feature B c Combine to obtain the reconstructed image of the second target domain

[0057] In step S400, the training expectation also includes the target domain image and the first target domain reconstructed image. Second target domain reconstructs the image All three satisfy the loss constraint condition. The loss constraint condition used in step S400 is specifically the cross-cyclic consistency loss, the details of which can be found in other embodiments.

[0058] Furthermore, the training expectation also includes at least one of the following: (1) second style transfer images Content features With the first content feature A c (2) First style transfer image satisfies loss constraints; Style vectors With the first style vector A s (3) Second style transfer image satisfies loss constraints; Style vectors With the second style vector B s (4) First style transfer image satisfies the loss constraint conditions; Content features With the second content feature B c The loss constraint is satisfied.

[0059] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0060] In one embodiment, the implementation process of steps S100 to S400 in the image cross-domain migration network model is explained in conjunction with the image cross-domain migration network model.

[0061] The image cross-domain transfer network model includes a content encoder, a style encoder, and a decoder. The image cross-domain transfer method further includes: extracting relevant content features using the content encoder, extracting relevant style vectors using the style encoder, combining the content features and style vectors using the decoder, and obtaining the corresponding image through merging and concatenation. The relevant content features and style vectors refer to the features extracted in steps S100 to S400 for different images, such as the source domain image, the target domain image, and the first style-transfer image. Second style vector B s Second style transfer images Extraction is performed using the corresponding content encoder and style encoder.

[0062] The content encoder includes multiple convolutional layers. After each convolutional layer, normalization is performed using parameterless instance normalization, followed by activation using the ReLU function. The content encoder includes a preprocessing layer, a downsampling layer, and a residual layer. The downsampling layer includes a compression extraction module. For details on the relative positions and inclusion relationships of the convolutional layers, preprocessing layer, downsampling layer, and residual layer of the content encoder, please refer to other embodiments.

[0063] The style encoder consists of a preprocessing layer, a downsampling layer, a pooling layer, and a convolutional layer. The decoder consists of a residual layer, an upsampling layer, and a convolutional layer. The upsampling layer includes transposed convolution, layer normalization, and a compression extraction module.

[0064] The compression extraction module includes: (1) a spatial compression-channel extraction module, which performs global average pooling on the input features and then obtains a weight vector through a nonlinear function to represent the scaling scale of each channel in the globally observed input features; and (2) a channel compression-spatial extraction module, which performs convolution on the input features to obtain a spatial attention map and then obtains the scaling scale through a Sigmoid activation function. The output feature of the compression extraction module is the larger value of the outputs of the spatial compression-channel extraction module and the channel compression-spatial extraction module.

[0065] This embodiment utilizes a "compression-extraction" module (i.e., compression-extraction module) in the generator based on the idea of ​​"weight readjustment." This module comprehensively uses spatial global information and the relationships between channels to adjust feature weights, highlighting important regions in the image and improving the quality of the transformed image. The "compression-extraction" module network focuses more on the content instances in the image that need to be transformed at the feature level, achieving cross-domain image transformation in deformation tasks.

[0066] Specifically, in conjunction with the image cross-domain transfer network model, this embodiment first obtains the content features of the source domain image and the style vector of the target domain image through a content encoder and a style encoder, respectively. Before the content features are fused, they go through a "compression-extraction" module, which can adjust the weights of the feature maps so that the network focuses on the content region that needs to be transformed. Then, the adjusted content features and style vectors are fused by concat merging and connecting, and the fused features are then decoded into the image space.

[0067] See Figures 2-7 In one embodiment, the image cross-domain transfer method is described in detail in conjunction with an image cross-domain transfer network model. This mainly includes: 1. Design of the image cross-domain transfer network model; 2. Design of constraints for image cross-domain transfer; 3. Training and parameter tuning of the image cross-domain transfer network model; 4. Performance evaluation of the image cross-domain transfer network model. These are described in detail below:

[0068] I. Design of Image Cross-Domain Transfer Network Model

[0069] See Figure 2 For ease of display, Figure 2The discriminator is omitted. The image cross-domain transfer network model in this embodiment is based on Siamese networks. To improve the quality of the images generated by the method, the algorithm is implemented based on GAN. It uses a pair of identical network models for the two domains (source domain and image domain), which we call "Siamese networks". Each network includes a content encoder, a style encoder, and a decoder. A "compression-extraction" module is used in each downsampling module in the content encoder. Compared with the unidirectional algorithm, the bidirectional cross-domain (forward transformation + backward transformation) algorithm in this embodiment can balance network training and reduce mode collapse to a certain extent.

[0070] During training, any pair of source and target domain images are first input into the corresponding image domain content encoder and style encoder to obtain their respective content features and style vectors, denoted as A. c A s B c B s For simplicity, this explanation uses the source-to-target domain conversion as an example. The decoder will convert A... c and B s Combine to generate images Generate image The discriminator in the target domain calculates the adversarial loss by inputting the target domain image B and the target domain image B, while also generating an image. The content encoder and style encoder of the target domain are input again to obtain its content features. and style vectors Will and A s Combined decoding to obtain the reconstructed image in image space At the same time, A c and Combined decoding to obtain the reconstructed image in image space Using source domain image A, Calculate the cross-cycle consistency loss. Similarly, we can obtain... Furthermore, regarding B, The cross-cyclic consistency loss is calculated. In summary, this paper uses a GAN-based Siamese network structure to perform bidirectional cross-domain image transformation through adversarial loss and cross-cyclic consistency loss.

[0071] See Figure 3The content encoder network structure is shown in the figure. The content encoder of this invention includes a preprocessing layer, two downsampling layers, and four residual layers. A "compression-extraction" module is used after each downsampling layer. Because the network structure and feature input / output dimensions of each residual layer are the same, the four residual layers are merged in the figure, and the last residual layer is shared by both domains. To avoid the content encoder being affected by extreme values ​​in the samples and to accelerate network convergence, and because the content of each sample is crucial for the learning of the generative model, this algorithm uses parameterless instance-normalized features for normalization after each convolutional layer in the content encoder, followed by ReLU activation. Where x... nchw μ represents the feature value with coordinates (h, w) in the c-th channel of the n-th sample. nc and σ nc The mean and standard deviation represent the features.

[0072] See Figure 4 Unlike traditional downsampling modules, this algorithm uses a "compression-extraction" module to enable the content encoder to focus on the content regions that need to be transformed, thereby increasing the weights of the corresponding features. Its network model is as follows: Figure 4 As shown, convolution operations implicitly incorporate the spatial and channel relationships between features. This hybrid learning mode, to some extent, limits the learning ability of the network model, i.e., the representational power of features decreases. "Compression-extraction," by adjusting weights, decouples the spatial and channel relationships of features, thereby enhancing the representational power of features.

[0073] See Figure 5 The "Compression-Extraction" module is divided into two types: one is "Spatial Compression and Channel Excitation" (cSE), such as... Figure 4 As shown in the upper half, it first performs global average pooling on the input features to obtain the representation information of the maximum receptive field in the channels. This information is then passed through a nonlinear function to obtain a weight vector, which is a global description relative to the input features, that is, it represents the scaling scale of each channel in the input features under global observation.

[0074] Another type is "Channel Squeeze and Spatial Excitation" (sSE), such as... Figure 4The lower half is shown. This operation first uses a 1×1 convolution on the input features to obtain a spatial attention map. Based on the properties of the 1×1 convolution operation, it is equivalent to weighted summation of feature values ​​at the same location in each channel, and then obtaining the scaling factor at that location through a sigmoid activation function. This algorithm uses both "compression-extraction" modules simultaneously, taking the larger value from their outputs as the final output feature of that module. Its network structure is as follows: Figure 5 As shown.

[0075] See Figure 6 The network structure of the style encoder is shown in the figure. The style encoder aims to encode an image into an N-dimensional style vector. In the experiment, N=8. The style encoder includes a preprocessing layer, three downsampling layers, a pooling layer and a 1×1 convolutional layer.

[0076] See Figure 7 The network structure of the decoder is shown in the figure. The decoder first needs to consider how to combine content features and style features. This algorithm uses the "Con-cat" method. There is no explicit style loss constraint in the decoder. In order to inject style features into the content more flexibly and effectively, the algorithm first uses a multilayer perceptron (MLP) to learn the style vector to dynamically obtain the features required for the Concat operation. Then it is used in four residual layers. Therefore, the input channel dimension of Conv2 and Conv5 layers in the residual layers is twice the output channel dimension of the previous layer. Afterwards, the decoder restores it to the resolution of the input image through two upsampling layers. Finally, it reduces the dimensionality to the image space through a convolutional layer. The upsampling layer consists of transposed convolution, layer normalization, and a "compression-extraction" module (consistent with the usage in the content encoder).

[0077] Because the decoder and content encoder have a symmetrical structure, the network uses reverse encoding to make it easier to understand. Similarly, the four residual layers have the same structure and feature input and output dimensions, so they are presented in a merged manner.

[0078] Discriminators (omitted in the diagram): Multiple discriminators are used to gradually increase the receptive field through downsampling, thus taking into account both local and global information. The structures of each discriminator are completely identical, overcoming the drawbacks of L1 or L2 loss in smoothing images and thus losing high-frequency information. This encourages the generator to produce high-resolution images with sharp edges. The discriminator scale used in this algorithm is 3, meaning three identical discriminators are used to downsample the image, resulting in different resolution inputs for each discriminator. Therefore, with identical discriminators, the smaller the image resolution, the larger the receptive field of the discriminator, achieving the effect of discrimination at both local and global levels. The input image resolution for the first discriminator is 256×256, for the second discriminator it is 128×128, and for the third discriminator it is 64×64.

[0079] II. Design of constraints for cross-domain image migration.

[0080] This invention uses adversarial loss for cross-domain constraints and cross-cyclic consistency loss for content structure constraints. Cross-cyclic consistency loss mainly imposes the following constraints: by decoupling an image into content features and style vectors, combining the content features of any image in the source domain with the style vector of an image in the target domain can generate an image that conforms to the style of the target domain.

[0081] To achieve this decoupled learning, the algorithm calculates cross-cyclic consistency through "forward transformation" and "backward transformation." "Forward transformation" refers to converting an image from the source domain to an image in the target domain; that is, for any image A from domain A (the source domain), we first transform it through the content encoder E... C A. Style Encoder E S A extracts the content features of the image. c and style vector A s Similarly, for any image B from domain B (target domain), B is extracted accordingly. c B s Then, two "forward transformations" are performed, as shown in equations (1) and (2). Where D... A and D B It is a decoder corresponding to two fields.

[0082]

[0083]

[0084] "Inverse transformation" is the process of transforming the image in the target domain back into the source domain. and Before performing the "reverse conversion", we also need to extract... and The content characteristics and style characteristics are respectively denoted as

[0085] It can be seen that, Should be with A c Consistent, A s If consistent with, then and A s By combining decoding to the image space, image A should be reconstructed, denoted as... At the same time, A c and The decoding process should also be able to reconstruct image A, denoted as... Similarly, we can obtain the reconstructed image of B. and Right now

[0086]

[0087]

[0088]

[0089]

[0090] Cross-cyclic consistency loss is essentially a reconstruction loss, that is, it uses L1 loss to constrain the source image and the reconstructed image obtained by the "inverse transform". This includes equations (7) and (8).

[0091]

[0092]

[0093] in,

[0094] The concepts of content encoder, style encoder, and decoder in the formula will not be elaborated again. For the images in each stage of the image cross-domain transfer method process, please refer to the above explanation. That is, loss constraints are applied to image A and its two reconstructed images. That is, loss constraints are applied to image B and its two reconstructed images.

[0095] This embodiment is based on a Siamese network, therefore each domain A and B has a corresponding discriminator to distinguish between generated and real data, using the adversarial loss of LSGAN. The total loss function is: The value of 1 ranges from 0 to 1, while the value of λ2 ranges from 2 to 5.

[0096] III. Training and Parameter Tuning of the Image Cross-Domain Transfer Network Model

[0097] During the training phase, both the source and target domain images are first scaled to 286×286, and then randomly cropped to 256×256 to make the model more robust. This algorithm simultaneously performs transformations from A to B and from B to A. Let the source domain image be A and the target domain image be B: First, A and B are input into their respective generators to obtain... and Will and The corresponding discriminator is input and its corresponding real image is used to calculate the adversarial loss, and then... and The reconstructed image is obtained by fusing the content features and style vectors obtained from the encoder with the style vectors and content features obtained from the corresponding real image and decoding them. The generator is trained using adversarial loss and cross-cyclic consistency loss, while the discriminator is trained using only adversarial loss.

[0098] In the experiment, the Adam optimizer was used to optimize the generator and discriminator. The initial learning rate was 0.0001, and the Mo-mentum parameters were 0.5 and 0.999, respectively. Due to memory limitations, the batch size was set to 1, and the hyperparameters in the loss function were set to 1.0 and 10.0, respectively.

[0099] During the testing phase, you only need to use the trained generator model and input the source domain image and the target domain image into the generator to obtain the generated image.

[0100] IV. Performance Evaluation of Image Cross-Domain Transfer Network Model

[0101] To demonstrate the effectiveness of the method, a comprehensive ablation study was conducted to show how each component affects the final result. Experimental results were obtained after conducting experiments on two deformed datasets and one non-deformed dataset, following the experimental procedure described in the previous section. Then, the results of comparative experiments on deformed datasets were presented, including qualitative and quantitative comparisons, between the proposed algorithm and leading cross-domain image algorithms in recent years. Finally, an ablation experiment was conducted on the "compression-extraction" module used in this algorithm to verify its effectiveness.

[0102] Furthermore, the performance was evaluated on metrics such as NDB, JS, and LPIPS. NDB measures the degree to which the generated image fits the target domain image while maintaining image realism. JSD calculates the JS divergence between the distributions of the generated and target domain images, also considering image quality. Therefore, this algorithm exhibits better diversity when the image is required to be as natural and realistic as possible. LPIPS, on the other hand, only measures the distance between generated images. It is very sensitive to "artifacts" in the image; that is, the lower the image quality, the higher the LPIPS sometimes is.

[0103] The image cross-domain transformation algorithms in the various embodiments of this application are based on Siamese networks. For each domain, a pair of content encoders and style encoders are used to decouple the learning of image content and style. A "compression-extraction" module is used to make the network focus more on the content instances in the image that need to be transformed at the feature level, achieving image cross-domain transformation in deformation tasks. First, the encoders obtain the content features of the source domain and the style vector of the target domain. Before fusion, the content features pass through a "compression-extraction" module, which adjusts the weights of the feature maps to make the network focus on the content regions that need to be transformed. Then, the adjusted content features and style vectors are fused using Concat, and the fused features are decoded into the image space. This algorithm, based on Siamese networks, achieves bidirectional cross-domain transformation, that is, a single training can achieve transformation from the source domain to the target domain and transformation from the target domain to the source domain.

[0104] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 8 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an image cross-domain migration method. The display screen can be a liquid crystal display (LCD) or an e-ink display. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0105] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0106] Step S100: Obtain source domain image and target domain image, extract first content features and first style vector from source domain image, and extract second content features and second style vector from target domain image;

[0107] Step S200: Combine the second content feature and the first style vector to obtain a first style transfer image that conforms to the style of the source domain image; combine the first content feature and the second style vector to obtain a second style transfer image that conforms to the style of the target domain image.

[0108] Step S300: Extract the content features of the second style transfer image and combine them with the first style vector to obtain the first source domain reconstructed image; extract the style vector of the first style transfer image and combine it with the first content features to obtain the second source domain reconstructed image;

[0109] Step S400: When the training expectation is met, the training of the image cross-domain transfer network model is completed. The training expectation includes that the source domain image, the first source domain reconstructed image, and the second source domain reconstructed image meet the loss constraint conditions.

[0110] Step S500: Using the trained image cross-domain transfer network model, output the first style transfer image and / or the second style transfer image.

[0111] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0112] Step S100: Obtain source domain image and target domain image, extract first content features and first style vector from source domain image, and extract second content features and second style vector from target domain image;

[0113] Step S200: Combine the second content feature and the first style vector to obtain a first style transfer image that conforms to the style of the source domain image; combine the first content feature and the second style vector to obtain a second style transfer image that conforms to the style of the target domain image.

[0114] Step S300: Extract the content features of the second style transfer image and combine them with the first style vector to obtain the first source domain reconstructed image; extract the style vector of the first style transfer image and combine it with the first content features to obtain the second source domain reconstructed image;

[0115] Step S400: When the training expectation is met, the training of the image cross-domain transfer network model is completed. The training expectation includes that the source domain image, the first source domain reconstructed image, and the second source domain reconstructed image meet the loss constraint conditions.

[0116] Step S500: Using the trained image cross-domain transfer network model, output the first style transfer image and / or the second style transfer image.

[0117] In one embodiment, a computer program product is provided, including computer instructions that, when executed by a processor, perform the following steps:

[0118] Step S100: Obtain source domain image and target domain image, extract first content features and first style vector from source domain image, and extract second content features and second style vector from target domain image;

[0119] Step S200: Combine the second content feature and the first style vector to obtain a first style transfer image that conforms to the style of the source domain image; combine the first content feature and the second style vector to obtain a second style transfer image that conforms to the style of the target domain image.

[0120] Step S300: Extract the content features of the second style transfer image and combine them with the first style vector to obtain the first source domain reconstructed image; extract the style vector of the first style transfer image and combine it with the first content features to obtain the second source domain reconstructed image;

[0121] Step S400: When the training expectation is met, the training of the image cross-domain transfer network model is completed. The training expectation includes that the source domain image, the first source domain reconstructed image, and the second source domain reconstructed image meet the loss constraint conditions.

[0122] Step S500: Using the trained image cross-domain transfer network model, output the first style transfer image and / or the second style transfer image.

[0123] In this embodiment, the computer program product includes a program code portion for performing the steps of the image cross-domain migration method in various embodiments of this application when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. It may also be provided for download via a data network (e.g., via a RAN, via the Internet, and / or via an RBS). Alternatively or additionally, the method may be encoded in a field-programmable gate array (FPGA) and / or an application-specific integrated circuit (ASIC), or its functionality may be provided for download by means of a hardware description language.

[0124] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0125] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered to be within the scope of this specification. When technical features of different embodiments are embodied in the same drawing, it can be regarded as the drawing also disclosing examples of combinations of the various embodiments involved.

[0126] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image cross-domain transfer method, which performs style transfer on an image between two image domains and is implemented in an image cross-domain transfer network model, characterized in that... The training process of the image cross-domain transfer network model includes: Obtain source and target domain images, extract the first content features and first style vector of the source domain image, and extract the second content features and second style vector of the target domain image; By combining the second content feature and the first style vector, a first style transfer image that conforms to the style of the source domain image is obtained; By combining the first content feature and the second style vector, a second style transfer image that conforms to the style of the target domain image is obtained; Extract the content features of the second style transfer image and combine them with the first style vector to obtain the first source domain reconstructed image; Extract the style vector from the first style transfer image and combine it with the first content feature to obtain the second source domain reconstructed image; When the training expectation is met, the training of the image cross-domain transfer network model is completed. The training expectation includes that the source domain image, the first source domain reconstructed image, and the second source domain reconstructed image satisfy the loss constraint condition. Using the trained image cross-domain transfer network model, output the first style transfer image and / or the second style transfer image.

2. The image cross-domain migration method as described in claim 1, characterized in that, The training process of the image cross-domain transfer network model also includes: Extract the content features of the first style transfer image and combine them with the second style vector to obtain the first target domain reconstructed image; The style vector of the second style transfer image is extracted and combined with the second content feature to obtain the second target domain reconstructed image; The training expectation also includes that the target domain image, the first target domain reconstructed image, and the second target domain reconstructed image satisfy the loss constraint condition.

3. The image cross-domain migration method as described in claim 2, characterized in that, The loss constraint used is the cross-cyclic consistency loss.

4. The image cross-domain migration method as described in claim 1, characterized in that, The image cross-domain transfer network model includes a content encoder, and the image cross-domain transfer method further includes: extracting corresponding content features using the content encoder, wherein the content encoder includes multiple convolutional layers, and after passing through a convolutional layer, normalization is performed sequentially using parameterless instance normalization and activation is performed using the ReLU function.

5. The image cross-domain migration method as described in claim 1, characterized in that, The image cross-domain transfer method further includes: extracting corresponding content features using a content encoder, wherein the content encoder includes a preprocessing layer, a downsampling layer, and a residual layer, and the downsampling layer includes a compression extraction module, wherein the compression extraction module includes: The spatial compression-channel extraction module performs global average pooling on the input features and then obtains a weight vector through a nonlinear function, which is used to represent the scaling scale of each channel in the globally observed input features. The channel compression-spatial extraction module convolves the input features to obtain a spatial attention map, and then uses the Sigmoid activation function to obtain the scaling scale. The output characteristic of the compression extraction module is the larger value output by the spatial compression-channel extraction module and the channel compression-spatial extraction module.

6. The image cross-domain migration method as described in claim 5, characterized in that, The image cross-domain transfer network model includes a style encoder, and the image cross-domain transfer method further includes: extracting the corresponding style vector using the style encoder, wherein the style encoder sequentially includes a preprocessing layer, a downsampling layer, a pooling layer, and a convolutional layer.

7. The image cross-domain migration method as described in claim 5, characterized in that, The image cross-domain transfer network model includes a decoder, and the image cross-domain transfer method further includes: using the decoder to combine content features and style vectors, and obtaining the corresponding image through merging and connection; The decoder comprises a residual layer, an upsampling layer, and a convolutional layer in sequence. The upsampling layer includes transposed convolution, layer normalization, and the compression extraction module.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the image cross-domain migration method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the image cross-domain migration method according to any one of claims 1 to 7.

10. A computer program product, including computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the image cross-domain migration method according to any one of claims 1 to 7.

Images