Image translation method fused with multi-branch neural network

Abstract

The invention discloses an image translation method fused with a multi-branch neural network. The method comprises the following steps: constructing a semi-pairing image conversion model, reconstructing an original image, adding a label to the original image, and taking the reconstructed image as label-free data; constructing a multi-branch image translation model, inputting the original image into the multi-branch image translation model, and extracting a first feature from the original image by a first branch of the multi-branch image translation model through depth feature difference and an attention mechanism; a second branch extracts a second feature from the original image through a style attention mechanism and self-normalization; fusing the extracted first feature and the second feature to generate a multi-modal feature, and generating a target image; and training the model by using label data, and evaluating the accuracy of the model by using label-free data. According to the technical scheme, the problem that the efficiency of an existing image conversion method is not high under the condition that paired images are rare is effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision, and in particular relates to an image translation method integrating a multi-branch neural network. Background Art

[0002] Image translation is a widely studied field in computer vision, and many problems in the field ultimately fall under the umbrella of image translation. Image translation encompasses many classic problems in computer vision and computer graphics, and holds significant scientific research value. Image translation models using generative adversarial networks have already been applied in numerous industrial scenarios, such as fashion editing, makeup transfer, and autonomous driving scene segmentation. In addition to improving people's quality of life and work efficiency, image translation research also offers significant commercial value, boosting industry economic growth and driving regional GDP. Future research on image translation will have even broader potential.

[0003] Pix2Pix is ​​a classic image translation model based on supervised learning. Supervision here refers to the use of paired images in training. Training the Pix2Pix model requires a dataset with fully paired images. For example, for seasonal translation tasks, the Pix2Pix model requires a large number of images of the same scene, location, and seasons across different seasons. Producing such a dataset undoubtedly consumes significant labor costs.

[0004] Unlike supervised learning image translation methods, unsupervised learning image translation methods do not require any additional labels beyond domain information during model training. The biggest challenge with supervised learning image translation methods is that in practice, the available paired image data for model training is quite limited. Consequently, an increasing number of unsupervised image translation methods have been proposed.

[0005] While the aforementioned methods address related issues in their respective fields, we have observed the following shortcomings: First, neither supervised nor unsupervised learning methods can effectively address the image translation problem when paired images are scarce. Second, existing image translation methods lack diversity in their generated results, or this diversity cannot be expressed at the semantic level. Third, existing reference image-guided image translation methods suffer from low style transfer accuracy, lack visual realism, and struggle to strike a balance between content and style. Summary of the Invention

[0006] The present invention provides an image translation method integrating a multi-branch neural network, aiming to solve the above-mentioned problems.

[0007] The present invention provides an image translation method integrating a multi-branch neural network, comprising:

[0008] Construct a semi-paired image conversion model to reconstruct the original image and add labels to the original image. The reconstructed image is used as unlabeled data. The original image includes: paired and no neighbor images, paired and neighbor images, unpaired and no neighbor images, and unpaired and neighbor images.

[0009] A multi-branch image translation model is constructed. The original image is input into the multi-branch image translation model. The first branch of the multi-branch image translation model extracts the first feature from the original image through deep feature difference and attention mechanism; the second branch extracts the second feature from the original image through style attention mechanism and self-normalization.

[0010] Fusing the extracted first feature with the second feature to generate a multimodal feature and generate a target image;

[0011] The multi-branch image translation model is trained using labeled data, and the accuracy of the multi-branch image translation model is evaluated using unlabeled data.

[0012] Furthermore, the semi-paired image conversion model consists of two generators and two discriminators. The first generator G xy Corresponding to the image conversion network from style X to style Y, learning the mapping from domain X to domain Y; the first discriminator D y Used to distinguish these generated Y domain images from the actual Y domain images; the second generator G yx Corresponding to the image conversion network from domain Y to domain X; the second discriminator D x Used to distinguish these generated X-domain images from actual X-domain images.

[0013] Furthermore, for the first branch, first, a style mapping network is used to generate a style code from the latent variable, and then the style code and the original image are input into the generator for image conversion to obtain multiple generated images; secondly, the deep feature differences between the generated images are obtained by using a pre-trained deep feature extraction network; then, based on the deep feature differences, the results of generating different style codes for the same input image are controlled to have differences in semantic features.

[0014] Furthermore, the deep feature extraction network uses a pre-trained AlexNet; the training process of the deep feature extraction network includes: first, extracting the deep features of each generated image as the first feature, then calculating the distance between the first features to measure the difference between the images, and by maximizing the difference between the generated images, forcing the generator to explore the high-level semantic space of the image and generate images with semantic diversity.

[0015] Furthermore, a channel-domain-based attention mechanism is adopted in the downsampling stage of the generator to obtain the importance of each feature channel through deep learning, and then weighted to the first feature channel by channel through multiplication to complete the recalibration of the first feature in the channel dimension.

[0016] Furthermore, for the second branch, first, a style mapping network is used to generate a style code from the latent variable. A style attention mechanism is introduced into the style mapping network, and a learnable soft attention network is used to convert the content feature F C Mapped to style feature F S The style feature is used as the second feature to obtain a style code with attention to the source image content, and then the style code and the original image are input into the generator for image conversion.

[0017] Furthermore, the content feature F C Mapped to style feature F S The weights include: first, the content feature F C and style characteristics F S Perform normalization and then convert to feature space f and g, calculate the attention between f and g, and use the content feature F C and style characteristics F S The relationship between , embeds a style feature at each position of the content feature.

[0018] Furthermore, the original image is input as a training set into a multi-branch image translation model to train the model. The first branch extracts the first feature, the second branch extracts the second feature, and the first feature and the second feature are input into the generator together to generate the target image.

[0019] Furthermore, the training of the multi-branch image translation model using the label data includes: calculating the accuracy of the model;

[0020]

[0021] Among them, Acc represents the accuracy of the model, TP represents the true positive examples in the sample, FP represents the false positive examples in the sample, FN represents the false negative examples in the sample, and TN represents the true negative examples in the sample.

[0022] Compared with the prior art, the present invention has the following beneficial effects:

[0023] (1) In response to the problem that it is difficult to obtain a fully paired image dataset, the present invention proposes a semi-paired image conversion model that combines neighbor information. This model combines unsupervised and supervised methods, which not only overcomes the limitation that the training dataset needs to be fully paired, but also can largely guarantee the image conversion quality. In addition, the neighbor information cycle consistency constraint proposed by the model aims to explore the association between image data as much as possible, thereby providing more guiding information for the training of the image translation model. Compared with existing image translation methods, this model is more flexible and versatile, and is applicable to a variety of task scenarios.

[0024] (2) In response to the problems that existing image translation methods generate insufficient diversity in results or that diversity cannot be expressed at the semantic level, the present invention proposes an image translation model based on deep feature difference and attention mechanism. This model constrains the converted image to maintain as large a difference as possible in high-level semantic features, thereby generating a converted image with semantic diversity. In addition, a residual network structure with an attention mechanism is used in the generator of the model to enhance useful features according to the importance of each feature channel. Compared with existing image translation methods, the images generated by this model have higher diversity and focus on the semantic level.

[0025] (3) To address the problem of insufficient translation accuracy of existing image translation models, the present invention proposes an image translation model that combines a style attention mechanism and self-normalization, which can generate high-quality stylized images while maintaining the content structure. The model introduces a style attention mechanism to match different style patterns according to the content structure of the image. Secondly, the model introduces self-normalization to recalibrate the features after style embedding, balancing the global style and local style patterns. Compared with existing image translation methods, this model has higher translation accuracy, can generate image results that are more consistent with the reference style, and has higher image quality and visual realism.

[0026] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1This is a flow chart of an image translation method integrating a multi-branch neural network provided by the present invention;

[0029] Figure 2 This is a structural diagram of the paired non-neighbor image conversion model provided by the present invention;

[0030] Figure 3 This is a structural diagram of the paired and nearest neighbor image conversion model provided by the present invention;

[0031] Figure 4 This is a structural diagram of the unpaired and unneighbored image conversion model provided by the present invention;

[0032] Figure 5 This is a structural diagram of the unpaired neighbor image conversion model provided by the present invention;

[0033] Figure 6 This is a structural diagram of a style attention network provided by the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0035] Example 1

[0036] The present invention proposes an image translation method integrating a multi-branch neural network, aiming to overcome the dataset limitation of image translation training and improve the diversity and accuracy of image translation results.

[0037] Figure 1 This is a flow chart of an image translation method integrating a multi-branch neural network provided in the first embodiment of the present invention. Figure 1 As shown, this embodiment provides an image translation method that integrates a multi-branch neural network, which is applicable to image translation scenarios such as public security, medical imaging, and artistic creation, and specifically includes the following steps:

[0038] S110: Construct a semi-paired image conversion model, reconstruct the original image, add a label to the original image, and use the reconstructed image as unlabeled data. The original image includes: paired and no neighbor images, paired and neighbor images, unpaired and no neighbor images, and unpaired and neighbor images.

[0039] For ease of presentation, we categorize samples into four categories based on whether they are paired or neighbors. Paired refers to a real image in the dataset that pairs with the input image, while neighboring refers to an image similar to the input image. For example, if input image x belongs to domain X, paired refers to a real image y in domain Y that pairs with input image x, and neighboring refers to an image x1 in domain X that is similar to input image x.

[0040] The X domain and the Y domain are two different scenes, for example, an image is divided into two domains: day and night.

[0041] It's important to note that generative adversarial networks (GANs) are a popular approach for image-to-image translation, achieving excellent performance in scenarios such as style transfer, object deformation, seasonal transitions, and photo enhancement. Image-to-image translation methods can generally be categorized into two types: paired methods, such as pix2pix, and unpaired methods, such as CycleGAN, DualGAN, and DiscoGAN.

[0042] Both paired and unpaired methods offer significant advantages under their respective conditions. Paired methods maintain high conversion performance and image quality when using fully paired image samples. Unpaired methods, on the other hand, have fewer data sample restrictions and can be applied to a wider range of scenarios. However, when paired images are scarce, fully paired methods can suffer from poor performance due to insufficient training samples, while unpaired methods cannot fully utilize the small number of paired samples.

[0043] Although the unpaired method has the advantage of not requiring paired image data, it lags behind the paired method in terms of conversion effect.

[0044] To this end, the technical solution of the present application proposes a generative adversarial network semi-paired image conversion model that combines neighbor information, namely the semi-paired image conversion model. Based on the generative adversarial network, this model uses a large number of unpaired samples to train the model's domain adaptation ability to learn a mapping from a source domain X to a target domain Y; on the other hand, it also makes full use of the one-to-one strong correlation information in the paired image set, and relies on the inter-domain correspondence relationship of the paired images to learn another mapping during the training of paired images. The fusion of these two mappings not only transfers semantic structure and high-frequency details on the basis of unpaired learning, but also helps to train the network when paired images are limited. In addition, there are some highly similar images in the image samples. The similarity relationship between images is used to construct a cycle consistency loss based on similar images, and the sample similarity relationship is still maintained during the image generation process.

[0045] In real-world image translation tasks, image datasets often contain a small number of paired images. The semi-paired image translation model aims to further leverage the correlation information between images, building on the foundation of unpaired image translation. This semi-pairing approach maximizes the utilization of useful information during training. Furthermore, it incorporates neighbor information between samples and applies a cycle consistency loss to these neighbor samples, thereby improving the model's overall image translation performance.

[0046] Specifically, the semi-paired image conversion model consists of two generators and two discriminators. The first generator G xy Corresponding to the image conversion network from style X to style Y, learning the mapping from domain X to domain Y; the first discriminator D y Trying to distinguish these generated Y-domain images from actual Y-domain images; the second generator G yx Corresponding to the image conversion network from domain Y to domain X; the second discriminator D x An attempt is made to distinguish these generated X-domain images from actual X-domain images.

[0047] The semi-paired image translation model can generate images from the true distribution regardless of whether the samples are paired or have neighbors.

[0048] Combine Figure 2 , for paired images without neighbors, take the X domain input image x as an example, the input image x enters the generator G xy , generate the Y domain image y ˋ, the Y domain image y ˋ enters the generator G yx The reconstructed image xˋˋ is obtained, and the Y domain image yˋ and the paired real image y enter the discriminator D together y Among them, the input image x is similar to the reconstructed image xˋˋ, and the Y domain image yˋ is similar to the paired real image y. Conversely, if the Y domain input image y is taken as an example, the input image y enters the generator G yx , generate X domain image xˋ, X domain image xˋ enters generator G xy The reconstructed image yˋˋ is obtained, and the X-domain image xˋ and the paired real image x enter the discriminator D together x Make a judgment.

[0049] Combine Figure 3 , for paired and neighboring images, take the X domain input image x as an example, the input image x enters the generator G xy , generate the Y domain image y ˋ, the Y domain image y ˋ enters the generator G yx The reconstructed image xˋˋ is obtained, and the Y domain image yˋ and the paired real image y enter the discriminator D together yThe input image x and the similar image x1 are similar to the reconstructed image xˋˋ, and the Y-domain image yˋ is similar to the paired real image y.

[0050] Combine Figure 4 , for unpaired and non-neighboring images, take the X domain input image x as an example, the input image x enters the generator G xy , generate the Y domain image y ˋ, the Y domain image y ˋ enters the generator G yx The reconstructed image xˋˋ is obtained, and the Y domain image yˋ and the real image y enter the discriminator D together y The input image x is similar to the reconstructed image xˋˋ.

[0051] Combine Figure 5 , for unpaired neighbor images, take the X domain input image x as an example, the input image x enters the generator G xy , generate the Y domain image y ˋ, the Y domain image y ˋ enters the generator G yx The reconstructed image xˋˋ is obtained, and the Y domain image yˋ and the real image y enter the discriminator D together y The input image x and the similar image x1 are both similar to the reconstructed image xˋˋ.

[0052] After training the semi-paired image conversion model, the y ˋ generated by the generator is similar to the real image y, so that the generated image y ˋ enters the discriminator D y After that, it will be judged as a real image, achieving preliminary image translation. The image dataset contains input images, similar images, paired real images, unpaired real images, generated images, and reconstructed images, which expands the images in the entire dataset.

[0053] The semi-paired image conversion model utilizes cycle consistency constraints to ensure that the generators are consistent. When an input image x passes through the first and second generators sequentially, similar images are generated. For the paired portion, because the relationship between images is one-to-one, relying on the image pairing relationship as a conditional constraint can improve the generator's image conversion performance. The semi-paired image conversion model uses unpaired samples to learn the mapping ability between the source and target domains. It also uses paired image samples to optimize network parameters, ensuring that the model maintains a certain degree of conversion capability while also preserving basic semantic information and detailed features.

[0054] In addition, when there are highly similar images in the image samples, the semi-paired image conversion model requires that this similarity relationship can be maintained throughout the training process.

[0055] By utilizing neighboring samples to further explore the correlation information between images, the model can be trained more efficiently.

[0056] The semi-paired image conversion model combines the advantages of both unpaired and paired methods, resulting in greater flexibility and efficiency. When a small number of paired relationships exist in an image dataset, unlike unpaired methods, these relationships are not discarded. Instead, the model leverages these relationships to further improve network performance. To maximize image conversion performance by leveraging limited inter-image information, the semi-paired image conversion model adds neighbor information constraints to semi-pairing. By exploring similar relationships between image samples and leveraging neighbor information to derive many-to-many weak associations, the semi-paired image conversion model expands the number of paired images in the dataset, increasing the weight of instructive image information.

[0057] By constructing a semi-paired image translation model, we effectively address the inefficiency of existing image translation methods when paired images are scarce. Furthermore, we introduce a cycle consistency loss based on neighbor information, leveraging the intrinsic connections between images to mine more useful information. This approach is applicable to diverse application scenarios, including object deformation, day-night conversion, and image segmentation.

[0058] S120: Construct a multi-branch image translation model, input the original image into the multi-branch image translation model, the first branch of the multi-branch image translation model extracts the first feature from the original image through deep feature difference and attention mechanism; the second branch extracts the second feature from the original image through style attention mechanism and self-normalization.

[0059] For the first branch, a style mapping network is used to convert latent variables into style codes. This code and the original image are then fed into the generator for image transformation to produce multiple generated images. A pre-trained deep feature extraction network is then used to determine the deep feature differences between the generated images. This deep feature difference is then used to ensure that the generated images, generated from the same input image corresponding to different style codes, have distinct semantic features. By combining deep feature differences with the attention mechanism, the semantic diversity of the resulting images is enhanced.

[0060] Image style diversity can be reflected in low-level features such as color. However, this diversity clearly contradicts human perception. Distance-based metrics, which compare image differences pixel by pixel, ignore structural information and only focus on low-level semantic features. An ideal image diversity translation model should consider high-level features, such as different hairstyles for people and different dog breeds, in addition to differences in low-level features.

[0061] In order to improve the visual authenticity of the image and the accuracy of style conversion, the output of the generator at each stage is subjected to authenticity judgment to ensure that the generator's generation results are sufficiently realistic.

[0062] In addition, a channel-based attention mechanism is used during the generator's downsampling phase. Through deep learning, the importance of each feature channel is determined, helping the model focus on the features of the original image that require style transformation. This mechanism also suppresses features that are less useful for the task, such as background information, to prevent them from interfering with the generated results during the style transformation process, further improving the model's image visual quality and detail.

[0063] Authenticity determination is achieved through a discriminator, a multi-task discriminator composed of multiple output branches. The discriminator determines whether image x is a real image from its domain Y or a generated image from the domain Y generated by the generator. To ensure image authenticity, traditional generative adversarial networks use a discriminator to perform authenticity checks on generated images, distinguishing whether the image belongs to the real distribution or the generated distribution. The input image is generated twice by the same generator to obtain a generated image and a reconstructed image. Both the generated image and the reconstructed image must be authentic, so authenticity checks are performed on both images during training, further improving the visual authenticity of the images generated by the generator.

[0064] Deep feature variance aims to further enhance image diversity. Traditional methods maximize the variance between generated images, forcing the generator to explore the image space and discover meaningful stylistic features to generate diverse images. However, these variances are calculated by considering only the pixel-by-pixel distance between generated images, which is the only difference perceived by the machine. They fail to consider both the local and global aspects of the image, resulting in a significant loss of semantic information.

[0065] To this end, the deep feature extraction network uses a pre-trained AlexNet. The training process of the deep feature extraction network includes: first extracting the deep features of each generated image, then calculating the distance between the deep features to measure the difference between the images, and by maximizing the difference between the generated images, forcing the generator to explore the high-level semantic space of the image and generate images with semantic diversity.

[0066] In order to achieve better image translation effects and feature details, we use a channel-based attention mechanism to help the model focus on the features of the part of the input image that needs to undergo style changes.

[0067] In practice, we introduce an attention mechanism into the residual block of the generator's downsampling phase, generating weights for each feature channel based on the correlation between them. The output of the attention mechanism can be viewed as the importance of each feature channel after feature selection. This is then weighted per channel by multiplication onto the previous features, completing the recalibration of the original features along the channel dimension. We then use the downsampling results with attention and the style code as input for upsampling. Ultimately, the generator's generated results focus on the areas we need to perform style transfer, improving the accuracy of image translation and reducing the interference of minor image information on the generated results.

[0068] The dual-discrimination approach of the channel attention mechanism and the generator not only highlights the characteristic details of the generated image but also ensures that the image is as realistic as possible at each stage in the network, improving the visual quality of the image translation model. A method based on deep feature differentiation and the attention mechanism improves the diversity of image translation results. A deep feature network extracts features at each level of the image to control the semantic diversity of the generated results. Based on these features, the feature differentiation between images is calculated, and this feature differentiation is used to constrain the translated images to have large differences in high-level semantic features.

[0069] For the second branch, first, a style mapping network is used to generate a style code from the latent variable. A style attention mechanism is introduced into the style mapping network, and a learnable soft attention network is used to convert the content feature F C Mapped to style feature F S The style feature is used as the second feature to obtain a style code with attention to the source image content, and then the style code and the original image are input into the generator for image conversion.

[0070] The style attention mechanism aims to appropriately embed a local style pattern at each position of the content feature map by learning the correlation between the content feature map and the style feature map.

[0071] Specifically, the content feature F C Mapped to style feature F S The weights include: first, the content feature F C and style characteristics F S Perform normalization and then convert to feature space f and g, calculate the attention between f and g, and use the content feature F C and style characteristics F S The relationship between , embeds a style feature at each position of the content feature.

[0072] Combine Figure 6 , using the style feature embedding of the style attention module SANet, the content feature map F C and style feature map FS First perform normalization, then convert into two feature spaces f and g, and calculate and Attention between As shown below:

[0073]

[0074] Figure 6 middle, h(F S )=W h F S , where W f , W g , W h is the learned weight matrix, which is implemented by 1×1 convolution during training. Normalize. In addition, i is the index of the output position, and j is the index of all possible positions. f , W g , W h is the learned weight matrix, implemented via 1×1 convolution during training. SANet's input consists of a content feature map and a style feature map. The style attention module learns the relationship between the content and style feature maps to appropriately embed a local style pattern at each position in the content feature map.

[0075] In computer vision, instance normalization is essentially a style-based normalization, which is equivalent to unifying different images into a certain style. In a convolutional neural network, assuming that each instance has C feature maps of size H×W, given A∈R H×W , instance normalization first normalizes the feature map, and then performs affine transformation to transform A into:

[0076]

[0077] Among them, μ A and σ A are the mean and standard deviation, respectively, while γ and β are the hyperparameters of the affine transformation to be learned. In deep learning, image style is the statistical information across the spatial distribution of each feature channel in the feature map, such as the mean and variance. During style transfer, the image undergoes significant changes after each instance of normalization, making training difficult and making it difficult to accurately match the reference style to the image content. Self-normalization, however, can recalibrate the statistics, highlighting distinctive features and minimizing subtle ones.

[0078] The self-normalization method uses μ A `=f(μ A ,σ A )μA and σ A `=g(μ A ,σ A )σ A Transform A into:

[0079] μ A `A+σ A `

[0080] Among them, f and g are attention functions, which are implemented through the fully connected layer. Through model training, f and g learn to measure the statistic μ according to their own feature maps. A ` and σ A `, thus completing the style recalibration of the feature map. In order to make the image better match the local style pattern and speed up the training of the model, we use the self-normalization method to calibrate the style of the feature map after the style embedding.

[0081] A self-normalization layer is added after each image style transfer layer of the generator to achieve style recalibration immediately after each style embedding, thereby improving the accuracy of style transfer and translating a stylized image that conforms to the content structure based on the style of the reference image.

[0082] The style mapping network converts latent variables into style codes, which are then used as input to the generator, ultimately generating a generated image that corresponds to the style code. The style codes extracted by this method do not reflect the content structure of the input image, significantly affecting the accuracy of style translation. Furthermore, the generator does not constrain the generated image after style embedding in the image style transfer layer, resulting in an imbalance between local and global styles.

[0083] To this end, we employ a method combining style attention mechanisms and self-normalization to generate high-quality stylized images while preserving content structure. We introduce a style attention mechanism into the model's style encoder, using a learnable soft attention network to map content features to weights for style features. This generates a style code that is attentive to the source image content. This helps the model match different style patterns based on the image's content structure, balancing content and style.

[0084] In the model's generator, we perform self-normalization after each image style transfer layer to recalibrate the style-embedded features. This allows the model to emphasize semantically relevant styles and suppress styles irrelevant to the content, helping to balance global and local style patterns. This approach, combining the style attention mechanism with self-normalization, achieves high image translation accuracy and visual quality, accurately translating the style of a given reference image while preserving the source image's content structure.

[0085] S130: Fusing the features extracted by the first branch and the second branch to generate multimodal features and a target image.

[0086] The first branch extracts the deep feature differences between the generated images and uses this deep difference feature as the first feature. The second branch establishes the connection between content features and style features, and uses the style features formed by different content features as the second feature.

[0087] The original image is input as a training set into the multi-branch image translation model to train the model. The first branch extracts the first feature, and the second branch extracts the second feature. The first feature and the second feature are input into the generator together to generate the target image.

[0088] By training the generator to capture the first and second features, an output image that matches the target distribution is generated.

[0089] S140: Using the labeled data to train the multi-branch image translation model, and using the unlabeled data to evaluate the accuracy of the multi-branch image translation model.

[0090] Training a multi-branch image translation model using labeled data includes: calculating the accuracy of the model;

[0091]

[0092] Among them, Acc represents the accuracy of the model, TP represents the true positive examples in the sample, FP represents the false positive examples in the sample, FN represents the false negative examples in the sample, and TN represents the true negative examples in the sample.

[0093] It also includes: calculating the recall rate of the model;

[0094]

[0095] Among them, Recall represents the recall rate of the model, TP represents the true positive examples in the sample, and FN represents the false negative examples in the sample.

[0096] Also includes: calculating the F1-Score value of the model;

[0097]

[0098] Among them, F1Score represents the F1-Score value of the model, Acc represents the accuracy of the model, and Recall represents the recall rate of the model.

[0099] F1-Score is an indicator used to comprehensively evaluate the performance of classification models. It combines the advantages of accuracy and recall, and can simultaneously reflect the classification effect and recall rate of the model.

[0100] An embodiment of the present invention provides an image translation method that integrates a multi-branch neural network. First, by constructing a semi-paired image translation model, a cycle consistency loss based on neighbor information is introduced, and the intrinsic connection between images is used to mine more available information. It is suitable for different application scenarios such as object deformation, day-night conversion, and image segmentation. Then, a multi-branch image translation model is constructed to extract features from the input image. The method based on deep feature differences and attention mechanism can highlight the feature details of the generated image and improve the visual quality of the image translation model; the method based on style attention mechanism and self-normalization can generate high-quality stylized images while maintaining the content structure. Finally, by training the model with reconstructed images, the accuracy of generating output images that match the target distribution is improved.

[0101] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in the present invention can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution of the present invention can be achieved. This is not limited herein.

[0102] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. An image translation method integrating a multi-branch neural network, characterized in that: include: A semi-paired image conversion model is constructed to reconstruct the original image and add labels to the original image. The reconstructed image is used as unlabeled data. The original image includes: paired images without neighbors, paired images with neighbors, unpaired images without neighbors, and unpaired images with neighbors. A multi-branch image translation model is constructed, and the original image is input into the multi-branch image translation model. The first branch of the multi-branch image translation model extracts the first feature from the original image through the deep feature difference and attention mechanism; the second branch extracts the second feature from the original image through the style attention mechanism and self-normalization; The extracted first feature and the second feature are fused to generate a multimodal feature, and a target image is generated; The multi-branch image translation model is trained using labeled data, and the accuracy of the multi-branch image translation model is evaluated using unlabeled data.

2. The method according to claim 1, characterized in that: The semi-paired image conversion model consists of two generators and two discriminators. The first generator G xy Corresponding to the image conversion network from style X to style Y, learning the mapping from domain X to domain Y; the first discriminator D y Used to distinguish these generated Y domain images from the actual Y domain images; the second generator G yx Corresponding to the image conversion network from domain Y to domain X; the second discriminator D x Used to distinguish these generated X-domain images from actual X-domain images.

3. The method according to claim 1, characterized in that For the first branch, first, use the style mapping network to generate style codes from latent variables, and then input the style codes and the original images into the generator for image conversion to obtain multiple generated images; secondly, use the pre-trained deep feature extraction network to obtain the deep feature differences between the generated images; Then, the results of generating different style codes corresponding to the same input image are controlled to have differences in semantic features according to the difference in deep features.

4. The method according to claim 3, characterized in that The deep feature extraction network uses a pre-trained AlexNet; the training process of the deep feature extraction network includes: first, extracting the deep features of each generated image as the first feature, and then calculating the distance between the first features to measure the difference between the images, and by maximizing the difference between the generated images, forcing the generator to explore the high-level semantic space of the image and generate images with semantic diversity.

5. The method according to claim 3, characterized in that: In the downsampling stage of the generator, a channel-domain-based attention mechanism is adopted to obtain the importance of each feature channel through deep learning, and then weighted to the first feature channel by channel through multiplication to complete the recalibration of the first feature in the channel dimension.

6. The method according to claim 5, characterized in that For the second branch, first, a style mapping network is used to generate a style code from hidden variables. A style attention mechanism is introduced into the style mapping network, and a learnable soft attention network is used to convert the content feature F C Mapped to style feature F S The style feature is used as the second feature to obtain a style code with attention to the source image content, and then the style code and the original image are input into the generator for image conversion.

7. The method according to claim 6, characterized in that The content feature F C Mapped to style feature F S The weights include: first, the content feature F C and style features F S Perform normalization and then convert to feature space f and g, calculate the attention between f and g, and use the content feature F C and style features F S The relationship between , embeds a style feature at each position of the content feature.

8. The method according to claim 7, characterized in that The original image is input as a training set into the multi-branch image translation model to train the model. The first branch extracts the first feature, the second branch extracts the second feature, and the first feature and the second feature are input into the generator together to generate the target image.

9. The method according to claim 8, characterized in that The training of the multi-branch image translation model using the label data includes: calculating the accuracy of the model; Among them, Acc represents the accuracy of the model, TP represents the true positive examples in the sample, FP represents the false positive examples in the sample, FN represents the false negative examples in the sample, and TN represents the true negative examples in the sample.

Images