A training method, device and electronic device for image style transfer model

The non-paired sample images are processed through forward and reverse networks, and combined with loss value training of paired sample images, the problem of high dependence on paired image data and slow convergence speed in the prior art is solved. , improves the accuracy and training efficiency of image style transfer models.

CN114154613BActive Publication Date: 2024-11-26BEIJING DAJIA INTERNET INFORMATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202010928062.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-09-07
Publication Date
2024-11-26
Estimated Expiration
2040-09-07

AI Technical Summary

Technical Problem

In the training of image style transfer models, supervised learning relies on a large amount of paired image data, resulting in high data dependence, while weakly supervised learning models converge slowly and have low accuracy, resulting in blurred image details and other problems.

Method used

The non-paired sample images are processed using forward and reverse networks, combined with paired sample images, and the network is trained through loss values ​​to reduce dependence on paired image data, and improve convergence speed and accuracy.

Benefits of technology

The training process of image style transfer model is simplified, the dependence on paired image data is reduced, and the convergence speed of the model and the accuracy of style transfer are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114154613B_ABST
    Figure CN114154613B_ABST
Patent Text Reader

Abstract

The present disclosure provides a training method, device and electronic device for an image style transfer model, and relates to the field of image processing technology. The method comprises: using a forward network and a reverse network to process unpaired sample images to determine a first loss value; the unpaired sample images include a first image with a first style and / or a second image with a second style; using at least one of the forward network and the reverse network to process paired sample images to determine a second loss value; the paired sample images include an image group consisting of a third image with a first style and a fourth image with a second style obtained by processing the third image; based on the first loss value and the second loss value, the forward network and the reverse network are trained to obtain the trained forward network and reverse network. The method reduces the dependence of the training of the image style transfer model on paired data, improves the training speed, and makes the results of the model training more controllable.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a training method, device and electronic device for an image style transfer model. Background Art

[0002] When training a model for image style transfer, related technologies often train the model through supervised learning and weakly supervised learning. However, the training method based on supervised learning requires a large amount of paired image data, that is, an image group consisting of an original image with an original style and an image with another style obtained by style conversion of the original image. The training of the model has a strong dependence on paired image data. The training method based on weakly supervised learning, due to the lack of paired image data restrictions, the model converges slowly during the training process, and the trained model has low accuracy in image style transfer, and is prone to problems such as blurred image details. Therefore, how to simplify the training process of the image style transfer model to improve the accuracy of image processing by the image style transfer model has become a problem that needs to be considered. Summary of the invention

[0003] The embodiments of the present disclosure provide a method, device and electronic device for training an image style transfer model, which are used to simplify the training process of the image style transfer model to improve the accuracy of image processing by the image style transfer model.

[0004] In a first aspect, the present disclosure provides a method for training an image style transfer model, comprising:

[0005] Using a forward network and a reverse network to process an unpaired sample image, and determining a first loss value of the image processed by the forward network and the reverse network; the unpaired sample image includes a first image with a first style and / or a second image with a second style; the forward network is used to process the image with the first style to obtain an image with the second style; the reverse network is used to process the image with the second style to obtain an image with the first style;

[0006] Using at least one of the forward network and the reverse network, a pair of sample images is processed to determine a second loss value of the at least one network processing the image; the pair of sample images includes an image group consisting of a third image having a first style and a fourth image having a second style obtained by processing the third image;

[0007] Based on the first loss value and the second loss value, the forward network and the reverse network are trained to obtain the trained forward network and reverse network.

[0008] In a possible implementation, the step of processing the unpaired sample image using the forward network and the reverse network and determining a first loss value of the image processed by the forward network and the reverse network includes:

[0009] The unpaired sample image is input into a forward network and a reverse network connected in series for processing to obtain a first reference image, and the first loss value is determined based on a pixel difference between the unpaired sample image and the first reference image.

[0010] In a possible implementation, the step of inputting the unpaired sample image into a forward network and a reverse network connected in series for processing to obtain a first reference image includes:

[0011] When the unpaired sample image is the first image, the forward network is used to perform forward style processing on the first image to obtain a first reference image with the second style, and the reverse network is used to perform reverse style processing on the first reference image to obtain a first reference image with the first style;

[0012] When the unpaired sample image is the second image, the reverse network is used to perform reverse style processing on the second image to obtain a second reference image with the first style, and the forward network is used to perform forward style processing on the second reference image to obtain a first reference image with the second style.

[0013] In a possible implementation, when the unpaired sample includes the first image and the second image, the step of determining a first loss value for image processing by the forward network and the reverse network based on a pixel difference between the unpaired sample image and the first reference image includes:

[0014] determining a first transformation loss value of the first image based on a pixel difference between the first reference image having the first style and the first image; and

[0015] determining a second transformation loss value of the second image based on a pixel difference between the first reference image having the second style and the second image;

[0016] The first loss value is determined according to the first transformation loss value and the second transformation loss value.

[0017] In a possible implementation, the step of determining a second loss value of the image processing by at least one of the forward network and the reverse network based on a result of processing the paired sample images includes:

[0018] At least one of the forward network and the reverse network is used to process the target sample image to obtain a second reference image, and the second loss value is determined based on the pixel difference between the second reference image and a reference sample image corresponding to the target sample image; wherein when the target sample image is the third image, the reference sample image is the fourth image, and when the target sample image is the fourth image, the reference sample image is the third image.

[0019] In a possible implementation, the step of using at least one of the forward network and the reverse network to process the target sample image to obtain the second reference image includes:

[0020] When the target sample image is the third image and the reference sample image is the fourth image, the forward network is used to perform forward style processing on the third image to obtain a second reference image having the second style;

[0021] When the target sample image is the fourth image and the reference sample image is the third image, the inverse network is used to perform inverse style processing on the fourth image to obtain a second reference image with the first style.

[0022] In a possible implementation, the target sample image includes the third image and the fourth image, and the step of determining the second loss value based on a pixel difference between the second reference image and a reference sample image corresponding to the target sample image includes:

[0023] determining a third transformation loss value of the third image based on a pixel difference between the second reference image having the second style and a fourth image corresponding to the third image; and

[0024] determining a fourth transformation loss value of the fourth image based on a pixel difference between the second reference image having the first style and a third image corresponding to the fourth image;

[0025] The second loss value is determined according to a third transformation loss value of the third image and a fourth transformation loss value of the fourth image.

[0026] In a possible implementation, the step of training the forward network and the reverse network based on the first loss value and the second loss value includes:

[0027] Determine a result of weighted summation of the first loss value and the second loss value as a comprehensive loss value;

[0028] Based on the comprehensive loss value, the forward network and the reverse network are trained.

[0029] In a second aspect of the present disclosure, a training device for an image style transfer model is provided, comprising:

[0030] A first image processing unit is configured to process an unpaired sample image using a forward network and a reverse network, and determine a first loss value of the image processed by the forward network and the reverse network; the unpaired sample image includes a first image with a first style and / or a second image with a second style; the forward network is used to process the image with the first style to obtain an image with the second style; the reverse network is used to process the image with the second style to obtain an image with the first style;

[0031] A second image processing unit is configured to execute processing of a pair of sample images using at least one of the forward network and the reverse network, and determine a second loss value of the at least one network processing the image; the pair of sample images includes an image group consisting of a third image having a first style and a fourth image having a second style obtained by processing the third image;

[0032] A training unit is configured to perform training on the forward network and the reverse network based on the first loss value and the second loss value to obtain the trained forward network and reverse network.

[0033] In a possible implementation manner, the first image processing unit is specifically configured to execute:

[0034] The unpaired sample image is input into a forward network and a reverse network connected in series for processing to obtain a first reference image, and the first loss value is determined based on a pixel difference between the unpaired sample image and the first reference image.

[0035] In a possible implementation manner, the first image processing unit is specifically configured to execute:

[0036] When the unpaired sample image is the first image, the forward network is used to perform forward style processing on the first image to obtain a first reference image with the second style, and the reverse network is used to perform reverse style processing on the first reference image to obtain a first reference image with the first style;

[0037] When the unpaired sample image is the second image, the reverse network is used to perform reverse style processing on the second image to obtain a second reference image with the first style, and the forward network is used to perform forward style processing on the second reference image to obtain a first reference image with the second style.

[0038] In a possible implementation manner, when the unpaired sample includes the first image and the second image, the first image processing unit is specifically configured to execute:

[0039] determining a first transformation loss value of the first image based on a pixel difference between the first reference image having the first style and the first image; and

[0040] determining a second transformation loss value of the second image based on a pixel difference between the first reference image having the second style and the second image;

[0041] The first loss value is determined according to the first transformation loss value and the second transformation loss value.

[0042] In a possible implementation manner, the second image processing unit is specifically configured to execute:

[0043] At least one of the forward network and the reverse network is used to process the target sample image to obtain a second reference image, and the second loss value is determined based on the pixel difference between the second reference image and a reference sample image corresponding to the target sample image; wherein when the target sample image is the third image, the reference sample image is the fourth image, and when the target sample image is the fourth image, the reference sample image is the third image.

[0044] In a possible implementation manner, the second image processing unit is specifically configured to execute:

[0045] When the target sample image is the third image and the reference sample image is the fourth image, the forward network is used to perform forward style processing on the third image to obtain a second reference image having the second style;

[0046] When the target sample image is the fourth image and the reference sample image is the third image, the inverse network is used to perform inverse style processing on the fourth image to obtain a second reference image with the first style.

[0047] In a possible implementation manner, the target sample image includes the third image and the fourth image, and the second image processing unit is specifically configured to execute:

[0048] determining a third transformation loss value of the third image based on a pixel difference between the second reference image having the second style and a fourth image corresponding to the third image; and

[0049] determining a fourth transformation loss value of the fourth image based on a pixel difference between the second reference image having the first style and a third image corresponding to the fourth image;

[0050] The second loss value is determined according to a third transformation loss value of the third image and a fourth transformation loss value of the fourth image.

[0051] In a possible implementation, the training unit is specifically configured to execute:

[0052] Determine a result of weighted summation of the first loss value and the second loss value as a comprehensive loss value;

[0053] Based on the comprehensive loss value, the forward network and the reverse network are trained.

[0054] According to a third aspect of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect and any one of the possible implementation modes is implemented.

[0055] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions. When the computer instructions are executed on a computer, the computer executes the method described in the first aspect and any one of the possible implementation modes.

[0056] The solution disclosed herein brings at least the following beneficial effects:

[0057] In the method provided in the embodiment of the present application, unpaired sample images and paired sample images are used to train the forward network, and a reverse network is used to assist the training of the forward network; compared with the training method based on supervised learning, the method provided in the embodiment of the present application uses unpaired sample images for training during the training process, which reduces the number of paired image data used and reduces the dependence on paired image data; compared with the training method based on weakly supervised learning, the method provided in the embodiment of the present application adds paired sample images during the training process, which improves the convergence speed of the forward network, and uses a reverse network to assist the training of the forward network during the training process, which improves the accuracy of the trained forward network in style transfer for images with the first style. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0059] Figure 1A flowchart of a training process of an image style transfer model provided for an exemplary embodiment of the present disclosure;

[0060] Figure 2 An exemplary diagram of a process of obtaining a first loss value provided for an exemplary embodiment of the present disclosure;

[0061] Figure 3 An exemplary diagram of a process of obtaining a first loss value provided for an exemplary embodiment of the present disclosure;

[0062] Figure 4 An exemplary diagram of a process of obtaining a second loss value provided for an exemplary embodiment of the present disclosure;

[0063] Figure 5 An exemplary diagram of a process of obtaining a second loss value provided for an exemplary embodiment of the present disclosure;

[0064] Figure 6 A flowchart of a round of forward network and reverse network training process provided for an exemplary embodiment of the present disclosure;

[0065] Figure 7 A flowchart of a round of forward network and reverse network training process provided for an exemplary embodiment of the present disclosure;

[0066] Figure 8 A flowchart of a training process of a forward network and a reverse network provided for an exemplary embodiment of the present disclosure;

[0067] Fig. 9 A flowchart of a round of forward network and reverse network training process provided for an exemplary embodiment of the present disclosure;

[0068] Fig.10 A flowchart of a training process of a forward network and a reverse network provided for an exemplary embodiment of the present disclosure;

[0069] Fig.11 A complete flow chart of an image style transfer model provided for an exemplary embodiment of the present disclosure;

[0070] Fig.12 A schematic diagram of the structure of a training device for an image style transfer model provided by an exemplary embodiment of the present disclosure;

[0071] Fig.13 A schematic structural diagram of an electronic device provided by an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0072] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0073] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein.

[0074] In order to help those skilled in the art better understand the technical solutions of the present disclosure, the technical terms involved in the present disclosure are explained below.

[0075] Style transfer: refers to the process of transforming digital images through computers (or other intelligent computing devices) to output images with style, such as common image styles such as oil paintings, fine brushwork, and hand-painted portraits. Image style processing is an important application in the field of computer vision and can be applied to various entertainment content production scenarios. For example, in smartphone short video applications, users can quickly edit stylized portrait photos, bringing a new experience to users.

[0076] Generative Adversarial Network (GAN): A deep learning model in unsupervised learning. The model includes at least two modules in the framework: a generative model and a discriminative model. The generative model and the discriminative model learn from each other to produce better quality output.

[0077] The design concept of the present disclosure is described below.

[0078] When training a model used for image style transfer, such as a generator, the training is usually performed in the following two ways: 1. A training method based on supervised learning, in which the generator model is trained through paired image data (i.e., an original image with an original style and an image with another style obtained by style conversion of the original image), the original image is processed by the generator to obtain an image of another style, and the model is adjusted by comparing the style image samples corresponding to the image of another style and the original image, where the style image samples are samples of images of another style converted from the original image; 2. A training method based on weakly supervised learning, in which the generator model is trained through paired image data (i.e., an original image with an original style and an image with another style obtained by style conversion of the original image). In this method, two generators are trained by GAN technology, one of which is used to transfer the style of the original image, and the other is used to perform reverse style transfer on the image after style transfer. The two generators are adjusted according to the comparison result between the image obtained by reverse style transfer and the original image. However, in the above training process, the training method based on supervised learning requires a large amount of paired image data and is dependent on paired image data. The training method based on weak supervised learning has no limitation on paired image data, so the generator converges slowly during the training process, and the trained generator has low accuracy in style transfer to the image, which is prone to problems such as blurred image details.

[0079] In view of this, the present disclosure designs a training method, device and electronic device for an image style transfer model to simplify the training process of the image style transfer model to improve the accuracy of the image style transfer model in processing images. In the embodiments of the present disclosure, it is considered to train the forward network and the reverse network used for image style transfer with paired sample images and unpaired sample images at the same time, so as to improve the convergence speed of the forward network and the reverse network while improving the accuracy of the style transfer of the image using the trained forward network. Specifically, the method of the present disclosure includes: using the forward network and the reverse network to process the unpaired sample images, and determining the first loss value of the forward network and the reverse network for processing the image; and using at least one of the forward network and the reverse network to process the paired sample images, and determining the second loss value of the at least one network for processing the image, and then based on the first loss value and the second loss value, training the forward network and the reverse network to obtain the trained forward network and reverse network; wherein:

[0080] The unpaired sample images include a first image with a first style and / or a second image with a second style; the paired sample images include an image group consisting of a third image with a first style and a fourth image with a second style obtained by processing the third image;

[0081] The forward network in the disclosed embodiment is used to process the image with the first style to obtain the image with the second style; the reverse network is used to process the image with the second style to obtain the image with the first style.

[0082] The solution of the present disclosure is described in detail below with reference to the accompanying drawings.

[0083] See also Figure 1 The present disclosure provides a network training process for image processing, which specifically includes the following steps:

[0084] Step S101, using a forward network and a reverse network to process an unpaired sample image, and determining a first loss value of the image processed by the forward network and the reverse network; the unpaired sample image includes a first image with a first style and / or a second image with a second style.

[0085] Considering that in order to improve the accuracy of the style transfer of the image by the trained forward network, the unpaired sample image in the embodiment of the present disclosure may include a first image with a first style, may include a second image with a second style, or may include the first image and the second image at the same time;

[0086] When the unpaired sample image includes the above-mentioned first image and the second image at the same time, the above-mentioned first image and the second image may correspond one-to-one, or the above-mentioned first image and the second image may have no corresponding relationship, or some of the above-mentioned first image and the second image may have a corresponding relationship; if style migration is performed on a first image with a first style to obtain a second image with a second style, then the first image and the obtained second image can be regarded as corresponding.

[0087] The above-mentioned first style and second style are not limited and refer to two different styles of images. Technical personnel in this field can set the first style and the second style according to actual needs; the style of the image can be but is not limited to a style with a set filter, and the set filter can be but is not limited to including a pure filter, a dream filter, a milk tea filter, a romantic filter, a retro filter, a film filter, a soft filter, a cool filter, etc.; the style of the image can also include a style with a set artistic effect, and the set artistic effect can be but is not limited to including: sketch, comics, oil painting, quick sketch, watercolor, ink, etc.

[0088] In step S101, the unpaired sample image may be input into a forward network and a reverse network connected in series for processing to obtain a first reference image, and the first loss value may be determined based on a pixel difference between the unpaired sample image and the first reference image.

[0089] Step S102, using at least one of the forward network and the reverse network to process paired sample images, and determining a second loss value of the at least one network processing the images; the paired sample images include an image group consisting of a third image with a first style and a fourth image with a second style obtained by processing the third image.

[0090] Specifically, it is possible but not limited to using at least one of the forward network and the reverse network to process the target sample image to obtain a second reference image, and determine the second loss value based on the pixel difference between the second reference image and the reference sample image corresponding to the target sample image; wherein when the target sample image is the third image, the reference sample image is the fourth image, and when the target sample image is the fourth image, the reference sample image is the third image.

[0091] Step S103: Based on the first loss value and the second loss value, the forward network and the reverse network are trained to obtain the trained forward network and reverse network.

[0092] As an embodiment, in step S103, when the forward network and the reverse network are trained, the model parameters of the forward network and the model parameters of the reverse network can be adjusted based on the first loss value and the second loss value; for example, the model parameters of the forward network and the model parameters of the reverse network model can be gradiently increased or gradiently decreased according to the target parameter adjustment direction; and then when it is determined that the forward network and the reverse network meet the preset convergence conditions, the trained forward network and the reverse network are obtained; the above-mentioned target parameter adjustment direction includes making the comprehensive loss value after adjusting the model parameters less than the comprehensive loss value before adjusting the model parameters.

[0093] The above-mentioned preset convergence condition may be, but is not limited to, using a trained forward network to perform style transfer on a test sample image having a first style, and obtaining a test image having a second style and a reference sample image corresponding to the test sample image, wherein the pixel difference is less than a pixel difference threshold. The above-mentioned test sample image may be, but is not limited to, an image having the second style generated by performing style transfer on the above-mentioned test sample image in other ways (such as a style transfer algorithm).

[0094] Those skilled in the art can also flexibly train the forward network and the reverse network based on the comprehensive loss value according to other practical needs.

[0095] It should be noted that the above steps S101 to S103 are only descriptions of the main steps of the embodiment of the present disclosure, and do not mean that the execution order of the above steps S101 to S103 is fixed. Those skilled in the art can flexibly adjust the execution process of the above steps S101 to S103 according to actual needs.

[0096] The following content of the embodiment of the present disclosure respectively describes the process of processing the unpaired sample images and the paired sample images in the above steps S101 and S102.

[0097] (i) The process of processing the unpaired sample images in step S101.

[0098] When the unpaired sample image is the first image A1 with the first style mentioned above, the forward network G_A is used to perform forward style processing on the first image A1 to obtain the first reference image G_A(A1) with the second style mentioned above, and the reverse network G_B is used to perform reverse style processing on the first reference image G_A(A1) to obtain the first reference image G_B(G_A(A1)) with the first style mentioned above.

[0099] When the unpaired sample image is the second image B1 with the second style, the reverse network G_B is used to perform reverse style processing on the second image B1 to obtain the second reference image G_B(B1) with the first style, and the forward network G_A is used to perform forward style processing on the second reference image G_B(B1) to obtain the first reference image G_A(G_B(B1)) with the second style.

[0100] Further, based on the pixel difference between the unpaired sample image and the first reference image, determining the first loss value of the image processed by the forward network and the reverse network includes the following three cases:

[0101] The first case: the unpaired sample image is a first image A1 with a first style.

[0102] Determine a first loss value based on a pixel difference between a first reference image G_B(G_A(A1)) having a first style and a first image A1 having a first style;

[0103] Specifically, the absolute value of the pixel difference between the first reference image G_B (G_A (A1)) with the first style and the first image A1 with the first style can be determined as the first loss value L1 based on but not limited to the following formula 1, or the square value of the pixel difference between the first reference image G_B (G_A (A1)) with the first style and the first image A1 with the first style can be determined as the first loss value L1 based on the following formula 2.

[0104] Formula 1: L1 = |G_B(G_A(A1))-A1|;

[0105] Formula 2: L1 = {G_B(G_A(A1))-A1} 2 ;

[0106] In the above formulas 1 and 2, L1 is the first loss value, and G_B(G_A(A1)) is the first reference image with the first style.

[0107] The second case: the first sample image is a second image B1 with a second style.

[0108] Determine a first loss value based on a pixel difference between a first reference image G_A(G_B(B1)) having a second style and a second image B1 having a second style;

[0109] Specifically, the absolute value of the pixel difference between the first reference image G_A(G_B(B1)) with the second style and the second image B1 with the second style can be determined as the first loss value L1 based on but not limited to the following formula 3, or the square value of the pixel difference between the first reference image G_A(G_B(B1)) with the second style and the second image B1 with the second style can be determined as the first loss value L1 based on the following formula 4.

[0110] Formula 3: L1 = |G_A(G_B(B1))-B1|;

[0111] Formula 4: L1 = {G_A(G_B(B1))-B1} 2 ;

[0112] In the above formulas 1 and 2, L1 is the first loss value, and G_A(G_B(B1)) is the first reference image with the second style.

[0113] The third case: the first sample image includes a first image A1 having a first style and a second image B1 having a second style.

[0114] Determine a first transformation loss value of the first image based on a pixel difference between a first reference image G_B(G_A(A1)) having a first style and a first image A1 having a first style; and

[0115] Based on the pixel difference between the second reference image G_A(G_B( B1 )) having the second style and the second image B1 having the second style, a second transformation loss value of the second image is determined.

[0116] Furthermore, it is possible but not limited to performing weighted summation processing on the first transformation loss value and the second transformation loss value based on the following formula 5 to obtain the first loss value L1, or based on the following formula 6, the first loss value L1 is determined by summing the square value of the first transformation loss value and the square value of the second transformation loss value.

[0117] Formula 5: L1=|G_B(G_A(A1))-A1|×q1+|G_A(G_B(B1))-B1|×q2;

[0118] Formula 6: L1 = {G_B (G_A (A1)) - A1} 2 +{G_A(G_B(B1))-B1} 2 ;

[0119] Among them, q1 and q2 in Formula 5 are weight values ​​of weighted processing of the first transformation loss value and the second transformation loss value, respectively, and q1 and q2 are non-negative numbers. q1 and q2 can be determined according to the importance or quantity of the first image and the second image in the first sample image. Technical personnel in this field can also determine the values ​​of the above q1 and q2 according to other reference factors according to actual needs; in Formula 5 and Formula 6, (G_B(G_A(A1))-A1) is the above first transformation loss value, and (G_A(G_B(B1))-B1) is the above second transformation loss value.

[0120] See also Figure 2 , an example diagram of a process of obtaining a first reference image G_B(G_A(A1)) having a first style from a first image A1 having a first style, and obtaining a first loss value L1 from the pixel difference between the first reference image G_B(G_A(A1)) having a first style and the first image A1 is given; see Figure 3 , an example diagram is given of a process of obtaining a first reference image G_A (G_B (B1)) having a second style from a second image B1 having a second style, and obtaining a first transformation loss value L1 from the pixel difference between the first reference image G_A (G_B (B1)) having a second style and the second image B1.

[0121] (ii) The process of processing the paired sample images in step S102.

[0122] The target sample image is a third image A2 with a first style, and the reference sample image is a fourth image B2 with a second style. The forward network G_A is used to perform forward style processing on the third image A2 to obtain a second reference image G_A(A2) with a second style; wherein the fourth image B2 is obtained by performing style transfer on the third image A2, and the third image A2 and the fourth image B2 are a paired sample image (i.e., a data group).

[0123] The target sample image is a fourth image B3 with a second style, and the reference sample image is a third image A3 with a first style. The fourth image B3 is subjected to inverse style processing using an inverse network G_B to obtain a second reference image G_B(B3) with a first style; wherein the third image A3 is obtained by performing inverse style transfer on the fourth image B3, and the third image A3 and the fourth image B3 are a pair of sample images (i.e., a data group).

[0124] Further, based on the pixel difference between the second reference image and the reference sample image corresponding to the target sample image, determining the second loss value includes the following three situations:

[0125] The fourth case: the target sample image package is the third image A2.

[0126] Based on the pixel difference between the second reference image G_A(A2) with the second style and the fourth image B2 with the second style, a third transformation loss value of the third image A2 is determined; wherein the third image A2 and the fourth image B2 are an image group (ie, a paired sample image).

[0127] Specifically, the absolute value of the pixel difference between the second reference image G_A(A2) with the second style and the fourth image B2 with the second style can be determined as the second loss value L2 based on but not limited to the following formula 7, or the square value of the pixel difference between the second reference image G_A(A2) with the second style and the fourth image B2 with the second style can be determined as the second loss value L2 based on the following formula 8, and the above-mentioned third image A2 and fourth image B2 are an image group.

[0128] Formula 7: L2 = |G_A(A2)-B2|;

[0129] Formula 8: L2 = {G_A(A2)-B2} 2 ;

[0130] In the above formulas 7 and 8, L2 is the second loss value.

[0131] The fifth case: the target sample image is the fourth image B3.

[0132] Based on the style deviation of the second reference image G_B(B3) having the first style and the third image A3 having the first style, a fourth transformation loss value of the fourth image B3 is determined; wherein the third image A3 and the fourth image B3 are an image group (ie, a paired sample image).

[0133] Specifically, the absolute value of the pixel difference between the second reference image G_B(B3) with the first style and the third image A3 with the first style can be determined as the second loss value L2 based on but not limited to the following formula 9, or the square value of the pixel difference between the second reference image G_B(B3) with the first style and the third image A3 with the first style can be determined as the second loss value L2 based on the following formula 10.

[0134] Formula 9: L2 = |G_B(B3)-A3|;

[0135] Formula 10: L2 = {G_B(B3)-A3} 2 ;

[0136] In the above formulas 9 and 10, L2 is the second loss value.

[0137] The sixth case: the target sample images include the third image A2 and the fourth image B3.

[0138] determining a third transformation loss value of the third image A2 based on a pixel difference between the second reference image G_A(A2) having the second style and the fourth image B2 having the second style; and

[0139] Determine a fourth transformation loss value of a fourth image B3 based on a pixel difference between a second reference image G_B(B3) having the first style and a third image A3 having the first style;

[0140] Furthermore, it is possible but not limited to performing weighted summation processing on the third transformation loss value and the fourth transformation loss value based on the following formula 11 to obtain the second loss value L2, or based on the following formula 12, the sum of the square value of the third transformation loss value and the square value of the fourth transformation loss value is determined as the read second loss value L2.

[0141] Formula 11: L2 = |G_A(A2)-B2|×q3+|G_B(B3)-A3|×q4;

[0142] Formula 12: L2 = {G_A(A2)-B2} 2 +{G_B(B3)-A3} 2 ;

[0143] Among them, q3 and q4 in formula 11 are the weight values ​​of the weighted processing of the third transformation loss value and the fourth transformation loss value, respectively, and q3 and q4 are non-negative numbers. q3 and q4 can be determined according to the importance or quantity of the third image and the fourth image in the paired sample images. Technical personnel in this field can also determine the values ​​of the above q3 and q4 according to other reference factors according to actual needs; in formulas 11 and 12, (G_A(A2)-B2) is the third transformation loss value, and (G_B(B3)-A3) is the above fourth transformation loss value.

[0144] It should be noted that the third image A2 and the fourth image B2 are an image group (i.e., a paired sample image), and the third image A3 and the fourth image B3 are an image group (i.e., a paired sample image); the third image A2 and the third image A3 may be the same image or different images, and correspondingly, the fourth image B2 and the fourth image B3 may also be the same image or different images. When the third image A2 and the third image A3 are the same image, and the fourth image B2 and the fourth image B3 are the same image, the above formulas 11 and 12 may be transformed into the following formulas 11a and 12a.

[0145] Formula 11a: L2 = |G_A(A2)-B2|×q3+|G_B(B2)-A2|×q4;

[0146] Formula 12a: L2 = {G_A(A2)-B2} 2 +{G_B(B2)-A2} 2 .

[0147] See also Figure 4 , an example diagram of a process of obtaining a second reference image G_A(A2) having a second style from a third image A2 having a first style, and obtaining a second loss value L2 from the pixel difference between the second reference image G_A(A2) having the second style and the fourth image B2 having the second style is given; see Figure 5 , an example diagram is given of a process of obtaining a second reference image G_B(B3) having a first style from a fourth image B3 having a second style, and obtaining a second loss value L2 from the pixel difference between the second reference image G_B(B3) having the first style and the third image A3 having the second style.

[0148] As an embodiment, in step S103, a comprehensive loss value may be determined based on the first loss value and the second loss value, and the forward network and the reverse network may be trained based on the comprehensive loss value; and the result of the weighted sum of the first loss value and the second loss value may be determined as the comprehensive loss value, but is not limited to; the following content further describes the method for obtaining the comprehensive loss value:

[0149] The comprehensive loss value can be obtained by, but is not limited to, the following methods:

[0150] Determine a first comprehensive transformation loss value L1 according to the first loss value of each unpaired sample image, such as determining the first comprehensive transformation loss value L1 as the mean or variance value of the first loss values ​​of all unpaired sample images;

[0151] Determine a second comprehensive transformation loss value L2 according to the second loss value of each target sample image, such as determining the second comprehensive transformation loss value L2 as the mean or variance value of the second loss values ​​of all target sample images;

[0152] Further, based on the principle of the following formula 13, the result of the weighted sum of the first comprehensive transformation loss value L1 and the second comprehensive transformation loss value L2 can be determined as the above-mentioned comprehensive loss value L; or; or based on the principle of the following formula 14, the sum of the square value L1 of the first comprehensive transformation loss value and the square value of the second comprehensive transformation loss value L2 can be determined as the comprehensive loss value L.

[0153] Formula 13: L = q5 × L1 + q6 × L2;

[0154] Formula 14: L = L1 2 +L2 2 ;

[0155] In the above formulas 13 and 14, L is the comprehensive loss value, L1 is the first comprehensive transformation loss value, L2 is the second comprehensive transformation loss value, q5 and q6 are the weight values ​​of the weighted processing of the first comprehensive transformation loss value and the second comprehensive transformation loss value, respectively, and q5 and q6 are non-negative numbers. q5 and q6 can be determined according to the importance or quantity of unpaired image samples and paired image samples. Technical personnel in this field can also determine the values ​​of the above q5 and q6 according to actual needs and other reference factors, such as but not limited to setting q5 and q6 to 1.

[0156] In the embodiment of the present disclosure, the forward network and the reverse network can be trained based on the first loss value and the second loss value in a variety of ways. The following provides several training processes of step S101 and step S103 combined with the embodiment.

[0157] The first training process: In each round of training, the model parameters of the forward network and the model parameters of the reverse network are adjusted based on the comprehensive loss value of the first loss value of the unpaired sample image and the second loss value of the paired sample image.

[0158] See also Figure 6 , each round of training for the forward network includes the following steps:

[0159] Step S601, using the current forward network and reverse network, determine the first loss value of each unpaired sample image, and determine the first comprehensive transformation loss value based on the first loss value of each unpaired sample image.

[0160] Step S602, using at least one of the current forward network and the reverse network, determine the second loss value of each target sample image in the paired sample images, and determine the second comprehensive transformation loss value based on the second loss value of each target sample image.

[0161] Step S603: Determine the comprehensive loss value of the sample image according to the first comprehensive transformation loss value and the second comprehensive transformation loss value.

[0162] Step S604: Based on the comprehensive loss value of the sample image, the model parameters of the forward network and the model parameters of the reverse network are adjusted.

[0163] That is, in this step, the model parameters of the forward network and the model parameters of the reverse network can be adjusted according to the comprehensive loss L obtained by formula 13 or formula 14.

[0164] The second training process: in each round of training, the model parameters of the forward network and the model parameters of the reverse network are adjusted based on the first loss value of each unpaired sample image; then, the second loss value of each target sample image in the paired sample images is determined through at least one of the adjusted forward network and reverse network, and then the model parameters of the above at least one network are adjusted based on the second loss value of each target sample image.

[0165] See also Figure 7 , providing a training process for the forward network in each round, including the following steps:

[0166] Step S701, obtaining a first loss value of each unpaired sample image through the current forward network and the reverse network, and determining a first comprehensive transformation loss value based on the first loss value of each unpaired sample image.

[0167] Step S702: Based on the first comprehensive transformation loss value, the model parameters of the forward network and the model parameters of the reverse network are adjusted.

[0168] Step S703, obtaining a second loss value of each target sample image in the paired sample images through at least one of the adjusted forward network and the reverse network, and determining a second comprehensive transformation loss value based on the second loss value of each target sample image.

[0169] Step S704: adjusting the model parameters of the at least one network based on the second comprehensive transformation loss value.

[0170] As an embodiment, the unpaired sample images can be batched into a plurality of unpaired sample image sets, and the target sample images in the paired sample images can be batched into a plurality of target sample image sets. In each round of training, the forward network is adjusted multiple times, wherein, see Figure 8 , each adjustment process of the forward network in each round includes the following steps:

[0171] Step S801, through the current forward network and reverse network, obtain the first loss value of each unpaired sample image in the currently used unpaired sample image set, and determine the comprehensive transformation loss value of the currently used unpaired sample image set based on the first loss value of each unpaired sample image.

[0172] Step S802, based on the comprehensive transformation loss value of the currently used unpaired sample image set, adjust the model parameters of the forward network and the model parameters of the reverse network.

[0173] Step S803, obtaining the second loss value of each target sample image in the currently used target sample image set through at least one of the adjusted forward network and the reverse network, and determining the comprehensive transformation loss value of the currently used target sample image set based on the second loss value of each target sample image.

[0174] Step S804: adjusting the model parameters of the at least one network based on the comprehensive transformation loss value of the target sample image set currently in use.

[0175] The third training process: In each round of training, the model parameters of the forward network and the reverse network are adjusted based on the second loss value of each target sample image in the paired sample images; then the first loss value of each unpaired sample image is determined through the adjusted forward network and reverse network, and then the model parameters of the forward network and the reverse network are adjusted based on each unpaired sample image.

[0176] See also Fig. 9 , providing a training process for each round, including the following steps:

[0177] Step S901, obtaining a second loss value of each target sample image in a pair of sample images through at least one of the current forward network and the reverse network, and determining a second comprehensive transformation loss value based on the second loss value of each target sample image.

[0178] Step S902: adjusting the model parameters of the at least one network based on the second comprehensive transformation loss value.

[0179] Step S903: Obtain a first loss value of each unpaired sample image through the adjusted generation network, and determine a first comprehensive transformation loss value based on the first loss value of each unpaired sample image.

[0180] Step S904: Based on the first comprehensive transformation loss value, the model parameters of the forward network and the model parameters of the reverse network in the adjusted generation network are adjusted.

[0181] It should be noted that, when at least one network in the above steps S901 and S902 includes a forward network, the adjusted generated network in steps S903 and S904 includes the adjusted forward network and the unadjusted reverse network; when at least one network in steps S901 and S902 includes a reverse network, the adjusted generated network in steps S903 and S904 includes the adjusted reverse network and the unadjusted forward network; when at least one network in steps S901 and S902 includes a forward network and a reverse network, the adjusted generated network in steps S903 and S904 includes the adjusted forward network and the adjusted reverse network.

[0182] As an embodiment, the unpaired sample images can be batched into a plurality of unpaired sample image sets, and the target sample images in the paired sample images can be batched into a plurality of target sample image sets. In each round of training, the forward network is adjusted multiple times, wherein, see Fig.10 , each adjustment process of the forward network in each round includes the following steps:

[0183] Step S1001, obtaining the second loss value of each target sample image in the currently used target sample image set through at least one of the current forward network and the reverse network, and determining the comprehensive transformation loss value of the currently used target sample image set based on the second loss value of each target sample image.

[0184] Step S1002: adjusting the model parameters of the at least one network based on the comprehensive transformation loss value of the target sample image set currently in use.

[0185] Step S1003, through the adjusted generation network, obtain the transformation loss parameters of each unpaired sample image in the currently used unpaired sample image set, and determine the comprehensive transformation loss value of the currently used unpaired sample image set based on the transformation loss parameters of each unpaired sample image.

[0186] Step S1004, based on the comprehensive transformation loss value of the currently used unpaired sample image set, the model parameters of the forward network and the model parameters of the reverse network in the adjusted generation network are adjusted.

[0187] It should be noted that, when at least one network in the above steps S1001 and S1002 includes a forward network, the adjusted generated network in steps S1003 and S1004 includes the adjusted forward network and the unadjusted reverse network; when at least one network in steps S1001 and S1002 includes a reverse network, the adjusted generated network in steps S1003 and S1004 includes the adjusted reverse network and the unadjusted forward network; when at least one network in steps S1001 and S1002 includes a forward network and a reverse network, the adjusted generated network in steps S1003 and S1004 includes the adjusted forward network and the adjusted reverse network.

[0188] As an embodiment, after obtaining the trained forward network and reverse network, the trained forward network can be used to perform forward style processing on an image with a first style to obtain an image with a second style; the trained reverse network can also be used to perform reverse style processing on an image with the second style to obtain an image with the first style.

[0189] Furthermore, the images with the second style generated by the forward network and / or the images with the first style generated by the reverse network can be used as unpaired sample images; and / or, the images with the second style generated by the forward network and / or the images with the first style generated by the reverse network can be used as target sample images in paired sample images to update sample images for training, and the updated sample images are used to train the currently used forward network and reverse network to obtain updated forward network and reverse network, such as but not limited to using images obtained by sampling the currently used forward network for forward style processing as unpaired sample images and target sample images in paired sample images, training the currently used forward network and reverse network to obtain updated forward network.

[0190] The following content of the embodiment of the present disclosure provides a specific example of training a forward network and a reverse network. Fig.11 , specifically including the following steps:

[0191] Step S1101, obtaining an unpaired sample image and a paired sample image, wherein the unpaired sample image includes a first image A1 having a first style and a second image B1 having a second style; the paired sample image includes a third image A2 having a first style and a fourth image B2 having a second style, wherein the fourth image B2 is obtained by performing forward style processing on the third image A2.

[0192] Step S1102, using the forward network G_A to perform forward style processing on the first image A1 to obtain a first reference image G_A(A1) with a second style, and using the reverse network G_B to perform reverse style processing on the first reference image G_A(A1) to obtain a first reference image G_B(G_A(A1)) with a first style; and

[0193] The reverse network G_B is used to perform reverse style processing on the second image B1 to obtain a second reference image G_B(B1) with the first style, and the forward network G_A is used to perform forward style processing on the second reference image G_B(B1) to obtain a first reference image G_A(G_B(B1)) with the second style.

[0194] Step S1103, the pixel difference between the first reference image G_B (G_A (A1)) with the first style and the first image A1 is determined as the first transformation loss value, and the pixel difference between the first reference image G_A (G_B (B1)) with the second style and the second image B1 is determined as the second transformation loss value; the sum of the square value of the first transformation loss value and the square value of the second transformation loss value is determined as the first loss value L1.

[0195] That is, in this step, the first loss value L1 can be determined based on the principle of the above formula 6.

[0196] Step S1104, using the forward network G_A to perform forward style processing on the third image A2 to obtain a second reference image G_A(A2) with a second style, and using the reverse network G_B to perform reverse style processing on the fourth image B2 to obtain a second reference image G_B(B2) with a first style.

[0197] Step S1105, the pixel difference between the first reference image G_A(A2) with the second style and the fourth image B2 is determined as the third transformation loss value, and the pixel difference between the reference image G_B(B2) with the first style and the third image A2 is determined as the fourth transformation loss value; the sum of the square value of the third transformation loss value and the square value of the fourth transformation loss value is determined as the second loss value L2.

[0198] That is, in this step, the second loss value L2 can be determined based on the principle of the above formula 12a.

[0199] Step S1106: determine the first loss value L1 and the second loss value L2 as the comprehensive loss value L.

[0200] Step S1107, based on the comprehensive loss value L, adjust the model parameters of the forward network and the model parameters of the reverse network.

[0201] Specifically, in this step, the model parameters of the forward network G_A can be adjusted according to, but not limited to, the principle illustrated by the following formula 15, and the model parameters of the reverse network G_B can be adjusted according to, the principle illustrated by the following formula 16.

[0202] Formula 15:

[0203] Formula 16:

[0204] In formulas 15 and 16, L is the above comprehensive loss value, i is the identification information of the current optimization times of the forward network and the reverse network during the training process, and i is a positive integer, G_A i is the forward network after the i-th optimization, G_B i is the inverse network after the i-th optimization.

[0205] Step S1108, determine whether the forward network after adjusting the model parameters meets the convergence conditions, if yes, proceed to step S1109, otherwise proceed to step S1102.

[0206] Step S1109, determining the forward network after adjusting the model parameters as the trained forward network; and determining the reverse network after adjusting the model parameters as the trained reverse network.

[0207] like Fig.12 As shown, based on the same inventive concept, the embodiment of the present disclosure also provides a training device 1200 for an image style transfer model, the device comprising:

[0208] The first image processing unit 1201 is configured to execute processing of an unpaired sample image using a forward network and a reverse network, and determine a first loss value of the image processed by the forward network and the reverse network; the unpaired sample image includes a first image with a first style and / or a second image with a second style; the forward network is used to process the image with the first style to obtain an image with the second style; the reverse network is used to process the image with the second style to obtain an image with the first style;

[0209] The second image processing unit 1202 is configured to execute processing of a pair of sample images using at least one of the forward network and the reverse network, and determine a second loss value of the image processing by the at least one network; the pair of sample images includes an image group consisting of a third image having a first style and a fourth image having a second style obtained by processing the third image;

[0210] The training unit 1203 is configured to perform training on the forward network and the reverse network based on the first loss value and the second loss value to obtain the trained forward network and reverse network.

[0211] As an embodiment, the first image processing unit 1201 is specifically configured to perform:

[0212] The unpaired sample image is input into a forward network and a reverse network connected in series for processing to obtain a first reference image, and the first loss value is determined based on a pixel difference between the unpaired sample image and the first reference image.

[0213] As an embodiment, the first image processing unit 1201 is specifically configured to perform:

[0214] When the unpaired sample image is the first image, the forward network is used to perform forward style processing on the first image to obtain a first reference image with the second style, and the reverse network is used to perform reverse style processing on the first reference image to obtain a first reference image with the first style;

[0215] When the above-mentioned unpaired sample image is the above-mentioned second image, the above-mentioned reverse network is used to perform reverse style processing on the above-mentioned second image to obtain a second reference image with the above-mentioned first style, and the above-mentioned forward network is used to perform forward style processing on the above-mentioned second reference image to obtain a first reference image with the above-mentioned second style.

[0216] As an embodiment, when the unpaired sample includes the first image and the second image, the first image processing unit 1201 is specifically configured to execute:

[0217] Determining a first transformation loss value of the first image based on a pixel difference between the first reference image having the first style and the first image; and

[0218] Determining a second transformation loss value of the second image based on a pixel difference between the first reference image having the second style and the second image;

[0219] The first loss value is determined according to the first transformation loss value and the second transformation loss value.

[0220] As an embodiment, the second image processing unit 1202 is specifically configured to perform:

[0221] At least one of the forward network and the reverse network is used to process the target sample image to obtain a second reference image, and the second loss value is determined based on the pixel difference between the second reference image and the reference sample image corresponding to the target sample image; when the target sample image is the third image, the reference sample image is the fourth image, and when the target sample image is the fourth image, the reference sample image is the third image.

[0222] As an embodiment, the second image processing unit 1202 is specifically configured to perform:

[0223] When the target sample image is the third image and the reference sample image is the fourth image, the forward network is used to perform forward style processing on the third image to obtain a second reference image having the second style.

[0224] When the target sample image is the fourth image and the reference sample image is the third image, the inverse network is used to perform inverse style processing on the fourth image to obtain a second reference image with the first style.

[0225] As an embodiment, the target sample image includes the third image and the fourth image, and the second image processing unit 1202 is specifically configured to execute:

[0226] Determining a third transformation loss value of the third image based on a pixel difference between the second reference image having the second style and a fourth image corresponding to the third image; and

[0227] determining a fourth transformation loss value of the fourth image based on a pixel difference between the second reference image having the first style and the third image corresponding to the fourth image;

[0228] The second loss value is determined according to the third transformation loss value of the third image and the fourth transformation loss value of the fourth image.

[0229] As an embodiment, the training unit 1203 is specifically configured to execute:

[0230] Determine the result of weighted sum of the first loss value and the second loss value as the comprehensive loss value;

[0231] Based on the above comprehensive loss value, the above forward network and the above reverse network are trained.

[0232] like Fig.13 As shown, the present disclosure provides an electronic device 1300, including a processor 1301, and a memory 1302 for storing instructions executable by the processor;

[0233] The processor 1301 is configured to execute any one of the network training methods for image processing in the embodiments of the present disclosure.

[0234] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory including instructions, and the instructions can be executed by a processor of the electronic device to complete the method. Optionally, the storage medium can be a non-transitory computer-readable storage medium, for example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0235] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0236] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for an image style transfer model, characterized in that: include: Processing an unpaired sample image using a forward network and a reverse network, and determining a first loss value of the image processed by the forward network and the reverse network; The unpaired sample images include a first image with a first style and / or a second image with a second style; the forward network is used to process the image with the first style to obtain an image with the second style; the reverse network is used to process the image with the second style to obtain an image with the first style; Using at least one of the forward network and the reverse network, a pair of sample images is processed to determine a second loss value of the at least one network processing the image; the pair of sample images includes an image group consisting of a third image having a first style and a fourth image having a second style obtained by processing the third image; Based on the first loss value and the second loss value, the forward network and the reverse network are trained to obtain a trained forward network and a trained reverse network; The step of training the forward network and the reverse network based on the first loss value and the second loss value to obtain the trained forward network and the reverse network comprises: Determine a result of weighted summation of the first loss value and the second loss value as a comprehensive loss value; Based on the comprehensive loss value, the forward network and the reverse network are trained, or, Determining a first comprehensive transformation loss value according to the first loss value, and adjusting model parameters of the forward network and the reverse network based on the first comprehensive transformation loss value; A second comprehensive transformation loss value is determined according to the second loss value; and model parameters of the forward network and the reverse network are adjusted based on the second comprehensive transformation loss value.

2. The method according to claim 1, characterized in that The step of processing the unpaired sample image using the forward network and the reverse network and determining a first loss value of the image processed by the forward network and the reverse network comprises: The unpaired sample image is input into a forward network and a reverse network connected in series for processing to obtain a first reference image, and the first loss value is determined based on a pixel difference between the unpaired sample image and the first reference image.

3. The method according to claim 2, characterized in that The step of inputting the unpaired sample image into a forward network and a reverse network connected in series for processing to obtain a first reference image comprises: When the unpaired sample image is the first image, the forward network is used to perform forward style processing on the first image to obtain a first reference image with the second style, and the reverse network is used to perform reverse style processing on the first reference image to obtain a first reference image with the first style; When the unpaired sample image is the second image, the reverse network is used to perform reverse style processing on the second image to obtain a second reference image with the first style, and the forward network is used to perform forward style processing on the second reference image to obtain a first reference image with the second style.

4. The method according to claim 3, characterized in that When the unpaired sample includes the first image and the second image, the step of determining a first loss value for processing the image by the forward network and the reverse network based on a pixel difference between the unpaired sample image and the first reference image comprises: determining a first transformation loss value of the first image based on a pixel difference between the first reference image having the first style and the first image; and determining a second transformation loss value of the second image based on a pixel difference between the first reference image having the second style and the second image; The first loss value is determined according to the first transformation loss value and the second transformation loss value.

5. The method according to claim 1, characterized in that The step of determining a second loss value of the image processing by at least one of the forward network and the reverse network based on the result of processing the paired sample images comprises: At least one of the forward network and the reverse network is used to process the target sample image to obtain a second reference image, and the second loss value is determined based on the pixel difference between the second reference image and a reference sample image corresponding to the target sample image; wherein when the target sample image is the third image, the reference sample image is the fourth image, and when the target sample image is the fourth image, the reference sample image is the third image.

6. The method according to claim 5, characterized in that The step of using at least one of the forward network and the reverse network to process the target sample image to obtain a second reference image includes: When the target sample image is the third image and the reference sample image is the fourth image, the forward network is used to perform forward style processing on the third image to obtain the second reference image having the second style; When the target sample image is the fourth image and the reference sample image is the third image, the inverse network is used to perform inverse style processing on the fourth image to obtain a second reference image with the first style.

7. The method according to claim 6, characterized in that The target sample image includes the third image and the fourth image, and the step of determining the second loss value based on the pixel difference between the second reference image and the reference sample image corresponding to the target sample image includes: determining a third transformation loss value of the third image based on a pixel difference between the second reference image having the second style and a fourth image corresponding to the third image; and determining a fourth transformation loss value of the fourth image based on a pixel difference between the second reference image having the first style and a third image corresponding to the fourth image; The second loss value is determined according to a third transformation loss value of the third image and a fourth transformation loss value of the fourth image.

8. A training device for an image style transfer model, characterized in that: include: A first image processing unit is configured to process an unpaired sample image using a forward network and a reverse network, and determine a first loss value of the image processed by the forward network and the reverse network; the unpaired sample image includes a first image with a first style and / or a second image with a second style; the forward network is used to process the image with the first style to obtain an image with the second style; the reverse network is used to process the image with the second style to obtain an image with the first style; A second image processing unit is configured to execute processing of a pair of sample images using at least one of the forward network and the reverse network, and determine a second loss value of the at least one network processing the image; the pair of sample images includes an image group consisting of a third image having a first style and a fourth image having a second style obtained by processing the third image; A training unit is configured to perform training on the forward network and the reverse network based on the first loss value and the second loss value to obtain a trained forward network and a trained reverse network; The step of training the forward network and the reverse network based on the first loss value and the second loss value to obtain the trained forward network and the reverse network comprises: Determine a result of weighted summation of the first loss value and the second loss value as a comprehensive loss value; Based on the comprehensive loss value, the forward network and the reverse network are trained, or, Determining a first comprehensive transformation loss value according to the first loss value, and adjusting model parameters of the forward network and the reverse network based on the first comprehensive transformation loss value; A second comprehensive transformation loss value is determined according to the second loss value; and model parameters of the forward network and the reverse network are adjusted based on the second comprehensive transformation loss value.

9. The device according to claim 8, characterized in that The first image processing unit is specifically configured to perform: The unpaired sample image is input into a forward network and a reverse network connected in series for processing to obtain a first reference image, and the first loss value is determined based on a pixel difference between the unpaired sample image and the first reference image.

10. The device according to claim 9, characterized in that The first image processing unit is specifically configured to perform: When the unpaired sample image is the first image, the forward network is used to perform forward style processing on the first image to obtain a first reference image with the second style, and the reverse network is used to perform reverse style processing on the first reference image to obtain a first reference image with the first style; When the unpaired sample image is the second image, the reverse network is used to perform reverse style processing on the second image to obtain a second reference image with the first style, and the forward network is used to perform forward style processing on the second reference image to obtain a first reference image with the second style.

11. The device according to claim 10, characterized in that When the unpaired sample includes the first image and the second image, the first image processing unit is specifically configured to execute: determining a first transformation loss value of the first image based on a pixel difference between the first reference image having the first style and the first image; and determining a second transformation loss value of the second image based on a pixel difference between the first reference image having the second style and the second image; The first loss value is determined according to the first transformation loss value and the second transformation loss value.

12. The device according to claim 8, characterized in that The second image processing unit is specifically configured to perform: At least one of the forward network and the reverse network is used to process the target sample image to obtain a second reference image, and the second loss value is determined based on the pixel difference between the second reference image and a reference sample image corresponding to the target sample image; wherein when the target sample image is the third image, the reference sample image is the fourth image, and when the target sample image is the fourth image, the reference sample image is the third image.

13. The device according to claim 12, characterized in that The second image processing unit is specifically configured to perform: When the target sample image is the third image and the reference sample image is the fourth image, the forward network is used to perform forward style processing on the third image to obtain a second reference image having the second style; When the target sample image is the fourth image and the reference sample image is the third image, the inverse network is used to perform inverse style processing on the fourth image to obtain a second reference image with the first style.

14. The device according to claim 13, characterized in that The target sample image includes the third image and the fourth image, and the second image processing unit is specifically configured to execute: determining a third transformation loss value of the third image based on a pixel difference between the second reference image having the second style and a fourth image corresponding to the third image; and determining a fourth transformation loss value of the fourth image based on a pixel difference between the second reference image having the first style and a third image corresponding to the fourth image; The second loss value is determined according to a third transformation loss value of the third image and a fourth transformation loss value of the fourth image.

15. An electronic device, characterized in that: comprising a processor, and a memory for storing instructions executable by the processor; The processor is configured to execute the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and when the computer instructions are executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 7.