Image style migration model training method and image style migration method and device

Abstract

The invention discloses an image style migration model training method and device, and an image style migration method and device, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to AIGC content generation and other scenes based on artificial intelligence. According to the scheme, a first sample and style migration instruction information corresponding to the first sample are obtained; wherein the first sample comprises an original image and a target image; performing feature extraction on the original image and the style migration instruction information to obtain a first image feature and a first text feature; obtaining a second image feature based on the target image; performing fusion according to the first image feature, the second image feature and the first text feature to obtain a first fusion feature; and performing model training on the initial model according to the first fusion feature and the first image feature to obtain an image style migration model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC content generation based on artificial intelligence, and specifically to an image style transfer model training method, an image style transfer method, and an apparatus. Background Art

[0002] With the rapid progress of text-to-image models, the development of image style transfer has been greatly promoted. The transfer of different styles of a given image has gradually become a hot and challenging research direction, such as converting it into oil painting art style, two-dimensional anime style or other creative art styles. Summary of the invention

[0003] The present application provides an image style transfer model training method, an image style transfer method and a device.

[0004] According to one aspect of the present application, a method for training an image style transfer model is provided, comprising:

[0005] Acquire a first sample and style transfer instruction information corresponding to the first sample; wherein the first sample includes an original image and a target image, and the style transfer instruction information is used to instruct to adjust the style of the original image to the target style of the target image;

[0006] Performing feature extraction on the original image and the style migration instruction information respectively to obtain a first image feature and a first text feature;

[0007] Based on the target image, obtaining a second image feature;

[0008] fusing the first image feature, the second image feature and the first text feature to obtain a first fused feature;

[0009] The initial model is trained according to the first fusion feature and the first image feature to obtain an image style transfer model.

[0010] According to another aspect of the present application, a method for image style transfer is provided, comprising:

[0011] Acquire an original image and style transfer instruction information corresponding to the original image; wherein the style transfer instruction information is used to instruct to adjust the style of the original image to a target style;

[0012] Performing feature extraction on the original image and the style migration instruction information respectively to obtain a first image feature and a text feature;

[0013] Extract the features of the target noise to obtain the noise features;

[0014] Fusing the first image feature, the noise feature, and the text feature to obtain a fused feature;

[0015] The fused features are decoded to generate a style transfer image of the target style of the original image.

[0016] According to another aspect of the present application, there is provided an image style transfer model training device, comprising:

[0017] An acquisition module, configured to acquire a first sample and style transfer instruction information corresponding to the first sample; wherein the first sample includes an original image and a target image, and the style transfer instruction information is used to instruct to adjust the style of the original image to the target style of the target image;

[0018] A first feature extraction module, used to extract features from the original image and the style migration instruction information respectively to obtain a first image feature and a first text feature;

[0019] A second feature extraction module, used for obtaining a second image feature based on the target image;

[0020] a fusion module, configured to fuse the first image feature, the second image feature and the first text feature to obtain a first fused feature;

[0021] A training module is used to perform model training on an initial model according to the first fusion feature and the first image feature to obtain an image style transfer model.

[0022] According to another aspect of the present application, there is provided an image style transfer apparatus, comprising:

[0023] An acquisition module, used to acquire an original image and style transfer instruction information corresponding to the original image; wherein the style transfer instruction information is used to instruct to adjust the style of the original image to a target style;

[0024] A first feature extraction module, used to extract features from the original image and the style migration instruction information respectively to obtain a first image feature and a text feature;

[0025] The second feature extraction module is used to extract the feature of the target noise to obtain the noise feature;

[0026] A fusion module, used for fusing the first image feature, the noise feature and the text feature to obtain a fusion feature;

[0027] A generation module is used to decode the fusion feature and generate a style transfer image of the target style of the original image.

[0028] According to another aspect of the present application, there is provided an electronic device, including:

[0029] at least one processor; and

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor so that the at least one processor can perform the method described in the above embodiment.

[0032] According to another aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method according to the above embodiment.

[0033] According to another aspect of the present application, a computer program product is provided, including a computer program, wherein the computer program implements the steps of the method described in the above embodiment when executed by a processor.

[0034] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0036] Figure 1 A schematic diagram of a flow chart of an image style transfer model training method provided in one embodiment of the present application;

[0037] Figure 2 A schematic diagram of a flow chart of an image style transfer model training method provided in another embodiment of the present application;

[0038] Figure 3 A schematic diagram of a flow chart of an image style transfer model training method provided in another embodiment of the present application;

[0039] Figure 4 A schematic diagram of a distillation base network provided in an embodiment of the present application;

[0040] Figure 5 A schematic diagram of a flow chart of an image style transfer model training method provided in another embodiment of the present application;

[0041] Figure 6 A schematic diagram of another distillation base network provided in an embodiment of the present application;

[0042] Figure 7 A schematic diagram of a flow chart of an image style transfer method provided in one embodiment of the present application;

[0043] Figure 8 A schematic diagram of an image style transfer model training provided in an embodiment of the present application;

[0044] Fig. 9 A schematic diagram of an image style transfer model provided in an embodiment of the present application;

[0045] Fig.10 A schematic diagram of the structure of an image style transfer model training device provided in one embodiment of the present application;

[0046] Fig.11 A schematic diagram of the structure of an image style transfer device provided in one embodiment of the present application;

[0047] Fig.12 It is a block diagram of an electronic device used to implement the image style transfer model training method of an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0049] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations and do not violate public order and good morals.

[0050] The following describes the image style transfer model training method, image style transfer method, device, electronic device and storage medium of the embodiments of the present application with reference to the accompanying drawings.

[0051] Figure 1 A flowchart of an image style transfer model training method provided in one embodiment of the present application.

[0052] The image style transfer model training method of the embodiment of the present application can be executed by the image style transfer model training device of the embodiment of the present application, and the device can be configured in an electronic device.

[0053] Among them, the electronic device can be any device with computing capabilities, such as a personal computer, a mobile terminal, a server, etc. The mobile terminal can be, for example, a vehicle-mounted device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, and other hardware devices with various operating systems, touch screens and / or display screens.

[0054] like Figure 1 As shown, the image style transfer model training method includes:

[0055] Step 101: Obtain a first sample and style transfer instruction information corresponding to the first sample.

[0056] The first sample may include an original image and a target image, wherein the content of the original image is the same as that of the target image, the style of the target image is the target style, and the style of the original image is different from that of the target image.

[0057] For example, the original image is in realistic style, and the target image is in traditional Chinese painting style. For another example, the original image is in traditional Chinese painting style, and the target image is in cartoon style.

[0058] The style transfer instruction information may be used to instruct to adjust the style of the original image to the target style.

[0059] For example, the original image A1 is in realistic style, and the target image A2 is in traditional Chinese painting style. The style transfer instruction information may be “change the original image A1 to traditional Chinese painting style”.

[0060] Step 102 : extract features from the original image and the style transfer instruction information respectively to obtain a first image feature and a first text feature.

[0061] In the present application, the image feature extraction layer can be used to perform feature extraction on the original image to obtain the first image feature, and the text feature extraction layer can be used to perform feature extraction on the style migration instruction information to obtain the first text feature.

[0062] The first image feature may include information such as color, texture, content, etc. of the original image.

[0063] Step 103: obtaining a second image feature based on the target image.

[0064] Exemplarily, feature extraction may be performed directly on the target image to obtain the second image feature.

[0065] Exemplarily, noise may be added to the target image to obtain a noisy target image, and then the image feature extraction layer may be used to perform feature extraction on the noisy target image to obtain a second image feature.

[0066] Exemplarily, Gaussian noise may be added to the target image to obtain a noisy target image.

[0067] Step 104: Fusing the first image feature, the second image feature and the first text feature to obtain a first fused feature.

[0068] Exemplarily, cross-attention calculation can be performed on the first image feature and the second image feature, self-attention calculation can be performed on the first image feature and the second image feature respectively, and then the features obtained based on the cross-attention calculation and the features obtained based on the self-attention calculation are added together to obtain a first fusion feature with stronger feature capability.

[0069] Step 105: training the initial model according to the first fusion feature and the first image feature to obtain an image style transfer model.

[0070] Exemplarily, the parameters of the initial model may be adjusted according to the difference between the first fusion feature and the first image feature, and the model with the adjusted parameters may be continuously trained until the training end condition is met to obtain an image style transfer model. The initial model may refer to an initial image style transfer model.

[0071] The training end condition may be reaching a preset number of training times or the model loss being less than a preset threshold, or other conditions, which are not limited.

[0072] In an embodiment of the present application, the image migration model can adjust the original image to the style indicated by the style migration instruction information based on the style migration instruction information.

[0073] In the embodiment of the present application, the first image feature and the first text feature are obtained by extracting features from the original image and the style transfer instruction information respectively, and the second image feature is obtained based on the target image. The first image feature, the second image feature and the text feature are fused to obtain the first fused feature, and then the initial model is trained based on the first fused feature and the first image feature to obtain the image style transfer model. Therefore, in the model training process, the detailed information of the original image is incorporated as reference information and simple style transfer instruction information is incorporated, so that the image style transfer model can learn more detailed information of the original image, enhance the model's ability to refer to the details of the original image, improve the accuracy of the image style transfer model, and maintain the original image composition while achieving style transfer. Figure 1 It not only improves consistency, but also makes the interaction between the model and the user more natural.

[0074] Figure 2 A flowchart of an image style transfer model training method provided in another embodiment of the present application.

[0075] like Figure 2As shown, the image style transfer model training method includes:

[0076] Step 201: Obtain a first sample and style transfer instruction information corresponding to the first sample.

[0077] Step 202 : extract features from the original image and the style transfer instruction information respectively to obtain a first image feature and a first text feature.

[0078] Step 203, adding noise to the target image to obtain a noisy target image, and extracting features from the noisy target image to obtain second image features.

[0079] In the present application, steps 201 to 203 may refer to any implementation method in the embodiments of the present application, and thus will not be described in detail here.

[0080] Step 204, performing cross-attention calculation on the first image feature and the second image feature to obtain a first cross-attention feature.

[0081] Exemplarily, a similarity matrix between the first image feature and the second image feature can be calculated, and based on the similarity matrix, an attention weight matrix is ​​obtained, and then the first image feature is weighted according to the attention weight matrix to obtain a first cross-attention feature. The first cross-attention feature can be used to characterize the cross-attention representation of the target image to the original image.

[0082] Step 205: Obtain a third image feature through self-attention calculation based on the first image feature.

[0083] Exemplarily, self-attention calculation may be performed on the first image feature to obtain the third image feature.

[0084] Exemplarily, the first image feature and the first text feature may be concatenated to obtain a first concatenated feature, and self-attention calculation may be performed on the first concatenated feature to obtain a first intermediate feature, and the first intermediate feature may be split to obtain a third image feature.

[0085] For example, if the first image feature and the first text feature are concatenated according to the dimension, the first intermediate feature can be split according to the dimension of the first image feature to obtain the third image feature, and the feature of the remaining dimension is the new text feature, or the first intermediate feature can be split according to the dimension of the first text feature to obtain the new text feature, and the feature of the remaining dimension is the third image feature. In other words, the first intermediate feature can be split into the third image feature and the new text feature.

[0086] Therefore, by splicing the first image feature and the first text feature, the information of two different modalities can be fused to achieve cross-modal information fusion. After performing self-attention calculation on the spliced ​​features and then splitting them, the new image features can be integrated with the text features, so that the model can refer to more contextual information when understanding the original image content, thereby improving the accuracy of the image migration model.

[0087] Step 206: Obtain a fourth image feature through self-attention calculation based on the second image feature.

[0088] Exemplarily, a self-attention calculation may be performed on the second image feature to obtain a fourth image feature.

[0089] Exemplarily, the second image feature and the first text feature may be concatenated to obtain a second concatenated feature, and self-attention calculation may be performed on the second concatenated feature to obtain a second intermediate feature, and the second intermediate feature may be split to obtain a fourth image feature.

[0090] For example, if the second image feature and the first text feature are concatenated according to the dimension, the second intermediate feature can be split according to the dimension of the second image feature to obtain a fourth image feature, and the feature of the remaining dimension is a new text feature, or the second intermediate feature can be split according to the dimension of the first text feature to obtain a new text feature, and the feature of the remaining dimension is a fourth image feature. In other words, the second intermediate feature can be split into a fourth image feature and a new text feature.

[0091] Therefore, by splicing the second image feature and the first text feature, the information of two different modalities can be fused to achieve cross-modal information fusion. After performing self-attention calculation on the spliced ​​features and then splitting them, the new image features can be integrated with the text features, so that the model can refer to more contextual information when understanding the target image content, thereby improving the accuracy of the image migration model.

[0092] Step 207, fusing the first cross-attention feature, the third image feature and the fourth image feature to obtain a first fused feature.

[0093] Exemplarily, the first cross-attention feature, the third image feature, and the fourth image feature may be added together to obtain a first fusion feature.

[0094] Step 208: training the initial model according to the first fusion feature and the first image feature to obtain an image style transfer model.

[0095] Exemplarily, the third image feature can be obtained through self-attention calculation based on the first image feature, and the initial model is trained according to the difference between the first fusion feature and the third image feature to obtain the image style transfer model.

[0096] The method for obtaining the third image feature can be found in the above embodiment, so it will not be described in detail here.

[0097] Exemplarily, the parameters of the initial model may be adjusted according to the difference between the first fusion feature and the third image feature, and the model with the adjusted parameters may continue to be trained until the training end condition is met, thereby obtaining an image style transfer model.

[0098] Since the third image feature also incorporates the features of the style transfer instruction, training the initial model based on the difference between the first fused feature and the third image feature can improve the training efficiency and accuracy of the image style transfer model.

[0099] In the present application, the detail information of the original image can be added to the denoising process, and the first fusion feature with stronger representation ability can be obtained through the denoising process and the denoising process.

[0100] In an embodiment of the present application, by performing cross-attention calculation on the features of the original image and the features of the target image after noise addition, the detail information of the original image is integrated into the features of the target image after noise addition, and by capturing the global dependency through self-attention calculation, the features obtained by the cross-attention calculation and the features obtained by the self-attention calculation are fused, thereby improving the accuracy of the first fused feature, and further improving the accuracy of the image style transfer model.

[0101] Figure 3 A flowchart of an image style transfer model training method provided in another embodiment of the present application.

[0102] like Figure 3 As shown, the method for obtaining the first sample may include:

[0103] Step 301, obtaining style description information.

[0104] The style description information may be used to describe the target style. For example, the style description information is "Chinese painting style".

[0105] Step 302: adding noise to the original image to obtain a noisy original image, and performing feature extraction on the noisy original image to obtain a fifth image feature.

[0106] In the present application, noise can be added to the original image to obtain the noisy original image, and then the image feature extraction layer is used to extract features from the noisy original image to obtain the fifth image feature.

[0107] Exemplarily, Gaussian noise may be added to the original image to obtain a noisy original image.

[0108] Step 303: extract features from the facial information in the original image to obtain facial features.

[0109] In the present application, facial information can be extracted from the original image, and features of the facial information can be extracted to obtain facial features.

[0110] Step 304: extract features from the style description information to obtain second text features.

[0111] In the present application, the text feature extraction layer can be used to extract features from the style description information to obtain the second text feature.

[0112] Step 305 , performing denoising processing according to the fifth image feature, the face feature and the second text feature to generate a target image of a target style.

[0113] As a possible implementation method, cross-attention calculation can be performed on the fifth image feature and the face feature, self-attention calculation can be performed on the face feature and the second text feature respectively, and then the features obtained by the cross-attention calculation and the features obtained by the self-attention calculation are added to obtain the second fusion feature. Thus, the face information can be added to the denoising process to ensure a high similarity between the original image and the face after style transfer.

[0114] As another possible implementation method, cross-attention calculation can be performed based on the fifth image feature and the facial feature to obtain a second cross-attention feature, and self-attention calculation can be performed based on the fifth image feature and the second text feature to obtain a sixth image feature. The second cross-attention feature and the sixth image feature are then fused to obtain a second fused feature, and the second fused feature is decoded to obtain a target image.

[0115] Exemplarily, the fifth image feature and the second text feature may be concatenated to obtain a third concatenated feature, and self-attention calculation may be performed on the third concatenated feature to obtain a third intermediate feature, and the third intermediate feature may be split to obtain a sixth image feature.

[0116] For example, the third intermediate feature may be split according to the dimension of the fifth image feature or according to the dimension of the second text feature to obtain the sixth image feature, thereby splitting the third intermediate feature into the sixth image feature and a new text feature.

[0117] Exemplarily, the second cross-attention feature may be added to the sixth image feature to obtain a second fusion feature.

[0118] Therefore, by performing cross-attention calculation on the features of the original image with added noise and the facial features, the facial detail information is integrated into the features of the original image with added noise, and the global dependency is captured through self-attention calculation. The features obtained by the cross-attention calculation and the features obtained by the self-attention calculation are fused, so as to improve the accuracy of the second fused features, and then improve the accuracy of the image style transfer model.

[0119] Step 306: Determine a first sample according to the original image and the target image.

[0120] Exemplarily, the original image and the target image can be used as the first sample. Thus, by inputting description information of different styles, multiple images of different styles of the same person can be generated, thereby obtaining multiple first samples.

[0121] For half-body stylization, there is often a lack of facial similarity. For example, during training, the original image can be added to the model diffusion generation process to ensure the consistency of the human body posture, while integrating facial detail information to ensure that the face has a high similarity.

[0122] For ease of understanding, the following Figure 4 To explain, Figure 4 A schematic diagram of a distillation base network provided in an embodiment of the present application. Figure 4 In , the first information flow is the information flow to which the text features are obtained based on the style description information, and the second information flow is the information flow to which the image features are obtained based on the original image after adding noise. Figure 4 As shown, the second text feature of the style description information can be spliced ​​with the fifth image feature of the original image after noise addition, and the spliced ​​feature is input into the self-attention layer 410 for self-attention calculation, and then the calculated feature is separated to obtain the sixth image feature and the new text feature, wherein the new text feature belongs to the first information flow, and the fifth image feature and the face feature are input into the cross-attention layer 420 for cross-attention calculation to obtain the second cross-attention feature, and then the sixth image feature is added to the second cross-attention feature to obtain a new image feature, which belongs to the second information flow.

[0123] It should be noted that Figure 4 FIG. 1 shows a structure in which a layer of a distilled base network includes a self-attention layer 410 and a cross-attention layer 420. The distilled base network may include multiple layers of such a structure. After multiple layers of processing, the features corresponding to the second information stream output by the last layer are decoded to obtain a target image.

[0124] It can be seen that based on Figure 4 The network shown can add the original image to the base network diffusion generation process to ensure the consistency of human posture, and add facial details through cross-attention calculation to ensure high similarity of faces. Therefore, by inputting description information of different styles, multiple images of the same person in different styles can be generated.

[0125] In an embodiment of the present application, when generating the target image in the first sample, the original image detail information and the face detail information are integrated, which can not only ensure that the posture of the target image is consistent with that of the original image, but also improve the similarity between the image after style transfer and the face in the original image. The generated target image is then used to train the image style transfer model, which can improve the style transfer ability of the image style transfer model for half-body images.

[0126] Figure 5 A flowchart of an image style transfer model training method provided in another embodiment of the present application.

[0127] like Figure 5 As shown, the method for obtaining the first sample may include:

[0128] Step 501, obtaining image description information.

[0129] The image description information may be used to describe the image after style transfer, and the image description information may include the content of the original image, the target style, etc.

[0130] For example, the image description information is "a cartoon-style kitten".

[0131] Step 502: extract the characteristics of the target noise to obtain the noise characteristics.

[0132] Exemplarily, the target noise may be initialization noise, such as Gaussian noise.

[0133] Step 503, obtaining a third fusion feature through a control network according to the noise feature and the original image.

[0134] In the present application, the image features and noise features of the original image can be input into the control network, and the noise features and the original features can be processed by the control network to obtain the third fusion features output by the control network.

[0135] Among them, the original image can be used as a control condition and injected into multiple levels for denoising through the control network. Injecting the control signal into the features of each layer can ensure that the control conditions can be referenced in different dimensions, so that the overall structure and details of the target image finally generated are controlled by the original image, ensuring that the overall layout of the target image is consistent with the original image.

[0136] Exemplarily, the control network may be ControlNet.

[0137] Step 504: extract features from the image description information to obtain a third text feature.

[0138] In the present application, the image description information may be subjected to feature extraction through a text feature extraction layer to obtain a third text feature.

[0139] Step 505 , performing denoising processing according to the noise feature, the third fusion feature and the third text feature to generate a target image of a target style.

[0140] In the present application, the seventh image feature can be obtained through self-attention calculation based on the noise feature and the third text feature, the seventh image feature and the third fusion feature are fused to obtain the fourth fusion feature, and the fourth fusion feature is decoded to obtain the target image.

[0141] Exemplarily, the noise feature and the third text feature may be concatenated to obtain a fourth concatenated feature, and self-attention calculation may be performed on the fourth concatenated feature to obtain a fourth intermediate feature, and the fourth intermediate feature may be split to obtain a seventh image feature.

[0142] For example, the fourth intermediate feature can be split according to the dimension of the noise feature to obtain the eighth image feature, or the fourth intermediate feature can be split according to the third text feature to obtain the eighth image feature. This is similar to the method of splitting the first intermediate feature recorded in the above embodiment, and will not be repeated here.

[0143] Exemplarily, the noise feature may be added to the third fusion feature to obtain a fourth fusion feature.

[0144] Therefore, by calculating the self-attention based on the noise feature and the third text feature, the two features can be fused to enrich the feature information, and the noise feature and the third fusion feature can be fused to incorporate more detail information of the original image, thereby improving the accuracy of the fourth fusion feature.

[0145] Step 506: Determine a first sample according to the original image and the target image.

[0146] Exemplarily, the original image and the target image may be used as the first sample.

[0147] Based on the first sample acquisition method of this embodiment, style transfer training samples of complex scenes can be generated in combination with the text description after style transfer.

[0148] For ease of understanding, the following Figure 6 To explain, Figure 6 A schematic diagram of another distillation base network provided in an embodiment of the present application. Figure 6 In the above, the first information flow is an information flow of text features obtained based on the rewritten description after style, and the rewritten description after style refers to the above-mentioned image description information. In the training phase, the second information flow is an information flow of image features obtained based on the target image after adding noise. In the inference phase, the second information flow is an information flow of features obtained based on the initialized noise.

[0149] Taking the inference stage as an example, the image features and noise features of the original image can be input into ControlNet 610 for processing to obtain the third fused features, and the third text features and noise features of the rewritten description of the style are spliced ​​and input into the self-attention layer 620 for self-attention calculation, and the calculated features are split into two parts: one part is the new text features, which belong to the second information flow, and the other part is the seventh image features. The seventh image features are added to the third fused features to obtain the fourth fused features, which belong to the second information flow.

[0150] It should be noted that Figure 6 FIG. 1 shows a structure in which a layer of a distilled base network includes a ControlNet 610 and a self-attention layer 620. The distilled base network may include multiple layers of such a structure. After multiple layers of processing, the features corresponding to the second information stream output by the last layer are decoded to obtain a target image.

[0151] For style transfer of complex scenes such as multiple people, buildings, and natural scenery, there is often a lack of consistency in the overall layout. Figure 6 The network shown, through ControlNet 610, can process complex line input and extract more layout details in the original image, so that in complex scenes such as multiple people, buildings, natural scenery, etc., combined with the stylized rewritten text description, it can generate training samples for complex scene style transfer.

[0152] In an embodiment of the present application, when generating a target image in the first sample, the original image is used as an additional control condition, and the generated target image is controlled by controlling the network to process more complex line inputs and extract more layout details from the original image, so that the generated target image is used to train the image style transfer model. The style-transferred image generated by the model can be consistent with the overall layout of the original image, thereby improving the style transfer capability of the image style transfer model for complex scenes.

[0153] Optionally, the Figure 3 and Figure 5 The training samples generated by the method shown are used to obtain a training data set for the image style transfer model. The image style transfer model is trained using the training data set.

[0154] In order to implement the above embodiment, the embodiment of the present application also proposes an image style transfer method. Figure 7 A flowchart of an image style transfer method provided in one embodiment of the present application.

[0155] like Figure 7 As shown, the image style transfer method includes:

[0156] Step 701: Obtain an original image and style transfer instruction information corresponding to the original image.

[0157] Step 702: extract features from the original image and the style transfer instruction information respectively to obtain first image features and text features.

[0158] In the present application, step 701-step 702 can be implemented in any of the embodiments of the present application, so they will not be described in detail here.

[0159] Step 703: extract features of the target noise to obtain noise features.

[0160] Exemplarily, the target noise may be initialization noise, such as Gaussian noise.

[0161] Step 704: Fusing the first image feature, the noise feature, and the text feature to obtain a fused feature.

[0162] Exemplarily, cross-attention calculation can be performed on the first image feature and the noise feature to obtain a cross-attention feature, and based on the first image feature, a second image feature can be obtained through self-attention calculation, and based on the noise feature, a third image feature can be obtained through self-attention calculation, and then the cross-attention feature, the second image feature and the third image feature are fused to obtain a fused feature.

[0163] Therefore, by performing cross-attention calculation on the features of the original image and the noise features, the detail information of the original image is integrated into the noise features, and the global dependencies are captured through self-attention calculation. The features obtained by the cross-attention calculation and the features obtained by the self-attention calculation are fused, so as to improve the accuracy of the fused features and thus improve the accuracy of the generated target image.

[0164] Exemplarily, the noise feature and the text feature may be concatenated to obtain a concatenated feature, self-attention calculation may be performed on the concatenated feature to obtain an intermediate feature, and the intermediate feature may be split according to the dimension of the noise feature to obtain a third image feature.

[0165] Therefore, by concatenating the noise features and the text features and then performing self-attention calculation, the text information and the noise information can be fused, so that the text information is integrated into the third image feature, enriching the information contained in the third image feature.

[0166] For obtaining the second image feature through self-attention calculation based on the first image feature, the first image feature and the text feature can be spliced, and the spliced ​​features can be subjected to self-attention calculation, and then split to obtain the second image feature.

[0167] Step 705: decode the fused features to generate a style-transferred image of the target style of the original image.

[0168] In the present application, the fusion features can be decoded in pairs to generate a style-transferred image of the target style of the original image. The style-transferred image refers to an image after style transfer, the style of the style-transferred image is the target style, and the content of the style-transferred image is consistent with the content of the original image.

[0169] In the embodiment of the present application, the first image feature and the text feature are obtained by extracting the features of the original image and the style transfer instruction information respectively, the target noise is extracted to obtain the noise feature, the first image feature, the noise feature and the text feature are fused to obtain the fused feature, the fused feature is decoded, and the style transfer image of the target style is generated. Thus, by incorporating the detailed information of the original image as reference information and incorporating simple style transfer instruction information to generate a style transfer image, not only can the consistency of the image and text be ensured, but also different style images can be generated through simple style transfer instruction information, which has strong flexibility and more natural interaction with the user.

[0170] In order to implement the above embodiment, the embodiment of the present application also proposes an image style transfer method. The image style transfer method includes: obtaining an original image and style transfer instruction information corresponding to the original image, inputting the original image and the style transfer instruction information into an image style transfer model, and obtaining a style transfer image of the target style output by the image style transfer model. The style transfer instruction information is used to instruct to adjust the style of the original image to the target style.

[0171] Exemplarily, the image style transfer model in this embodiment can be used Figure 1-Figure 3 , Figure 5 The image style transfer model described in any one of the embodiments is trained.

[0172] For ease of understanding, the image style transfer model training method and the image style transfer method of the embodiment of the present application are described below in combination with Figure 8 and Fig. 9 To explain, Figure 8 A training diagram of an image style transfer model provided in an embodiment of the present application, Fig. 9 A schematic diagram of an image style transfer model provided in an embodiment of the present application.

[0173] like Figure 8 As shown, an image pair of an original image and a target image can be randomly selected from the training data set 810, for example, a realistic style image and a Chinese painting style image corresponding to the image are selected, the realistic style image is input as the original image to the parallel reference network 820, the Chinese painting style image is used as the target image and noise is added to the base network 830 for noise training, and the style transfer instruction information corresponding to the selected two images "please change this image to Chinese painting style" is input into the parallel reference network 820 and the base network 830 respectively. The specific structure of the base network 830 is shown in FIG. Fig. 9 910, the specific structure of the parallel reference network 820 can be found in Fig. 9 920 of them.

[0174] For information about style transfer instructions, noisy target images, and original image processing, see Fig. 9 , Fig. 9 The image style transfer model shown includes a base network 910 and a parallel reference network 920 . Fig. 9 In the example, the first information flow in the base network 910 is an information flow to which text features are obtained based on the style transfer instruction information, and the second information flow is an information flow to which image features are obtained based on the target image after adding noise, and the parallel reference network 920 has an original image information flow and a second information flow. The original image information flow can be an information flow to which image features are obtained based on the original image,

[0175] like Fig. 9 As shown, the parallel reference network 920 interacts with the base network 910 through a cross-attention layer 912 between the features of the original image and the features of the noisy target image.

[0176] It should be noted that Fig. 9 The figure shows a structure in which a layer of the base network 910 includes a self-attention layer 911 and a cross-attention layer 912, and a self-attention layer 921 in the parallel reference network 920. After the style transfer instruction information, the noisy target image, and the original image are input, they are processed through multiple layers of such a structure. The image features corresponding to the second information stream output by the last layer of the base network 910 can be compared with the image features corresponding to the original image information stream output by the latter layer in the parallel reference network 920, and the model parameters can be adjusted according to the comparison results.

[0177] in, Figure 8The training data set 810 in can be based on 4 and Figure 6 The stylized datasets of different scenes generated by the distilled base network shown include pairs of original and target images.

[0178] In the inference stage, after the style transfer instruction information and the original image are input, the image features corresponding to the second information stream output by the last layer of the base network 910 can be fused with the image features corresponding to the original image information stream output by the next layer in the parallel reference network 920, and the fused features can be decoded to obtain a style transfer image.

[0179] The image style transfer model of the embodiment of the present application adopts a dual information flow base network, and adds original image detail information and style transfer instruction information through a parallel reference network.

[0180] The image style transfer method of the embodiment of the present application can use the distillation base Wenshengtu model to generate stylized data sets in different scenes, improve the quality of the stylized data sets, expand and unify the scenes included in the stylized data sets, and then use high-quality data to improve the effect of the style transfer model. In addition, through the parallel reference network, the model is given the stylization ability while enhancing the model's reference ability to the original image, thereby improving the accuracy of the model.

[0181] In order to implement the above embodiment, the embodiment of the present application also proposes an image style transfer model training device. Fig.10 A schematic diagram of the structure of an image style transfer model training device provided in one embodiment of the present application.

[0182] like Fig.10 As shown, the image style transfer model training device 1000 includes:

[0183] The acquisition module 1010 is used to acquire a first sample and style transfer instruction information corresponding to the first sample; wherein the first sample includes an original image and a target image, and the style transfer instruction information is used to instruct to adjust the style of the original image to the target style of the target image;

[0184] A first feature extraction module 1020, configured to extract features from the original image and the style transfer instruction information respectively to obtain a first image feature and a first text feature;

[0185] A second feature extraction module 1030, configured to obtain a second image feature based on the target image;

[0186] A fusion module 1040, configured to fuse the first image feature, the second image feature and the first text feature to obtain a first fused feature;

[0187] The training module 1050 is used to perform model training on the initial model according to the first fusion feature and the first image feature to obtain an image style transfer model.

[0188] Optionally, the fusion module 1040 is used to:

[0189] Performing cross-attention calculation on the first image feature and the second image feature to obtain a first cross-attention feature;

[0190] Obtaining a third image feature through self-attention calculation according to the first image feature;

[0191] According to the second image feature, a fourth image feature is obtained by self-attention calculation;

[0192] The first cross-attention feature, the third image feature and the fourth image feature are fused to obtain a first fused feature.

[0193] Optionally, the fusion module 1040 is used to:

[0194] splicing the first image feature and the first text feature to obtain a first splicing feature;

[0195] Performing self-attention calculation on the first concatenated feature to obtain a first intermediate feature;

[0196] The first intermediate feature is split according to the dimension of the first image feature to obtain the third image feature.

[0197] Optionally, the fusion module 1040 is used to:

[0198] splicing the second image feature and the first text feature to obtain a second splicing feature;

[0199] Performing self-attention calculation on the second concatenated feature to obtain a second intermediate feature;

[0200] The second intermediate feature is split according to the dimension of the second image feature to obtain the fourth image feature.

[0201] Optionally, the training module 1050 is used to:

[0202] Obtaining a third image feature through self-attention calculation according to the first image feature;

[0203] The initial model is trained according to the difference between the first fusion feature and the third image feature to obtain the image style transfer model.

[0204] Optionally, the acquisition module 1010 is used to:

[0205] Acquire style description information; wherein the style description information is used to describe the target style;

[0206] Adding noise to the original image to obtain the noisy original image, and performing feature extraction on the noisy original image to obtain a fifth image feature;

[0207] Extracting features of the facial information in the original image to obtain facial features;

[0208] Performing feature extraction on the style description information to obtain a second text feature;

[0209] Performing denoising processing according to the fifth image feature, the facial feature and the second text feature to generate the target image of the target style;

[0210] The first sample is determined according to the original image and the target image.

[0211] Optionally, the acquisition module 1010 is used to:

[0212] Performing cross-attention calculation according to the fifth image feature and the face feature to obtain a second cross-attention feature;

[0213] Obtaining a sixth image feature through self-attention calculation according to the fifth image feature and the second text feature;

[0214] fusing the second cross-attention feature and the sixth image feature to obtain the second fused feature;

[0215] The second fusion feature is decoded to obtain the target image.

[0216] Optionally, the acquisition module 1010 is used to:

[0217] Acquire image description information; wherein the image description information includes the content of the original image and the target style;

[0218] Extract the features of the target noise to obtain the noise features;

[0219] Obtaining a third fusion feature through a control network according to the noise feature and the original image;

[0220] Performing feature extraction on the image description information to obtain a third text feature;

[0221] Perform denoising processing according to the noise feature, the third fusion feature and the third text feature to generate the target image of the target style;

[0222] The first sample is determined according to the original image and the target image.

[0223] Optionally, the acquisition module 1010 is used to:

[0224] Obtaining a seventh image feature through self-attention calculation according to the noise feature and the third text feature;

[0225] fusing the seventh image feature and the third fusion feature to obtain the fourth fusion feature;

[0226] The fourth fusion feature is decoded to obtain the target image.

[0227] It should be noted that the explanation of the aforementioned image style transfer model training method embodiment is also applicable to the image style transfer model training device of this embodiment, so it will not be repeated here.

[0228] In the embodiment of the present application, the first image feature and the first text feature are obtained by extracting features from the original image and the style transfer instruction information respectively, and the second image feature is obtained based on the target image. The first image feature, the second image feature and the text feature are fused to obtain the first fused feature, and then the model is trained based on the first fused feature and the first image feature to obtain the image style transfer model. Therefore, in the model training process, the detailed information of the original image is incorporated as reference information and simple style transfer instruction information is incorporated, so that the image style transfer model can learn more detailed information of the original image, enhance the model's ability to refer to the details of the original image, improve the accuracy of the image style transfer model, and maintain the original image composition while achieving style transfer. Figure 1 It not only improves consistency, but also makes the interaction between the model and the user more natural.

[0229] In order to implement the above embodiment, the embodiment of the present application also proposes an image style transfer device. Fig.11 A schematic diagram of the structure of an image style transfer device provided in one embodiment of the present application.

[0230] like Fig.11 As shown, the image style transfer device 1100 includes:

[0231] The acquisition module 1110 is used to acquire an original image and style transfer instruction information corresponding to the original image; wherein the style transfer instruction information is used to instruct to adjust the style of the original image to a target style;

[0232] A first feature extraction module 1120, configured to extract features from the original image and the style migration instruction information respectively to obtain first image features and text features;

[0233] The second feature extraction module 1130 is used to extract features of the target noise to obtain noise features;

[0234] A fusion module 1140 is used to fuse the first image feature, the noise feature and the text feature to obtain a fusion feature;

[0235] The generating module 1150 is used to decode the fused features and generate a style transfer image of the target style of the original image.

[0236] Optionally, the fusion module 1140 is used to:

[0237] Performing cross attention calculation on the first image feature and the noise feature to obtain a cross attention feature;

[0238] According to the first image feature, obtain a second image feature through self-attention calculation;

[0239] Obtaining a third image feature through self-attention calculation according to the noise feature;

[0240] The cross-attention feature, the second image feature and the third image feature are fused to obtain a fused feature.

[0241] Optionally, the fusion module 1140 is used to:

[0242] Splicing the noise feature and the text feature to obtain a spliced ​​feature;

[0243] Performing self-attention calculation on the concatenated features to obtain intermediate features;

[0244] The intermediate feature is split according to the dimension of the noise feature to obtain the third image feature.

[0245] It should be noted that the explanation of the above-mentioned image style transfer method embodiment is also applicable to the image style transfer device of this embodiment, so it will not be repeated here.

[0246] In the embodiment of the present application, the first image feature and the text feature are obtained by extracting the features of the original image and the style transfer instruction information respectively, the target noise is extracted to obtain the noise feature, the first image feature, the noise feature and the text feature are fused to obtain the fused feature, the fused feature is decoded, and the style transfer image of the target style is generated. Thus, by incorporating the detailed information of the original image as reference information and incorporating simple style transfer instruction information to generate a style transfer image, not only can the consistency of the image and text be ensured, but also different style images can be generated through simple style transfer instruction information, which has strong flexibility and more natural interaction with the user.

[0247] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0248] Fig.12 A schematic block diagram of an example electronic device 1200 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0249] like Fig.12 As shown, the device 1200 includes a computing unit 1201, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 1202 or a computer program loaded from a storage unit 1208 to a RAM (Random Access Memory) 1203. In the RAM 1203, various programs and data required for the operation of the device 1200 can also be stored. The computing unit 1201, the ROM 1202, and the RAM 1203 are connected to each other via a bus 1204. An I / O (Input / Output) interface 1205 is also connected to the bus 1204.

[0250] A number of components in the device 1200 are connected to the I / O interface 1205, including: an input unit 1206, such as a keyboard, a mouse, etc.; an output unit 1207, such as various types of displays, speakers, etc.; a storage unit 1208, such as a disk, an optical disk, etc.; and a communication unit 1209, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1209 allows the device 1200 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0251] The computing unit 1201 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1201 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 1201 performs the various methods and processes described above, such as the image style transfer model training method. For example, in some embodiments, the image style transfer model training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1208. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1200 via the ROM 1202 and / or the communication unit 1209. When the computer program is loaded into the RAM 1203 and executed by the computing unit 1201, one or more steps of the image style transfer model training method described above may be performed. Alternatively, in other embodiments, the computing unit 1201 may be configured to execute the image style transfer model training method in any other appropriate manner (for example, by means of firmware).

[0252] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, FPGAs (Field Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), ASSPs (Application Specific Standard Products), SOCs (System On Chips), CPLDs (Complex Programmable Logic Devices), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that may be executed and / or interpreted on a programmable system including at least one programmable processor that may be a dedicated or general-purpose programmable processor that may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0253] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0254] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, EPROM (Electrically Programmable Read-Only-Memory) or flash memory, optical fiber, CD-ROM (Compact Disc Read-Only Memory), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0255] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (Cathode-Ray Tube) or LCD (Liquid Crystal Display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0256] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: LAN (Local Area Network), WAN (Wide Area Network), the Internet, and blockchain networks.

[0257] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0258] It should be noted that the electronic device used to implement the image style transfer method of the embodiment of the present application is similar to the structure of the above-mentioned electronic device, so it will not be described in detail here.

[0259] According to an embodiment of the present application, the present application also provides a computer program product. When the instruction processor in the computer program product is executed, it executes the image style transfer model training method proposed in the above embodiment of the present application, or executes the image style transfer method proposed in the above embodiment of the present application.

[0260] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0261] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A method for training an image style transfer model, comprising: Acquire a first sample and style transfer instruction information corresponding to the first sample; wherein the first sample includes an original image and a target image, and the style transfer instruction information is used to instruct to adjust the style of the original image to the target style of the target image; Performing feature extraction on the original image and the style migration instruction information respectively to obtain a first image feature and a first text feature; Based on the target image, obtaining a second image feature; fusing the first image feature, the second image feature and the first text feature to obtain a first fused feature; The initial model is trained according to the first fusion feature and the first image feature to obtain an image style transfer model.

2. The method of claim 1, wherein: The fusing the first image feature, the second image feature and the first text feature to obtain a first fused feature includes: Performing cross-attention calculation on the first image feature and the second image feature to obtain a first cross-attention feature; Obtaining a third image feature through self-attention calculation according to the first image feature; According to the second image feature, a fourth image feature is obtained by self-attention calculation; The first cross-attention feature, the third image feature and the fourth image feature are fused to obtain a first fused feature.

3. The method of claim 2, wherein: The obtaining of the third image feature by self-attention calculation according to the first image feature includes: splicing the first image feature and the first text feature to obtain a first splicing feature; Performing self-attention calculation on the first concatenated feature to obtain a first intermediate feature; The first intermediate feature is split according to the dimension of the first image feature to obtain the third image feature.

4. The method of claim 2, wherein: The obtaining of the fourth image feature by self-attention calculation according to the second image feature includes: splicing the second image feature and the first text feature to obtain a second splicing feature; Performing self-attention calculation on the second concatenated feature to obtain a second intermediate feature; The second intermediate feature is split according to the dimension of the second image feature to obtain the fourth image feature.

5. The method of claim 1, wherein: The performing model training on the initial model according to the first fusion feature and the first image feature to obtain the image style transfer model includes: Obtaining a third image feature through self-attention calculation according to the first image feature; The initial model is trained according to the difference between the first fusion feature and the third image feature to obtain the image style transfer model.

6. The method of claim 1, wherein: The obtaining of the first sample comprises: Acquire style description information; wherein the style description information is used to describe the target style; Adding noise to the original image to obtain the noisy original image, and performing feature extraction on the noisy original image to obtain a fifth image feature; Extracting features of the facial information in the original image to obtain facial features; Performing feature extraction on the style description information to obtain a second text feature; Performing denoising processing according to the fifth image feature, the facial feature and the second text feature to generate the target image of the target style; The first sample is determined according to the original image and the target image.

7. The method of claim 6, wherein: The performing denoising processing according to the fifth image feature, the face feature and the second text feature to generate the target image of the target style includes: Performing cross-attention calculation according to the fifth image feature and the face feature to obtain a second cross-attention feature; Obtaining a sixth image feature through self-attention calculation according to the fifth image feature and the second text feature; fusing the second cross-attention feature and the sixth image feature to obtain the second fused feature; The second fusion feature is decoded to obtain the target image.

8. The method of claim 1, wherein: The obtaining of the first sample comprises: Acquire image description information; wherein the image description information includes the content of the original image and the target style; Extract the features of the target noise to obtain the noise features; Obtaining a third fusion feature through a control network according to the noise feature and the original image; Performing feature extraction on the image description information to obtain a third text feature; Performing denoising processing according to the noise feature, the third fusion feature and the third text feature to generate the target image of the target style; The first sample is determined according to the original image and the target image.

9. The method of claim 8, wherein: The performing denoising processing according to the noise feature, the third fusion feature and the third text feature to generate the target image of the target style includes: Obtaining a seventh image feature through self-attention calculation according to the noise feature and the third text feature; fusing the seventh image feature and the third fusion feature to obtain the fourth fusion feature; The fourth fusion feature is decoded to obtain the target image.

10. A method for image style transfer, comprising: Acquire an original image and style transfer instruction information corresponding to the original image; wherein the style transfer instruction information is used to instruct to adjust the style of the original image to a target style; Performing feature extraction on the original image and the style migration instruction information respectively to obtain a first image feature and a text feature; Extract the features of the target noise to obtain the noise features; Fusing the first image feature, the noise feature, and the text feature to obtain a fused feature; The fused features are decoded to generate a style transfer image of the target style of the original image.

11. The method of claim 10, wherein: The fusing the first image feature, the noise feature and the text feature to obtain a fused feature includes: Performing cross attention calculation on the first image feature and the noise feature to obtain a cross attention feature; According to the first image feature, obtain a second image feature through self-attention calculation; Obtaining a third image feature through self-attention calculation according to the noise feature; The cross-attention feature, the second image feature and the third image feature are fused to obtain a fused feature.

12. The method of claim 11, wherein: The step of obtaining a third image feature by self-attention calculation according to the noise feature includes: Splicing the noise feature and the text feature to obtain a spliced ​​feature; Performing self-attention calculation on the concatenated features to obtain intermediate features; The intermediate feature is split according to the dimension of the noise feature to obtain the third image feature.

13. An image style transfer model training device, comprising: An acquisition module, configured to acquire a first sample and style transfer instruction information corresponding to the first sample; wherein the first sample includes an original image and a target image, and the style transfer instruction information is used to instruct to adjust the style of the original image to the target style of the target image; A first feature extraction module, used to extract features from the original image and the style migration instruction information respectively to obtain a first image feature and a first text feature; A second feature extraction module, used for obtaining a second image feature based on the target image; a fusion module, configured to fuse the first image feature, the second image feature and the first text feature to obtain a first fused feature; A training module is used to perform model training on an initial model according to the first fusion feature and the first image feature to obtain an image style transfer model.

14. The device of claim 13, wherein: The fusion module is used to: Performing cross-attention calculation on the first image feature and the second image feature to obtain a first cross-attention feature; Obtaining a third image feature through self-attention calculation according to the first image feature; According to the second image feature, a fourth image feature is obtained by self-attention calculation; The first cross-attention feature, the third image feature and the fourth image feature are fused to obtain a first fused feature.

15. The device of claim 14, wherein: The fusion module is used to: splicing the first image feature and the first text feature to obtain a first splicing feature; Performing self-attention calculation on the first concatenated feature to obtain a first intermediate feature; The first intermediate feature is split according to the dimension of the first image feature to obtain the third image feature.

16. The device of claim 14, wherein: The fusion module is used to: splicing the second image feature and the first text feature to obtain a second splicing feature; Performing self-attention calculation on the second concatenated feature to obtain a second intermediate feature; The second intermediate feature is split according to the dimension of the second image feature to obtain the fourth image feature.

17. The device of claim 13, wherein: The training module is used to: Obtaining a third image feature through self-attention calculation according to the first image feature; The initial model is trained according to the difference between the first fusion feature and the third image feature to obtain the image style transfer model.

18. The device of claim 13, wherein: The acquisition module is used to: Acquire style description information; wherein the style description information is used to describe the target style; Adding noise to the original image to obtain the noisy original image, and performing feature extraction on the noisy original image to obtain a fifth image feature; Extracting features of the facial information in the original image to obtain facial features; Performing feature extraction on the style description information to obtain a second text feature; Performing denoising processing according to the fifth image feature, the facial feature and the second text feature to generate the target image of the target style; The first sample is determined according to the original image and the target image.

19. The device of claim 18, wherein: The acquisition module is used to: Performing cross-attention calculation according to the fifth image feature and the face feature to obtain a second cross-attention feature; Obtaining a sixth image feature through self-attention calculation according to the fifth image feature and the second text feature; fusing the second cross-attention feature and the sixth image feature to obtain the second fused feature; The second fusion feature is decoded to obtain the target image.

20. The device of claim 13, wherein: The acquisition module is used to: Acquire image description information; wherein the image description information includes the content of the original image and the target style; Extract the features of the target noise to obtain the noise features; Obtaining a third fusion feature through a control network according to the noise feature and the original image; Performing feature extraction on the image description information to obtain a third text feature; Performing denoising processing according to the noise feature, the third fusion feature and the third text feature to generate the target image of the target style; The first sample is determined according to the original image and the target image.

21. The device of claim 20, wherein: The acquisition module is used to: Obtaining a seventh image feature through self-attention calculation according to the noise feature and the third text feature; fusing the seventh image feature and the third fusion feature to obtain the fourth fusion feature; The fourth fusion feature is decoded to obtain the target image.

22. An image style transfer device, comprising: An acquisition module, used to acquire an original image and style transfer instruction information corresponding to the original image; wherein the style transfer instruction information is used to instruct to adjust the style of the original image to a target style; A first feature extraction module, used to extract features from the original image and the style migration instruction information respectively to obtain a first image feature and a text feature; The second feature extraction module is used to extract the feature of the target noise to obtain the noise feature; A fusion module, used for fusing the first image feature, the noise feature and the text feature to obtain a fusion feature; A generation module is used to decode the fusion feature and generate a style transfer image of the target style of the original image.

23. The device of claim 22, wherein: The fusion module is used to: Performing cross attention calculation on the first image feature and the noise feature to obtain a cross attention feature; According to the first image feature, obtain a second image feature through self-attention calculation; Obtaining a third image feature through self-attention calculation according to the noise feature; The cross-attention feature, the second image feature and the third image feature are fused to obtain a fused feature.

24. The device of claim 23, wherein: The fusion module is used to: Splicing the noise feature and the text feature to obtain a spliced ​​feature; Performing self-attention calculation on the concatenated features to obtain intermediate features; The intermediate feature is split according to the dimension of the noise feature to obtain the third image feature.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1-9, or to execute the method of any one of claims 10-12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1-9, or to execute the method according to any one of claims 10-12.

27. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 9, or implements the steps of the method according to any one of claims 10 to 12.

Images