Image style migration method and device, equipment, storage medium and program product

Abstract

The invention relates to an image style migration method and apparatus, a device, a storage medium and a program product. The method comprises the steps of obtaining a to-be-processed image and a style image; extracting first style information of the to-be-processed image and content information of the style image, and generating guidance information based on the first style information and the content information; carrying out noise addition processing on the to-be-processed image, and carrying out negative guidance on the noise addition processing through the guidance information to obtain a noise image; and extracting second style information of the style image, and performing denoising processing on the noise image based on the second style information to obtain a style migration image. In this way, the influence of the content information of the style image and the style information of the to-be-processed image on the style migration image can be reduced to a great extent, data and time costs brought by additional model training can be avoided, and then the processing efficiency of style migration can be improved to a great extent.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image style transfer method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of generative artificial intelligence technology, style transfer is increasingly used in the field of images. Style transfer refers to the process of introducing the style information of a style image into the content image given a content image and a style image, so that the content image can also show the style characteristics of the style image. Currently, style transfer has been widely used in film, television, animation, games and other fields.

[0003] In related technologies, style transfer is generally based on diffusion models to generate images with high artistic quality and consistent style. However, diffusion models usually rely on additional model training, which makes related technologies require high resources such as data and time. In the case of limited resources, the processing efficiency of style transfer in related technologies is low. Summary of the invention

[0004] The present disclosure provides an image style transfer method, apparatus, device, storage medium and program product.

[0005] According to a first aspect of the present disclosure, a method for image style transfer is provided, the method comprising:

[0006] Get the image to be processed and the style image;

[0007] Extracting first style information of the image to be processed and content information of the style image, and generating guidance information based on the first style information and the content information;

[0008] Performing noise processing on the image to be processed, and negatively guiding the noise processing by using the guiding information to obtain a noise image;

[0009] Second style information of the style image is extracted, and denoising is performed on the noisy image based on the second style information to obtain a style transfer image.

[0010] According to a second aspect of the present disclosure, there is provided an image style transfer apparatus, the apparatus comprising:

[0011] An image acquisition module, used to acquire the image to be processed and the style image;

[0012] a guide information generating module, configured to extract first style information of the image to be processed and content information of the style image, and generate guide information based on the first style information and the content information;

[0013] A noise image generating module, used for performing noise processing on the image to be processed, and negatively guiding the noise processing through the guiding information, so as to obtain a noise image;

[0014] The style transfer image acquisition module is used to extract second style information of the style image, and perform denoising on the noisy image based on the second style information to obtain a style transfer image.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0017] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements the above method of the present disclosure when executed by a processor.

[0018] The image style transfer method, apparatus, device, storage medium and program product provided by the embodiments of the present disclosure obtain the image to be processed and the style image, extract the first style information of the image to be processed and the content information of the style image, and generate guidance information based on the first style information and the content information. The image to be processed is subjected to noise processing, and the noise processing is negatively guided by the guidance information to obtain a noise image. The second style information of the style image is extracted, and the noise image is denoised based on the second style information to obtain a style transfer image. In the embodiment, the generated noise image is gradually moved away from the content information of the style image and the style information of the content image by the guidance information during the noise adding process, which can greatly reduce the influence of the content information of the style image and the style information of the image to be processed on the style transfer image, and can also avoid the data and time costs caused by additional model training, thereby greatly improving the processing efficiency of style transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram of a style transfer process provided by an exemplary embodiment of the present disclosure;

[0021] Figure 2A A content image provided for an exemplary embodiment of the present disclosure;

[0022] Figure 2B A style image provided for an exemplary embodiment of the present disclosure;

[0023] Figure 2C A result image generated by style transfer provided by an exemplary embodiment of the present disclosure;

[0024] Figure 3 A flowchart of an image style transfer method provided by an exemplary embodiment of the present disclosure;

[0025] Figure 4 A schematic block diagram of functional modules of an image style transfer apparatus provided by an exemplary embodiment of the present disclosure;

[0026] Figure 5 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0027] Figure 6 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0029] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0030] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0031] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0032] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0033] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0034] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0035] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0036] With the rapid development of generative artificial intelligence technology, style transfer has been widely used in the field of images, but it also brings many challenges. Style transfer is an innovative image processing technology that integrates the visual style of a reference image into the content image to generate an image that has both the semantics of the content image and the style characteristics of the target image. In recent years, many style transfer methods have been based on diffusion models and have achieved remarkable results in generating images with high artistic quality and consistent style. However, related technologies usually rely on additional model training, which places high demands on time cost and data acquisition. In scenarios where computing resources are limited or fast response is required, the processing efficiency of related technologies is low.

[0037] Therefore, in order to solve the above technical problems, the embodiments of the present disclosure provide a style transfer method that does not require additional training and can avoid the influence of content information in the style image on the style transfer, thereby improving the processing efficiency and processing effect of the style transfer.

[0038] Specifically, Figure 1 A schematic diagram of the processing of style transfer provided by an embodiment of the present disclosure, such as Figure 1 As shown, a content image provided by a user is obtained. The content image is an image that needs to be processed and may also be referred to as an image to be processed.

[0039] In the denoising stage, the content image is taken as input and converted into its noise latent space representation (i.e. the inverted image) through the inversion process of the Denoising Diffusion Implicit Models (DDMI). This allows the content image to be embedded into the latent space of the diffusion model while retaining the content information of the content image.

[0040] In the inversion stage, additional guidance information can be introduced. The style information of the content image and the content information of the style image can be extracted using IP-Adapter-Instruct, which can be used as negative guidance to adjust the inversion process to control the potential representation to stay away from the guidance information in the feature space.

[0041] It should be noted that IPAdapter-Instruct is a model that adds "Instruct prompts" based on IP-Adapter. It can achieve multiple interpretations of the same conditional image by combining natural image conditions and instruction prompts, thereby flexibly switching between different tasks, such as style transfer and object extraction.

[0042] Specifically, in the denoising stage, the content image can be preprocessed by the inversion technology of the denoising diffusion implicit model, and the category-free guidance size parameter of the inversion of the denoising diffusion implicit model is set to ω i , the input high-resolution content image I with dimension W*H c Mapped to the noise space of the denoising diffusion implicit model, the feature vector in the final noise space is expressed as Its dimension is N*C1*C2.

[0043] By adding noise to the content image in the noise adding stage, noise is gradually added to the content image during the noise adding process, and a noise image that is semantically consistent with the input content image is generated as the basis for subsequent operations. And by extracting the content information of the style image and the style information of the content image, guidance information is generated. The noise adding process is negatively guided by the guidance information, so that the noise image after noise adding gradually moves away from the content information of the style image and the style information of the content image, which is convenient for obtaining the result image consistent with the style of the style image in the subsequent process, and the content information of the content image can also be retained. Among them, N, C1, and C2 are all hyperparameters, representing the feature dimensions respectively; W and H are hyperparameters, representing the length dimension and width dimension of the image. The specific process of noise adding is as follows:

[0044] 1) Initial processing.

[0045] (1) The style image I s Input the pre-trained IPAdapter-Instruct model and use the content instruction prompt P c , get the content feature vector of the style image At the same time, the content image I c Input the pre-trained IPAdapter-Instruct model and use the style instruction prompt P s , get the style feature vector of the content image The content feature vector of the style image and the style feature vector of the content image are concatenated to obtain the overall control feature vector ξ - .Right now:

[0046]

[0047] Among them, concat means concatenating two feature vectors along the last dimension.

[0048] (2) Given the noise scheduling sequence in the diffusion process T i is the total time step of adding noise, t = 0 indicates the complete denoising state, and the corresponding eigenvector is By directly converting the content image I c Input to the image encoder to get. t = T i When , it indicates a completely noisy state, and the corresponding eigenvector is and will as input to the denoising stage.

[0049] 2) Prediction of noise.

[0050] The noise component at time step t under the control of condition c can be calculated by the prediction head of the denoising diffusion implicit model. - , guiding the inversion process away from the content information of the style image, and away from the style information of the content image. That is:

[0051]

[0052] in, represents the noise component, is a noise-added sample, represents the prediction noise of the diffusion model under condition c, t=0…,t-1,t,…,T i ω i and ε θ are weights, which can be set as needed and can be empirical values. The diffusion model is a generative artificial intelligence model that can be used to generate high-quality images, audio, text or other data types.

[0053] 3) Inversion update.

[0054] In an embodiment, the noise samples in the same time step can be The noise component in the same time step Combined, we get the noise sample for the next time step Right now:

[0055]

[0056] 4) Iteration process.

[0057] In the embodiment provided by the present disclosure, it is possible to Start by repeating steps 2) and 3) above, following the inversion order 0→T i Update samples sequentially until t = T i ,get

[0058] Based on the above embodiment, the image feature vector after adding noise is obtained in the noise adding stage. After that, we can use the denoising diffusion implicit model to The denoising process can be performed step by step. It represents the noise feature vector obtained by each denoising step, and injects the style information of the style image into the denoising process to obtain the final result image. The result image has the style features of the style image and retains the content information of the content image.

[0059] In the embodiment, the pre-trained denoising diffusion implicit model can be used to perform T-step denoising to complete the style transfer process. The style strength injection parameter, the ControlNet injection strength parameter, and the denoising unclassified guidance scale parameter are defined as ω s ,ω CN ,ω. Specifically, the following steps may be included:

[0060] (1) The style image I s Input the image encoder, which can be a pre-trained CLIP (Contrastive Language-Image Pre-training, multi-modal pre-training) model, to obtain the style image I s The style feature vector z s .

[0061] (2) The style feature vector z s The image cross-attention module trained by the IPAdapter-Instruct model is injected into the first module of the downsampling of U-Net with an injection intensity of ω s .

[0062] (3) Content image I c Inject the ControlNet model as an additional control module with an injection intensity of ω CN .

[0063] It should be noted that the ControlNet model is an auxiliary neural network model structure, which is mainly used to introduce additional conditional controls in the Stable Diffusion model to more finely control the image generation process. ControlNet supports more input conditions to control the image generation process by adding auxiliary modules to the Stable Diffusion model.

[0064] (4) Generate description text of content images using the BLIP2 model As a text editor controlled by the CLIP module E t Injected into U-Net. Among them, the BLIP2 model is a multimodal model, mainly used for pre-training tasks of visual and language models. BLIP2 can reduce training costs and achieve efficient visual language learning through frozen image encoders and large language models.

[0065] (5) After completing the above condition injection, the noise prediction vector generated by U-Net can be used right Perform gradual denoising to obtain the image feature vector after style transfer in the latent space After the VAE decoder module (i.e. Decoder), the image I after style transfer is obtained cs .Right now:

[0066]

[0067] Where, t = T,…,t,t-1,…,0; is the scheduling sequence of the noise in the diffusion process, α t is positive and monotonically decreasing. For other parameters, refer to the above noise adding process. The image is denoised in the t-1th step. After T steps of denoising, the image feature vector after style transfer in the latent space can be obtained.

[0068] Based on the above embodiment, when a user inputs a content image and a style image, after the processing in the above embodiment, an image processed by style transfer is generated in the embodiment. Figure 2A , Figure 2B and Figure 2C As shown, Figure 2A The content image provided by the embodiment of the present disclosure, Figure 2B The style image provided by the embodiment of the present disclosure, Figure 2C The result image is generated by style transfer in the embodiment of the present disclosure. The result image has a style similar to the style image, while being able to avoid contamination of the result image by the image content in the style image.

[0069] The hyperparameter settings of the embodiment can be as follows: image size W = 1024, H = 1024, feature vector dimension N = 4, C1 = 128, C2 = 128, inversion of the non-classified guidance scale parameter ω i =1.5, denoising without category guidance scale parameter ω=3.5, style injection strength parameter ω s =2.5, ControlNet injection strength parameter ω CN =0.25. Noise adding step number T i =50, and the number of denoising steps T = 50. The embodiment may use the Stable Diffusion XL pre-trained model, and the image encoder before style image injection uses the ViT-L / 14 (Vision Transformer Large model with 14patches) model.

[0070] It should be noted that Stable Diffusion XL is a two-stage cascade diffusion model consisting of a Base model and a Refiner model. The Base model has functions such as text-generated images, image-generated images, and image inpainting, while the Refiner model further refines the image Latent features generated by the Base model to improve the quality and details of the generated image. In addition, the ViT-L / 14 model is a model based on the Vision Transformer (ViT), which is a variant of the Transformer architecture. The model uses a large Transformer architecture and divides the input image into 14 regions for processing, which can maintain high efficiency and accuracy when processing large-scale images.

[0071] The disclosed embodiment can avoid the additional cost of training time and data collection without the need for additional model training. In addition, in the inversion stage of style transfer, the guidance information is used for negative guidance. By guiding the initial point away from the content information of the style image in the noise space during the inversion process, the pollution of the content information of the style image on the generated result can be effectively alleviated, and then an image that is not affected by the content information of the style image and the style information of the content image can be obtained, so that the image obtained by style transfer is more thorough.

[0072] Based on the above embodiments, in another embodiment provided by the present disclosure, the present disclosure also provides an image style transfer method, such as Figure 3 As shown, the method may include the following steps:

[0073] In step S310, the image to be processed and the style image are obtained.

[0074] In the embodiment, the image to be processed may specifically be the content image in the above embodiment. By performing style transfer processing on the image to be processed and the style image, the image to be processed has the style features of the style image.

[0075] In step S320, first style information of the image to be processed and content information of the style image are extracted, and guidance information is generated based on the first style information and the content information.

[0076] In an embodiment, the first style information of the image to be processed and the content information of the style image may be extracted through the trained related model.

[0077] Specifically, the image to be processed and the style image can be input into the trained target instruction prompt model respectively, and the style feature vector of the image to be processed can be extracted through the trained target instruction prompt model based on the style instruction prompt, and the first style information can be obtained based on the style feature vector. Based on the content instruction prompt, the content feature vector of the style image can be extracted through the trained target instruction prompt model, and the content information can be obtained based on the content feature vector. The trained target instruction prompt model is used to extract the feature vector of the image based on the instruction prompt.

[0078] For example, the style image I s Input the pre-trained IPAdapter-Instruct model, use the content instruction prompt to obtain the content feature vector of the style image. At the same time, input the style image into the pre-trained IPAdapter-Instruct model, use the style instruction prompt to obtain the style feature vector of the image to be processed. In this way, the content feature vector of the style image can be used as the content information of the style image, and the style feature vector of the image to be processed can be used as the first style information of the image to be processed.

[0079] In step S330, the image to be processed is subjected to noise addition processing, and the noise addition processing is negatively guided by the guidance information to obtain a noise image.

[0080] In an embodiment, the image to be processed can be gradually denoised by inverting the pre-trained denoising diffusion implicit model, and the denoising process can be negatively guided by the guidance information during the denoising process, so that the generated noise image is far away from the content information of the style image and the first style information of the image to be processed. The content feature vector of the style image and the style feature vector of the content image in the above embodiment can be concatenated to obtain an overall control feature vector, and the overall control feature vector is used as the guidance information.

[0081] In step S340, second style information of the style image is extracted, and denoising is performed on the noisy image based on the second style information to obtain a style transfer image.

[0082] In an embodiment, by extracting the second style information of the style image and performing multi-step denoising on the noisy image, during the denoising process, the second style information of the style image can be gradually injected into the denoising image, and finally a result image after style transfer is obtained, which is the style transfer image.

[0083] The image style transfer method provided by the embodiment of the present disclosure obtains the image to be processed and the style image, extracts the first style information of the image to be processed and the content information of the style image, and generates guidance information based on the first style information and the content information. The image to be processed is subjected to noise processing, and the noise processing is negatively guided by the guidance information to obtain a noise image. The second style information of the style image is extracted, and the noise image is denoised based on the second style information to obtain a style transfer image. In the embodiment, the generated noise image is gradually moved away from the content information of the style image and the style information of the content image by the guidance information during the noise adding process, which can greatly reduce the content pollution of the style image and the style information of the image to be processed on the style transfer image, and can also avoid the data and time costs brought by additional model training, thereby greatly improving the processing efficiency of style transfer.

[0084] Based on the above embodiment, in another embodiment provided by the present disclosure, the above step S320 may further include the following steps:

[0085] In step S321, the number of noise adding time steps is set, and the number of noise adding time steps includes a plurality of noise adding time points.

[0086] In step S322, based on the number of denoising time steps, the image to be processed is denoised at each denoising time point using a pre-trained denoising diffusion implicit model.

[0087] In the embodiment, the number of noise adding time steps T can be set i For example, you can set T i =50. This can set T i Noise adding time points, so that the image to be processed can be denoised at each noise adding time point by using the pre-trained denoising diffusion implicit model inversion (DDMIInversion), and the noise adding process is negatively guided by the guiding information to obtain a noisy image.

[0088] Based on the above embodiment, in the embodiment provided by the present disclosure, when the noise addition process is negatively guided by the guidance information, the above step S330 may further specifically include the following steps:

[0089] In step S331, the prediction noise is obtained by pre-training the prediction head of the denoising diffusion implicit model.

[0090] In step S332, a noise component is generated based on the predicted noise and the guidance information.

[0091] In an embodiment, the prediction noise at each noise adding time point can be obtained through the prediction head of the denoising diffusion implicit model, and based on the guidance information, the prediction noise at each noise adding time point is eliminated to perform feature elimination processing to obtain the noise components corresponding to each noise adding time point. The feature elimination processing is used to eliminate at least a part of the features of the guidance information contained in the prediction noise.

[0092] Specifically, referring to the above formula (2), the noise component at time step t under the control of condition c can be calculated by the prediction head of the denoising diffusion implicit model. - , guiding the inversion process away from the content information of the style image and away from the style information of the content image.

[0093] In step S333, the noisy sample obtained at the t-th noisy time point is acquired, and the noisy sample is subjected to noise processing by the noise component at the t+1-th noisy time point to obtain a noisy image.

[0094] Where 0≤t≤T i , at the 0th noise adding time point, the noise adding sample is the image to be processed; at the T i The noise adding time point, the noise adding sample is the noise image, t and T i are positive integers respectively.

[0095] In an embodiment, during the noise addition process, guidance information is used for negative guidance, so that the gradually obtained noisy image can be away from the first style information of the image to be processed and the content information of the style image in the guidance information. The noise image obtained in this way can remove at least a part of the first style information of the image to be processed and the content information of the style image, which can greatly reduce the content contamination of the style transfer image by the content information of the style image and the style information of the image to be processed.

[0096] In the embodiment provided by the present disclosure, in the process of performing denoising on the noisy image based on the second style information to obtain the style transfer image, the denoising time step number can be set, and the denoising time step number is consistent with the denoising time step number. A noise prediction vector is generated based on the denoised sample, the second style information and the denoising time step number. In this way, the noisy image can be gradually denoised according to the denoising time step number based on the noise prediction vector to obtain the style transfer image.

[0097] In an embodiment, the noise prediction vector generated by U-Net can be used to gradually denoise the image feature vector after adding noise, obtain the image feature vector after style transfer in the latent space, and obtain the image after style transfer through the VAE (Variational Auto Encoder) decoder module (i.e., Decoder). Among them, the VAE decoder is a key component in the VAE model, and its main function is to sample and generate new data from the latent space. VAE consists of two parts: an encoder and a decoder. The encoder is responsible for mapping the input data (such as an image) to a probability distribution of a latent space, while the decoder generates new data from the points sampled in these latent spaces.

[0098] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides an image style migration device, which may be a server, a terminal, or a chip applied to a server. Figure 4 This is a schematic block diagram of the functional modules of an image style transfer device provided by an exemplary embodiment of the present disclosure. Figure 4 As shown, the image style transfer device includes:

[0099] An image acquisition module 10 is used to acquire an image to be processed and a style image;

[0100] A guide information generating module 20, configured to extract first style information of the image to be processed and content information of the style image, and generate guide information based on the first style information and the content information;

[0101] A noise image generating module 30, configured to perform noise processing on the image to be processed, and negatively guide the noise processing through the guiding information, so as to obtain a noise image;

[0102] The style transfer image obtaining module 40 is used to extract second style information of the style image, and perform denoising on the noisy image based on the second style information to obtain a style transfer image.

[0103] In another embodiment provided by the present disclosure, the guidance information generating module is specifically used to:

[0104] Inputting the image to be processed and the style image into the trained target instruction prompt model respectively; wherein the trained target instruction prompt model is used to extract the feature vector of the image based on the instruction prompt;

[0105] Extracting a style feature vector of the image to be processed through the trained target instruction prompt model based on the style instruction prompt, and obtaining the first style information based on the style feature vector;

[0106] The content feature vector of the style image is extracted based on the content instruction prompt through the trained target instruction prompt model, and the content information is obtained based on the content feature vector.

[0107] In another embodiment provided by the present disclosure, the noise image generating module is specifically used to:

[0108] Setting a noise adding time step number, wherein the noise adding time step number includes a plurality of noise adding time points;

[0109] Based on the number of noise adding time steps, the image to be processed is subjected to noise adding processing at each noise adding time point respectively through a pre-trained denoising diffusion implicit model.

[0110] In another embodiment provided by the present disclosure, the noise image generating module is specifically used to:

[0111] Obtaining prediction noise through the prediction head of the pre-trained denoising diffusion implicit model;

[0112] generating a noise component based on the predicted noise and the guidance information;

[0113] Obtain the noisy sample obtained at the t-th noisy time point, and perform a noisy process on the noisy sample using the noise component at the t+1-th noisy time point to obtain a noisy image; wherein 0≤t≤T i , at the 0th noise adding time point, the noise adding sample is the image to be processed; at the Tth i Noise adding time point, the noise adding sample is the noise image, t and T i are positive integers respectively.

[0114] In another embodiment provided by the present disclosure, the noise image generating module is further configured to:

[0115] Obtaining predicted noise at each noise addition time point through the prediction head of the denoising diffusion implicit model;

[0116] Based on the guidance information, feature elimination processing is performed on the predicted noise at each noise-added time point to obtain noise components corresponding to each noise-added time point; wherein the feature elimination processing is used to eliminate at least part of the features of the guidance information contained in the predicted noise.

[0117] In another embodiment provided by the present disclosure, the style transfer image acquisition module is specifically used to:

[0118] Set the denoising time step, the denoising time step is consistent with the denoising time step;

[0119] Generate a noise prediction vector based on the noisy sample, the second style information and the denoising time step;

[0120] Based on the noise prediction vector, the noisy image is subjected to step-by-step denoising processing according to the denoising time steps to obtain a style transfer image.

[0121] The image style transfer device provided by the embodiment of the present disclosure obtains the image to be processed and the style image, extracts the first style information of the image to be processed and the content information of the style image, and generates guidance information based on the first style information and the content information. The image to be processed is subjected to noise processing, and the noise processing is negatively guided by the guidance information to obtain a noise image. The second style information of the style image is extracted, and the noise image is denoised based on the second style information to obtain a style transfer image. In the embodiment, the generated noise image is gradually moved away from the content information of the style image and the style information of the content image by the guidance information during the noise adding process, which can greatly reduce the content pollution of the style image and the style information of the image to be processed on the style transfer image, and can also avoid the data and time costs brought by additional model training, thereby greatly improving the processing efficiency of style transfer.

[0122] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0123] Figure 5 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801 , and the processor 1801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0124] The processor 1801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1801 or by instructions in the form of software. The processor 1801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and completes the steps of the method in combination with its hardware.

[0125] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 6 The computer system 1900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 6 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.

[0126] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0127] like Figure 6As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 to a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0128] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including: an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 may be any type of device capable of inputting information to the computer system 1900, and the input unit 1906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 may include, but is not limited to, a disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0129] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0130] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0131] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0133] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the above method disclosed in the embodiments of the present disclosure is implemented.

[0134] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0135] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0137] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0138] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0139] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for image style transfer, characterized in that: The method comprises: Get the image to be processed and the style image; Extracting first style information of the image to be processed and content information of the style image, and generating guidance information based on the first style information and the content information; Performing noise processing on the image to be processed, and negatively guiding the noise processing by using the guiding information to obtain a noise image; Second style information of the style image is extracted, and denoising is performed on the noisy image based on the second style information to obtain a style transfer image.

2. The method according to claim 1, characterized in that The extracting the first style information of the image to be processed and the content information of the style image includes: Inputting the image to be processed and the style image into a trained target instruction prompt model respectively; wherein the trained target instruction prompt model is used to extract a feature vector of the image based on the instruction prompt; Extracting a style feature vector of the image to be processed through the trained target instruction prompt model based on the style instruction prompt, and obtaining the first style information based on the style feature vector; The content feature vector of the style image is extracted based on the content instruction prompt through the trained target instruction prompt model, and the content information is obtained based on the content feature vector.

3. The method according to claim 1, characterized in that The step of performing noise adding processing on the image to be processed comprises: Setting a noise adding time step number, wherein the noise adding time step number includes a plurality of noise adding time points; Based on the number of noise adding time steps, the image to be processed is subjected to noise adding processing at each noise adding time point respectively through a pre-trained denoising diffusion implicit model.

4. The method according to claim 3, characterized in that The step of negatively guiding the noise addition process by using the guide information to obtain a noise image includes: Obtaining prediction noise through the prediction head of the pre-trained denoising diffusion implicit model; generating a noise component based on the predicted noise and the guidance information; Obtain the noisy sample obtained at the t-th noisy time point, and perform a noisy process on the noisy sample using the noise component at the t+1-th noisy time point to obtain a noisy image; wherein 0≤t≤T i , at the 0th noise adding time point, the noise adding sample is the image to be processed; at the Tth i Noise adding time point, the noise adding sample is the noise image, t and T i are positive integers respectively.

5. The method according to claim 4, characterized in that The generating a noise component based on the predicted noise and the guidance information comprises: Obtaining predicted noise at each noise addition time point through the prediction head of the denoising diffusion implicit model; Based on the guidance information, feature elimination processing is performed on the predicted noise at each noise-added time point to obtain noise components corresponding to each noise-added time point; wherein the feature elimination processing is used to eliminate at least part of the features of the guidance information contained in the predicted noise.

6. The method according to claim 4, characterized in that The denoising process is performed on the noisy image based on the second style information to obtain a style-transferred image, including: Set the denoising time step, the denoising time step is consistent with the denoising time step; Generate a noise prediction vector based on the noisy sample, the second style information and the denoising time step; Based on the noise prediction vector, the noisy image is subjected to step-by-step denoising processing according to the denoising time steps to obtain a style transfer image.

7. An image style transfer device, characterized in that: The device comprises: An image acquisition module, used to acquire the image to be processed and the style image; a guide information generating module, configured to extract first style information of the image to be processed and content information of the style image, and generate guide information based on the first style information and the content information; A noise image generating module, used for performing noise processing on the image to be processed, and negatively guiding the noise processing through the guiding information, so as to obtain a noise image; The style transfer image acquisition module is used to extract second style information of the style image, and perform denoising on the noisy image based on the second style information to obtain a style transfer image.

8. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images