Image style migration method and device, equipment, storage medium and program product

Abstract

The invention relates to an image style migration method and apparatus, a device, a storage medium and a program product. The method comprises the steps of obtaining a to-be-processed image and a style image; performing noise addition processing on the to-be-processed image to obtain a first noise image; removing at least one part of low-frequency components in the first noise image to obtain a second noise image; and extracting style information of the style image, and carrying out de-noising processing on the second noise image based on the style information to obtain a style migration image. According to the embodiment of the invention, the high-frequency component part of the image can be highlighted by removing at least one part of the low-frequency component in the first noise image, so that the obtained style migration image can improve the fidelity of the image content while having the style characteristics of the style image. And data and time costs caused by extra model training can be avoided, so that the processing efficiency of style migration can be improved to a great extent.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, and in particular to an image style transfer method, device, equipment, storage medium and program product. Background Art

[0002] With the rapid development of generative artificial intelligence technology, style transfer is increasingly used in the field of images. Style transfer refers to the process of introducing the style information of a style image into the content image given a content image and a style image, so that the content image can also show the style characteristics of the style image. Currently, style transfer has been widely used in film, television, animation, games and other fields.

[0003] In related technologies, style transfer is generally based on diffusion models to generate images with high artistic quality and consistent style. However, diffusion models usually rely on additional model training, which makes related technologies require high resources such as data and time. In the case of limited resources, the processing efficiency of style transfer in related technologies is low. Summary of the invention

[0004] The present disclosure provides an image style transfer method, apparatus, device, storage medium and program product.

[0005] According to a first aspect of the present disclosure, a method for image style transfer is provided, the method comprising:

[0006] Get the image to be processed and the style image;

[0007] Performing noise processing on the image to be processed to obtain a first noise image;

[0008] removing at least a portion of the low-frequency components in the first noise image to obtain a second noise image;

[0009] Style information of the style image is extracted, and denoising is performed on the second noisy image based on the style information to obtain a style transferred image.

[0010] According to a second aspect of the present disclosure, there is provided an image style transfer apparatus, the apparatus comprising:

[0011] An image acquisition module, used to acquire the image to be processed and the style image;

[0012] A noise adding module, used for performing noise adding processing on the image to be processed to obtain a first noise image;

[0013] A low-frequency processing module, used to remove at least a part of the low-frequency components in the first noise image to obtain a second noise image;

[0014] The style transfer module is used to extract style information of the style image and perform denoising on the second noisy image based on the style information to obtain a style transferred image.

[0015] According to a third aspect of the present disclosure, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0016] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the above method of the present disclosure is implemented.

[0017] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method of the present disclosure when executed by a processor.

[0018] The image style transfer method, apparatus, device, storage medium and program product provided by the embodiments of the present disclosure obtain a first noise image by acquiring an image to be processed and a style image, and performing noise processing on the image to be processed. A second noise image can be obtained by removing at least a portion of the low-frequency component in the first noise image. In this way, the style information of the style image is extracted, and the second noise image is denoised based on the style information to obtain a style transfer image. The embodiment can highlight the high-frequency component of the image by removing at least a portion of the low-frequency component in the first noise image. The style transfer image obtained in this way can not only have the style features of the style image, but also improve the fidelity of the image content. It can also avoid the data and time costs caused by additional model training, thereby greatly improving the processing efficiency of style transfer. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A schematic diagram of a style transfer process provided by an exemplary embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of frequency domain operation on a content image provided by an exemplary embodiment of the present disclosure;

[0022] Figure 3A A content image provided for an exemplary embodiment of the present disclosure;

[0023] Figure 3B A style image provided for an exemplary embodiment of the present disclosure;

[0024] Figure 3CA style transfer image generated for an exemplary embodiment of the present disclosure;

[0025] Figure 3D Style images generated in related technologies;

[0026] Figure 4 A flowchart of an image style transfer method provided by an exemplary embodiment of the present disclosure;

[0027] Figure 5 A schematic block diagram of functional modules of an image style transfer apparatus provided by an exemplary embodiment of the present disclosure;

[0028] Figure 6 A structural block diagram of an electronic device provided for an exemplary embodiment of the present disclosure;

[0029] Figure 7 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0030] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0031] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0032] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0033] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0034] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0035] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0036] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0037] As an optional but non-limiting implementation, in response to receiving an active request from the user, the method of sending a prompt message to the user may be, for example, a pop-up window, in which the prompt message may be presented in text. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device. It is understandable that the above notification and the process of obtaining user authorization are only illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0038] With the rapid development of generative artificial intelligence technology, style transfer has been widely used in the field of images, but it also brings many challenges. Style transfer is an innovative image processing technology that integrates the visual style of a reference image into the content image to generate an image that has both the semantics of the content image and the style characteristics of the target image. In recent years, many style transfer methods have been based on diffusion models and have achieved remarkable results in generating images with high artistic quality and consistent style. However, related technologies usually rely on additional model training, which places high demands on time cost and data acquisition. In scenarios where computing resources are limited or fast response is required, the processing efficiency of related technologies is low.

[0039] Therefore, in order to solve the above technical problems, the embodiments of the present disclosure provide a style transfer method that does not require additional training and can avoid the influence of content information in the style image on the style transfer, thereby improving the processing efficiency and processing effect of the style transfer.

[0040] In the embodiments provided by the present disclosure, the inversion process of the Denoising Diffusion Implicit Models (DDMI) is used in the denoising stage to accurately map the features of the content image to the noise space with the help of the pre-trained diffusion model, thereby providing an accurate initial point containing content information for the denoising stage in the Denoising Diffusion Implicit Model. In addition, the style information of the style image is injected in the denoising stage, and the style information is injected only in the style-specific network module through cross-attention, which can greatly improve the effect of style transfer.

[0041] In the embodiment, by performing additional content control based on the ControlNet model in the denoising stage, the spatial structure can be maintained and the content preservation effect can be improved. In addition, in order to improve the style transfer result's ability to retain image layout, contours, and details, the embodiment enhances the features of the initial points obtained after the noise processing without training, which can strengthen the model's attention to the image layout in the subsequent denoising stage, and can effectively improve the content preservation effect of the style transfer image.

[0042] Specifically, Figure 1 As shown, Figure 1 A schematic diagram of the style transfer process provided by an embodiment of the present disclosure is provided. A content image provided by a user is obtained. The content image is an image to be processed, and may also be referred to as an image to be processed.

[0043] In the denoising stage, the content image is taken as input and converted into its noise latent space representation (i.e., the inverted image) through the inversion process of the denoising diffusion implicit model to obtain the noise image. This allows the content image to be embedded into the latent space of the diffusion model while retaining the content information of the content communication.

[0044] Based on the noise image obtained after inversion, a certain low-frequency component of the image can be filtered out through frequency filtering operation, so that the network pays more attention to the high-frequency component of the image. Since the high-frequency component of the image represents the contour details of the image, this can improve the content preservation effect of the image after image style transfer.

[0045] Specifically, in the denoising stage, the content image can be preprocessed by the inversion technology of the denoising diffusion implicit model, and the category-free guidance size parameter of the inversion of the denoising diffusion implicit model is set to ω i , the input high-resolution content image I with dimension W*H c Mapped to the noise space of the denoising diffusion implicit model, the feature vector in the final noise space is expressed as Its dimension is N*C1*C2.

[0046] By adding noise to the content image in the noise adding stage, noise is gradually added to the content image during the noise adding process to generate a noise image that is semantically consistent with the input content image, which serves as the basis for subsequent frequency domain operations. Among them, N, C1, and C2 are all hyperparameters, representing the feature dimensions respectively; W and H are both hyperparameters, representing the length dimension and width dimension of the image. The specific process of noise adding is as follows:

[0047] In an embodiment, the content image I can be transformed into c Encoded as a feature vector in feature space Among them, the VAE image encoder is mainly responsible for compressing the input image data into a low-dimensional latent space representation. VAE is a generative model whose core idea is to generate new data samples by learning the potential probability distribution of data. VAE combines the ideas of autoencoder and variational Bayesian inference and can be used for image generation.

[0048] In the embodiment, the pre-trained denoising diffusion implicit model can be used to iteratively transfer the content image I c The eigenvector of Add noise ε θ (z t ,t), the noise is predicted by the pre-trained denoising diffusion implicit model, and the expression of gradually adding noise is as follows:

[0049]

[0050] Where t=0,…,T i , is the scheduling sequence of the noise in the diffusion process, α t is positive and monotonically decreasing.

[0051] This will give us a noise image that contains the noise feature vector. Used for subsequent frequency domain operations.

[0052] In the embodiment provided in the present disclosure, in the frequency domain operation stage, such as Figure 2 As shown, the eigenvector in the generated noise space is Then, for the feature vector Perform frequency domain decomposition and filtering operation (Frequency Manipulation, referred to as FM), which may specifically include the following steps:

[0053] (1) Fourier transform (FT).

[0054] Will After FT conversion to the frequency domain, the spectrum f is obtained c .

[0055] (2) Frequency band separation.

[0056] Using Gaussian high-pass filter H r and Gaussian low-pass filter L r The spectrum f c Decompose into high frequency components and low frequency components Gaussian high pass filter H r and Gaussian low-pass filter L r is defined as follows:

[0057]

[0058] in, ⊙ represents the element-wise multiplication operator, and σ represents the Gaussian function variance.

[0059] (3) Spectrum enhancement.

[0060] In an embodiment, the high frequency component can be and low frequency components Add them proportionally to get the frequency domain feature vector with reduced low-frequency components The details are as follows:

[0061]

[0062] Among them, α is a hyperparameter, 0<α<1, and α represents the weighted ratio.

[0063] (4) Inverse Fourier Transform (IFT).

[0064] The frequency domain feature vector that reduces the low-frequency components Convert back to image space and get the feature vector with reduced low-frequency components Generate a N*C*C dimensional random matrix X~N(0,1) that obeys a normal distribution, and compare X with Weighted fusion to obtain the feature vector z after the final spectrum operation c， ′, and used as the input for the subsequent denoising process. That is:

[0065]

[0066] In the denoising stage, z can be denoised based on the denoising diffusion implicit model. c， 'Step by step denoising can be performed by Represents the noise feature vector obtained by each denoising step, and injects the style information of the style image into the denoising process to obtain the final result image. The result image has the style features of the style image and can enhance the details and contour information in the content image.

[0067] In the embodiment, the pre-trained denoising diffusion implicit model can be used to perform T-step denoising to complete the style transfer process. The style strength injection parameter, the ControlNet injection strength parameter, and the denoising unclassified guidance scale parameter are defined as ω s ,ω CN ,ω. Specifically, the following steps may be included:

[0068] (1) The style image I s Input the image encoder, which can be a pre-trained CLIP (Contrastive Language-Image Pre-training, multi-modal pre-training) model, to obtain the style image I s The style feature vector z s .

[0069] (2) The style feature vector z s The image cross-attention module trained by the IPAdapter-Instruct model is injected into the first module of the downsampling of U-Net with an injection intensity of ω s .

[0070] (3) Content image I c Inject the ControlNet model as an additional control module with an injection intensity of ω CN .

[0071] It should be noted that the ControlNet model is an auxiliary neural network model structure, which is mainly used to introduce additional conditional controls in the Stable Diffusion model to more finely control the image generation process. ControlNet supports more input conditions to control the image generation process by adding auxiliary modules to the Stable Diffusion model.

[0072] (4) Generate description text of content images using the BLIP2 model As a text editor controlled by the CLIP module E t Injected into U-Net. Among them, the BLIP2 model is a multimodal model, mainly used for pre-training tasks of visual and language models. BLIP2 can reduce training costs and achieve efficient visual language learning through frozen image encoders and large language models.

[0073] (5) After completing the above condition injection, the noise prediction vector generated by U-Net can be used right Perform gradual denoising to obtain the image feature vector after style transfer in the latent space After the CAE decoder module (i.e. Decoder), the image I after style transfer is obtained cs .Right now:

[0074]

[0075] Where, t = T,…,t,t-1,…,0; is the scheduling sequence of the noise in the diffusion process, α t is positive and monotonically decreasing. The image is denoised in the t-1th step. After T steps of denoising, the image feature vector after style transfer in the latent space can be obtained.

[0076] Based on the above embodiment, when a user inputs a content image and a style image, after the processing in the above embodiment, a style-transferred image is generated in the embodiment, which not only maintains the entire content of the content image, but also has the style features of the style image. Figure 3A , Figure 3B , Figure 3C and Figure 3D As shown, Figure 3A The content image provided by the embodiment of the present disclosure, Figure 3B The style image provided by the embodiment of the present disclosure, Figure 3C The style transfer image generated by the embodiment of the present disclosure, Figure 3D Style images generated in related techniques.

[0077] By comparison Figure 3C and Figure 3D It can be seen that the style transfer image generated by the embodiment of the present disclosure has the style features of the style image while maintaining all the content of the content image through image style transfer processing. However, the style transfer image obtained by the related art after style transfer processing will lack part of the content image.

[0078] The hyperparameter settings of the embodiment can be as follows: image size W = 1024, H = 1024, feature vector dimension N = 4, C1 = 128, C2 = 128, Gaussian function variance σ = 0.3, weighting ratio α = 0.7, inversion non-category guidance scale parameter ω i =1.0, denoising without category guidance scale parameter ω=5.0, style injection strength parameter ω s =2.5, ControlNet injection strength parameter ωCN =0.25. Noise adding step number T i =50, and the number of denoising steps T = 50.

[0079] The embodiment can use the Stable Diffusion XL pre-trained model, and the image encoder before the style image injection uses the ViT-L / 14 (Vision Transformer Large model with 14patches) model. The embodiment can use the Stable Diffusion XL pre-trained model, and the image encoder before the style image injection uses the ViT-L / 14 (Vision Transformer Large model with 14patches) model.

[0080] It should be noted that Stable Diffusion XL is a two-stage cascade diffusion model consisting of a Base model and a Refiner model. The Base model has functions such as text-generated images, image-generated images, and image inpainting, while the Refiner model further refines the image Latent features generated by the Base model to improve the quality and details of the generated image. In addition, the ViT-L / 14 model is a model based on the Vision Transformer (ViT), which is a variant of the Transformer architecture. The model uses a large Transformer architecture and divides the input image into 14 regions for processing, which can maintain high efficiency and accuracy when processing large-scale images.

[0081] The embodiments provided by the present disclosure can generate images with thorough style transfer and excellent content preservation without additional training process, which can greatly reduce the computational cost and data set dependence and improve practicality. And by introducing frequency domain operation, the content preservation effect can be improved. Introducing frequency domain processing in style transfer, through filtering operations on specific frequency parts of the image, the denoising process can enhance the attention to content layout, partial filter details and contours, thereby significantly improving the content fidelity and style consistency of the generated image.

[0082] Based on the above embodiments, the present disclosure also provides an image style transfer method, such as Figure 4 As shown, the method may include the following steps:

[0083] In step S410, an image to be processed and a style image are obtained.

[0084] In the embodiment, the image to be processed may specifically be the content image in the above embodiment. By performing style transfer processing on the image to be processed and the style image, the image to be processed has the style features of the style image.

[0085] In step S420, the image to be processed is subjected to noise addition processing to obtain a first noise image.

[0086] In an embodiment, the image to be processed may be gradually denoised by inverting the pre-trained denoising diffusion implicit model.

[0087] Specifically, the number of noise adding time steps may be set, and the number of noise adding time steps includes a plurality of noise adding time points. Based on the number of noise adding time steps, the image to be processed is subjected to noise adding processing at each noise adding time point respectively through the pre-trained denoising diffusion implicit model.

[0088] In the embodiment, the number of noise adding time steps T can be set i For example, you can set T i =50. This can set T i Noise adding time points, so that the image to be processed can be denoised at each noise adding time point through the pre-trained denoising diffusion implicit model inversion (DDMIInversion) to obtain a first noisy image.

[0089] In the embodiments provided by the present disclosure, the predicted noise can be obtained by pre-training the prediction head of the denoising diffusion implicit model. The noisy sample obtained at the t-th noise adding time point is obtained, and the noisy sample is subjected to noise adding processing by the predicted noise at the t+1-th noise adding time point to obtain a noisy image. Where 0≤t≤T i At the 0th noise adding time point, the noise adding sample is the image to be processed; at the T i Noise adding time point, the noise adding sample is the noise image, t and T i are positive integers respectively.

[0090] In step S430, at least a portion of the low-frequency component in the first noise image is removed to obtain a second noise image.

[0091] In an embodiment, the first noise image can be transformed into the frequency domain by Fourier transform or fast Fourier transform to obtain a spectrum, and the first noise image can be filtered by a high-pass filter to filter out the low-frequency part of the first noise image, so as to obtain the second noise image.

[0092] In the embodiment, the high-frequency part of the first noise image can be obtained by a high-pass filter, and the low-frequency part of the first noise image can be obtained by a low-pass filter, so that the high-frequency part and the low-frequency part are weighted and summed to obtain the second noise image. In order to highlight the high-frequency part in the content communication, the weight of the high-frequency part can be set to be greater than the weight of the low-frequency part, so that the image content in the content image can be retained, and the details and contours in the content image can also be highlighted.

[0093] In step S440, style information of the style image is extracted, and denoising is performed on the second noisy image based on the style information to obtain a style transferred image.

[0094] In an embodiment, by extracting style information of a style image and performing multi-step denoising on the noisy image, during the denoising process, the style information of the style image can be gradually injected into the denoising image, and finally a result image after style transfer is obtained, which is a style transfer image.

[0095] The image style transfer method provided by the embodiment of the present disclosure obtains a to-be-processed image and a style image, and performs noise processing on the to-be-processed image to obtain a first noise image. At least a portion of the low-frequency component in the first noise image is removed to obtain a second noise image. In this way, the style information of the style image is extracted, and the second noise image is denoised based on the style information to obtain a style transfer image. The embodiment can highlight the high-frequency component of the image by removing at least a portion of the low-frequency component in the first noise image, so that the style transfer image obtained in this way can not only have the style features of the style image, but also improve the fidelity of the image content. It can also avoid the data and time costs brought by additional model training, thereby greatly improving the processing efficiency of style transfer.

[0096] Based on the above embodiment, in another embodiment provided by the present disclosure, when removing at least a part of the low-frequency component in the first noise image, the above step S430 may further specifically include the following steps:

[0097] In step S431, frequency domain filtering is performed on the first noise image to obtain high-frequency components and low-frequency components.

[0098] In step S432, the weights of the high frequency component and the low frequency component are obtained respectively, and the high frequency component and the low frequency component are weighted summed based on the weights to obtain a second noise image, wherein the weight of the high frequency component is greater than the weight of the low frequency component.

[0099] In the embodiment, referring to the above formula (3), since the weight of the high-frequency component is greater than the weight of the low-frequency component, the frequency domain feature vector with reduced low-frequency component is obtained by adding the high-frequency component and the low-frequency component in proportion. In this way, by filtering the high-frequency and low-frequency parts of the image, the denoising process can enhance the attention to content layout, partial filter details and contours, thereby significantly improving the content fidelity and style consistency of the generated image.

[0100] Based on the above embodiment, in another embodiment provided by the present disclosure, when extracting style information of a style image, the above step S440 may further specifically include the following steps:

[0101] In step S441, the style image is input into the trained target instruction prompt model.

[0102] The trained target instruction prompt model is used to extract the feature vector of the image based on the instruction prompt;

[0103] In step S442, a style feature vector of the style image is extracted based on the style instruction prompt through the trained target instruction prompt model, and style information is obtained based on the style feature vector.

[0104] In the embodiment, the style image I s Input the pre-trained IPAdapter-Instruct model, use the content instruction prompt to obtain the content feature vector of the style image. At the same time, input the style image into the pre-trained IPAdapter-Instruct model, use the style instruction prompt to obtain the style feature vector of the image to be processed. In this way, the content feature vector of the style image can be used as the content information of the style image, and the style feature vector of the image to be processed can be used as the style information of the image to be processed.

[0105] Based on the above embodiment, in another embodiment provided by the present disclosure, when denoising the second noisy image based on style information, the above step S440 may further specifically include the following steps:

[0106] In step S443, the denoising time step number is set, and the denoising time step number is consistent with the denoising time step number.

[0107] In the embodiment, the denoising time step number is set to be the same as the denoising time step number in the denoising processing stage, and the filtered image is denoised step by step according to the denoising time step number.

[0108] In step S444, a noise prediction vector is generated based on the noisy samples, style information and denoising time steps.

[0109] In an embodiment, a noise prediction vector may be generated by U-Net based on the noisy sample, style information, and denoising time steps.

[0110] In step S445, the noisy image is subjected to step-by-step denoising processing according to the denoising time steps based on the noise prediction vector to obtain a style transfer image.

[0111] In an embodiment, the noise prediction vector generated by U-Net can be used to gradually denoise the image feature vector obtained after filtering, obtain the image feature vector after style transfer in the latent space, and obtain the image after style transfer through the VAE decoder module (i.e., Decoder). Among them, the VAE decoder is a key component in the VAE model, and its main function is to sample and generate new data from the latent space. VAE consists of two parts: an encoder and a decoder. The encoder is responsible for mapping the input data (such as an image) to a probability distribution of a latent space, while the decoder generates new data from the points sampled in these latent spaces.

[0112] In the case of dividing each functional module according to each function, an embodiment of the present disclosure provides an image style migration device, which may be a server, a terminal, or a chip applied to a server. Figure 5 This is a schematic block diagram of the functional modules of an image style transfer device provided by an exemplary embodiment of the present disclosure. Figure 5 As shown, the image style transfer device includes:

[0113] An image acquisition module 10 is used to acquire an image to be processed and a style image;

[0114] A noise adding module 20, configured to perform noise adding processing on the image to be processed to obtain a first noise image;

[0115] A low-frequency processing module 30, configured to remove at least a portion of the low-frequency components in the first noise image to obtain a second noise image;

[0116] The style transfer module 40 is used to extract style information of the style image, and perform denoising on the second noisy image based on the style information to obtain a style transferred image.

[0117] In another embodiment provided by the present disclosure, the noise adding module is specifically used for:

[0118] Setting a noise adding time step number, wherein the noise adding time step number includes a plurality of noise adding time points;

[0119] Based on the number of noise adding time steps, the image to be processed is subjected to noise adding processing at each noise adding time point respectively through a pre-trained denoising diffusion implicit model.

[0120] In another embodiment provided by the present disclosure, the noise adding module is further configured to:

[0121] Obtaining prediction noise through the prediction head of the pre-trained denoising diffusion implicit model;

[0122] Obtain the noisy sample obtained at the t-th noisy time point, and perform a noisy process on the noisy sample using the predicted noise at the t+1-th noisy time point to obtain a noisy image; wherein 0≤t≤T i , at the 0th noise adding time point, the noise adding sample is the image to be processed; at the Tth i Noise adding time point, the noise adding sample is the noise image, t and T i are positive integers respectively.

[0123] In another embodiment provided by the present disclosure, the style transfer module is specifically used to:

[0124] Set the denoising time step, the denoising time step is consistent with the denoising time step;

[0125] Generate a noise prediction vector based on the noise-added sample, the style information and the denoising time step;

[0126] Based on the noise prediction vector, the noisy image is subjected to step-by-step denoising processing according to the denoising time steps to obtain a style transfer image.

[0127] In another embodiment provided by the present disclosure, the low frequency processing module is specifically used for:

[0128] Performing frequency domain filtering on the first noise image to obtain a high-frequency component and a low-frequency component;

[0129] The weights of the high-frequency component and the low-frequency component are respectively obtained, and based on the weights, the high-frequency component and the low-frequency component are weighted summed to obtain a second noise image; wherein the weight of the high-frequency component is greater than the weight of the low-frequency component.

[0130] In another embodiment provided by the present disclosure, the style transfer module is specifically used to:

[0131] Inputting the style image into a trained target instruction prompt model; wherein the trained target instruction prompt model is used to extract a feature vector of the image based on the instruction prompt;

[0132] Based on the style instruction prompt, a style feature vector of the style image is extracted through the trained target instruction prompt model, and the style information is obtained based on the style feature vector.

[0133] The image style transfer device provided by the embodiment of the present disclosure obtains a to-be-processed image and a style image, and performs noise processing on the to-be-processed image to obtain a first noise image. At least a portion of the low-frequency component in the first noise image is removed to obtain a second noise image. In this way, the style information of the style image is extracted, and the second noise image is denoised based on the style information to obtain a style transfer image. The embodiment can highlight the high-frequency component of the image by removing at least a portion of the low-frequency component in the first noise image. The style transfer image obtained in this way not only has the style features of the style image, but also can improve the fidelity of the image content. It can also avoid the data and time costs brought by additional model training, thereby greatly improving the processing efficiency of style transfer.

[0134] An embodiment of the present disclosure also provides an electronic device, comprising: at least one processor; and a memory for storing instructions executable by the at least one processor; wherein the at least one processor is configured to execute the instructions to implement the above method disclosed in the embodiment of the present disclosure.

[0135] Figure 6 The structure diagram of an electronic device provided by an exemplary embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the electronic device 1800 includes at least one processor 1801 and a memory 1802 coupled to the processor 1801 , and the processor 1801 can execute corresponding steps in the above method disclosed in the embodiment of the present disclosure.

[0136] The processor 1801 may also be referred to as a central processing unit (CPU), which may be an integrated circuit chip having signal processing capabilities. Each step in the method disclosed in the embodiment of the present disclosure may be completed by an integrated logic circuit of hardware in the processor 1801 or by instructions in the form of software. The processor 1801 may be a general-purpose processor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present disclosure may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a memory 1802, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, or other mature storage media in the art. The processor 1801 reads the information in the memory 1802 and completes the steps of the method in combination with its hardware.

[0137] In addition, when various operations / processes according to the present disclosure are implemented by software and / or firmware, they can be transmitted from a storage medium or a network to a computer system having a dedicated hardware structure, such as Figure 7 The computer system 1900 shown is installed with the programs constituting the software. When the various programs are installed, the computer system can perform various functions, including the functions described above. Figure 7 A structural block diagram of a computer system provided for an exemplary embodiment of the present disclosure.

[0138] Computer system 1900 is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0139] like Figure 7As shown, the computer system 1900 includes a computing unit 1901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1902 or a computer program loaded from a storage unit 1908 to a random access memory (RAM) 1903. In the RAM 1903, various programs and data required for the operation of the computer system 1900 can also be stored. The computing unit 1901, the ROM 1902, and the RAM 1903 are connected to each other via a bus 1904. An input / output (I / O) interface 1905 is also connected to the bus 1904.

[0140] A plurality of components in the computer system 1900 are connected to the I / O interface 1905, including: an input unit 1906, an output unit 1907, a storage unit 1908, and a communication unit 1909. The input unit 1906 may be any type of device capable of inputting information to the computer system 1900, and the input unit 1906 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 1907 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 1908 may include, but is not limited to, a disk, an optical disk. The communication unit 1909 allows the computer system 1900 to exchange information / data with other devices over a network such as the Internet, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0141] The computing unit 1901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1901 performs the various methods and processes described above. For example, in some embodiments, the above methods disclosed in the embodiments of the present disclosure may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1908. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via ROM 1902 and / or communication unit 1909. In some embodiments, the computing unit 1901 may be configured to perform the above methods disclosed in the embodiments of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0142] The embodiment of the present disclosure further provides a computer-readable storage medium, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the above method disclosed in the embodiment of the present disclosure.

[0143] The computer-readable storage medium in the disclosed embodiments may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specifically, the computer-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0144] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0145] The embodiments of the present disclosure further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor, the above method disclosed in the embodiments of the present disclosure is implemented.

[0146] In embodiments of the present disclosure, computer program codes for performing the operations of the present disclosure may be written in one or more programming languages ​​or combinations thereof, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer.

[0147] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0148] The modules, components or units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the names of the modules, components or units do not, in some cases, limit the modules, components or units themselves.

[0149] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0150] The above descriptions are only some embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, a technical solution formed by replacing the above features with the technical features with similar functions disclosed in the present disclosure (but not limited to).

[0151] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for image style transfer, characterized in that: The method comprises: Get the image to be processed and the style image; Performing noise processing on the image to be processed to obtain a first noise image; removing at least a portion of the low-frequency components in the first noise image to obtain a second noise image; Style information of the style image is extracted, and denoising is performed on the second noisy image based on the style information to obtain a style transferred image.

2. The method according to claim 1, characterized in that The step of performing noise processing on the image to be processed to obtain a first noise image includes: Setting a noise adding time step number, wherein the noise adding time step number includes a plurality of noise adding time points; Based on the number of noise adding time steps, the image to be processed is subjected to noise adding processing at each noise adding time point respectively through a pre-trained denoising diffusion implicit model.

3. The method according to claim 2, characterized in that The method of performing noise addition processing on the image to be processed at each noise addition time point by using the pre-trained denoising diffusion implicit model comprises: Obtaining prediction noise through the prediction head of the pre-trained denoising diffusion implicit model; Obtain the noisy sample obtained at the t-th noisy time point, and perform a noisy process on the noisy sample using the predicted noise at the t+1-th noisy time point to obtain a noisy image; wherein 0≤t≤T i , at the 0th noise adding time point, the noise adding sample is the image to be processed; at the Tth i Noise adding time point, the noise adding sample is the noise image, t and T i are positive integers respectively.

4. The method according to claim 3, characterized in that Performing denoising on the second noisy image based on the style information to obtain a style-transferred image includes: Set the denoising time step, the denoising time step is consistent with the denoising time step; Generate a noise prediction vector based on the noise-added sample, the style information and the denoising time step; Based on the noise prediction vector, the noisy image is subjected to step-by-step denoising processing according to the denoising time steps to obtain a style transfer image.

5. The method according to claim 1, characterized in that The removing at least a part of the low-frequency component in the first noise image to obtain the second noise image includes: Performing frequency domain filtering on the first noise image to obtain a high-frequency component and a low-frequency component; The weights of the high-frequency component and the low-frequency component are respectively obtained, and based on the weights, the high-frequency component and the low-frequency component are weighted summed to obtain a second noise image; wherein the weight of the high-frequency component is greater than the weight of the low-frequency component.

6. The method according to claim 1, characterized in that The extracting the style information of the style image includes: Inputting the style image into a trained target instruction prompt model; wherein the trained target instruction prompt model is used to extract a feature vector of the image based on the instruction prompt; Based on the style instruction prompt, a style feature vector of the style image is extracted through the trained target instruction prompt model, and the style information is obtained based on the style feature vector.

7. An image style transfer device, characterized in that: The device comprises: An image acquisition module, used to acquire the image to be processed and the style image; A noise adding module, used for performing noise adding processing on the image to be processed to obtain a first noise image; A low-frequency processing module, used to remove at least a part of the low-frequency components in the first noise image to obtain a second noise image; The style transfer module is used to extract style information of the style image and perform denoising on the second noisy image based on the style information to obtain a style transferred image.

8. An electronic device, characterized in that: include: at least one processor; a memory for storing the at least one processor-executable instruction; The at least one processor is configured to execute the instructions to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the method as claimed in any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images