Image processing method and device, program product, computer equipment and storage medium

Abstract

The invention discloses an image processing method and device, a program product, computer equipment and a storage medium. The method comprises the following steps: acquiring an original image; carrying out noise addition processing on the original image to obtain a noise-added image of the original image; and inputting a noise-added image of the original image into the trained diffusion model to obtain a de-noised noise output by the trained diffusion model, and performing de-noising processing according to the noise-added image of the original image and the de-noised noise to obtain a target image. By adopting the method and the device, the image style and texture details of the target style image are learned by using the trained diffusion model, so that the image style of the original image can be migrated to the target style and the texture information of the target image can be added in the process of de-noising the noise-added image of the original image, thereby improving the quality of the generated image and improving the image quality. And a more real image is generated.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, program product, computer equipment and storage medium. Background Art

[0002] With the continuous development of computer and artificial intelligence (AI) technology, image processing has become an important direction in the field of computer science and artificial intelligence. Among them, using generative networks to generate images to produce desired images has become an important branch in the field of image processing. When the number of target samples is small (for example, less than or equal to 10), traditional image generation methods often have the problem of generating unrealistic images. Therefore, how to use generative models to generate more realistic images has become an urgent problem to be solved. Summary of the invention

[0003] The present application provides an image processing method, apparatus, program product, computer device and storage medium, which can improve the quality of generated images and generate more realistic images.

[0004] On the one hand, the present application provides an image processing method, including: obtaining an original image; performing noise processing on the original image to obtain a noisy image of the original image; inputting the noisy image of the original image into a trained diffusion model to obtain denoised noise output by the trained diffusion model, wherein the trained diffusion model is trained using a plurality of first sample images, a plurality of second sample images, a diffusion loss and a style loss, the image styles of the plurality of second sample images are inconsistent with the image styles of the plurality of first sample images, the diffusion loss is used to characterize the difference between the predicted denoised noise output by the diffusion model based on the input noisy image of the target first sample image and the preset noise used to obtain the noisy image of the target first sample image, the style loss is used to characterize the style difference between the denoised image of the target second sample image and the plurality of first sample images, the denoised image of the target second sample image is obtained by the diffusion model based on the input noisy image of the target second sample image and the predicted denoised noise output by the diffusion model, the target first sample image is one of the plurality of first sample images, and the target second sample image is one of the plurality of second sample images; denoising is performed based on the denoised image and the denoised noise of the original image to obtain the target image.

[0005] On the other hand, the present application provides an image processing device, including: an acquisition module, used to acquire an original image; a processing module, used to perform noise processing on the original image to obtain a noisy image of the original image; the processing module is also used to input the noisy image of the original image into a trained diffusion model to obtain denoised noise output by the trained diffusion model, wherein the trained diffusion model is trained using multiple first sample images, multiple second sample images, diffusion loss and style loss, the image styles of the multiple second sample images are inconsistent with the image styles of the multiple first sample images, and the diffusion loss is used to characterize the diffusion model according to the input target first sample image. The predicted denoising noise outputted by the denoised image is different from the preset noise used to obtain the denoised image of the target first sample image. The style loss is used to characterize the style difference between the denoised image of the target second sample image and multiple first sample images. The denoised image of the target second sample image is obtained by the diffusion model based on the denoised image of the input target second sample image and the predicted denoised noise outputted by the diffusion model. The target first sample image is one of the multiple first sample images, and the target second sample image is one of the multiple second sample images. The processing module is also used to perform denoising processing based on the denoised image and denoised noise of the original image to obtain the target image.

[0006] On the other hand, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the image processing method in the present application.

[0007] On the other hand, the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor executes the above-mentioned image processing method.

[0008] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned image method.

[0009] The present application can obtain an original image; perform noise processing on the original image to obtain a noisy image of the original image; input the noisy image of the original image into a trained diffusion model to obtain denoised noise output by the trained diffusion model, wherein the trained diffusion model is trained using a plurality of first sample images, a plurality of second sample images, diffusion loss and style loss, the image styles of the plurality of second sample images are inconsistent with the image styles of the plurality of first sample images, the diffusion loss is used to characterize the difference between the predicted denoised noise output by the diffusion model based on the input noisy image of the target first sample image and the preset noise used to obtain the noisy image of the target first sample image, the style loss is used to characterize the style difference between the denoised image of the target second sample image and the plurality of first sample images, the denoised image of the target second sample image is obtained by the diffusion model based on the input noisy image of the target second sample image and the predicted denoised noise output by the diffusion model, the target first sample image is one of the plurality of first sample images, and the target second sample image is one of the plurality of second sample images; denoising is performed based on the denoised image and the denoised noise of the original image to obtain the target image. It can be seen that in the method proposed in the present application, in the process of denoising the noisy image of the original image, the image style of the original image can be transferred to the target style, and the texture information of the target image can be added, thereby improving the image quality of the generated image and generating a more realistic target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0011] Figure 1 is a flowchart of an example of a diffusion model training method provided by the present application;

[0012] Figure 2 is a diagram of an example of a training framework of a diffusion model training method provided in the present application;

[0013] Figure 3 is a flow chart of an image processing method provided by the present application;

[0014] Figure 4 is a diagram of examples of real defect samples and generated defect samples provided by this application;

[0015] Figure 5 It is a structural schematic diagram of an image processing device provided by the present application;

[0016] Figure 6It is a structural schematic diagram of a computer device provided by this application. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in this application to clearly and completely describe the technical solutions in this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0018] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. With the development of technology, artificial intelligence technology has been applied in more fields and played an increasingly important role.

[0019] This application involves artificial intelligence related technologies. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0020] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0021] This application mainly involves machine learning in artificial intelligence. Among them, machine learning (ML) is a multi-field interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines, specializing in how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching.

[0022] The machine learning involved in this application mainly refers to using machine learning to obtain a neural network for image processing, so as to perform image processing through the neural network. In the process of denoising the noisy image of the original image, texture information with a target style can be added to generate an image with a target style. For details, please refer to the description in the following embodiments.

[0023] The image processing method provided in the embodiment of the present application can be applied in a computer device such as a terminal or a server. After acquiring the original image, the computer device can perform noise addition and denoising on the original image. In the process of denoising the noisy image of the original image, the image style can be transferred to the target style, and texture information with the target style can be added, thereby generating an image with the target style.

[0024] Specifically, the computer device may acquire an original image, perform noise processing on the original image, and obtain a noisy image of the original image. Then, the computer device may also determine to input the noisy image of the original image into a trained diffusion model to obtain denoised noise output by the trained diffusion model. Finally, the computer device may perform denoising processing based on the noisy image of the original image and the denoised noise to obtain a target image.

[0025] In other words, after receiving the original image, the computer device can perform noise addition and denoising on the original image. During the denoising process, the image style is transferred to the target style and texture information with the target style is added to generate a target image with the target style, and the generated image is more realistic.

[0026] Optionally, the above-mentioned image may be acquired by the computer device from a local memory of the computer device, or may be acquired from the cloud, or may be acquired from other devices.

[0027] Optionally, the computer device may be a server. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0028] Optionally, the computer device may be a terminal device, which may be a smart terminal such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, etc.

[0029] The computer device can perform noise addition and denoising on the image through an AI-based deep learning method. For example, the computer device can call a diffusion model based on a U-net network to perform noise addition and denoising on the image. The computer device can obtain a trained diffusion model and a small number of sample images with a target style in advance, and then the computer device can train the diffusion model based on these sample images, and use the trained diffusion model to generate an image with a target style.

[0030] See also Figure 1 , Figure 1 It is a flowchart of an example of a method for training a diffusion model provided by the present application. The execution subject in the embodiment of the present application may be the above-mentioned computer device. The computer device may call the diffusion model and train it to obtain the trained diffusion model provided by the present application.

[0031] Diffusion models can generally perform two processing modes: diffusion process and reverse diffusion process. Diffusion process is the process of gradually adding Gaussian noise to the original image to turn it into a noise image. By continuously adding noise to the original image for a preset number of times, the resulting image can basically be said to be a noise image, thereby achieving the purpose of destroying the image. Reverse diffusion process is the process of gradually restoring an image from noise. The reverse diffusion process will lose information. The image restored by reverse diffusion process is similar to the original image and may be different from the original image. The image obtained by reverse diffusion process can be used as a newly generated image.

[0032] Optionally, the diffusion model may be a U-type network (Unet), including a Unet encoder and a Unet decoder.

[0033] like Figure 1 As described, the training method of the diffusion model provided by the present application may include:

[0034] Step S101 , performing noise addition processing on a target first sample image according to a first noise addition step number and a preset noise, to obtain a noisy image of the target first sample image.

[0035] Optionally, the computer device may sample the first noise addition step number from the first step number interval.

[0036] For example, the computer device may sample in a step interval, for example, from 0 to T, to obtain a first noisy step number t1. Optionally, t1 may be sampled by the computer device from a uniform distribution U(0,T) from 0 to T. Furthermore, the computer device may randomly sample from a plurality of first sample images to obtain a target first sample image. For the sampled target first sample image, the computer device may, based on a preset Gaussian noise, gradually add Gaussian noise to the sampled target first sample image until the step number reaches the sampled first noisy step number t1, to perform noise processing on the target first sample image, to obtain a noisy image of the target first sample image. For example, the computer device may obtain a t1-step noisy image of the target first sample image. The properties of the preset Gaussian noise and various parameters may be obtained in the process of training the diffusion model.

[0037] Optionally, the plurality of first sample images may be sample images having a target style, that is, it is desired to train the diffusion model to be able to generate images having image features of the first sample images.

[0038] Optionally, the number of images of the first sample is m, and m may be less than or equal to ten.

[0039] Step S102: inputting the noisy image of the target first sample image into a diffusion model to obtain predicted noisy noise output by the diffusion model.

[0040] The computer device can call the diffusion model to predict the noise added to the noisy image of the target first sample image to obtain the predicted noisy noise. For example, the computer device can predict the noise added to the t1-step noisy image of the first sample image at the first noisy step number t1. It is easy to understand that if the computer device predicts the noise added at the first noisy step number t1, then removes the noise added at the first noisy step number t1 from the t1-step noisy image of the first sample image, and obtains the t1-1-step noisy image of the first sample image.

[0041] The diffusion model can be obtained through training based on the existing multiple second sample images. The number of second sample images can be sufficiently large, and the image styles of the existing large number of second sample images are inconsistent with the target style of the multiple first sample images. For example, the image style of the first sample image can be the character portrait style of cartoon animation, and the image style of the second sample image can be the style of ordinary character head portrait photos, that is, the diffusion model can be obtained through training based on a large number of character head portrait photos.

[0042] Step S103 , performing noise addition processing on the target second sample image according to the second noise addition step number to obtain a noisy image of the target second sample image.

[0043] Optionally, the computer device may sample from a second step number interval to obtain a second noise adding step number.

[0044] For example, the computer device may sample in a step interval of, for example, 0 to T to obtain a second noise step number t2. Alternatively, t2 may be sampled by the computer device in a uniform distribution U(0,T) from 0 to T. Since the image processed by the diffusion model is approximately a completely noisy image when the noise step number is close to T, in order to speed up the training of the diffusion model, t2 may be sampled by the computer device in a uniform distribution U(0,T) from 0 to T. e The uniform distribution U(0,T e ), for example, T = 1000, T e = 800. In addition, the computer device may randomly sample from the plurality of second sample images to obtain a target second sample image.

[0045] Optionally, the second noise adding step number may be the same as the first noise adding step number.

[0046] Optionally, the second noise adding step number may be different from the first noise adding step number.

[0047] Step S104: input the noisy image of the target second sample image into the diffusion model to obtain the predicted denoised noise output by the diffusion model.

[0048] The computer device can call the diffusion model to predict the denoising noise required for the noisy image of any second sample image, and obtain the predicted denoising noise. In other words, in the process of denoising the noisy image of the target second sample image, the computer device can predict the denoising noise required for denoising the noisy image of the target second sample image, thereby obtaining the predicted denoising noise. For example, the computer device can predict the denoising noise required for denoising the t2-1 step noisy image of the target second sample image. Even for example, the computer device can predict the denoising noise required for directly denoising the t2-1 step noisy image of the target second sample image to obtain the denoising noise required for the 0-step noisy image (i.e., the predicted original image, or the generated image). It is easy to understand that the denoising effect of the diffusion model with a small number of steps is better than the denoising effect with a large number of steps.

[0049] Optionally, step S104 may include: inputting the noisy image of the target second sample image into an encoder in the diffusion model to obtain a feature image of the noisy image of the target second sample image; when the second noisy step number is greater than or equal to a preset step number threshold, fusing the feature image of the noisy image of the target second sample image with the feature image of the target second sample image output by the encoder to obtain a fused feature image; inputting the fused feature image into a decoder of the diffusion model, and outputting it to obtain the predicted denoising noise output by the diffusion model.

[0050] A step number threshold can be preset to allow the training of the diffusion model to be divided into two stages. When the number of noise addition steps is greater than or equal to the preset step number threshold, the feature image of the target second sample image is added during the training of the diffusion model so that the diffusion model can better learn the content information of the second sample image.

[0051] Optionally, step S104 may include: inputting the noisy image of the target second sample image into an encoder in the diffusion model to obtain a feature image of the noisy image of the target second sample image; when the second noisy step number is less than a preset step number threshold, fusing the feature image of the noisy image of the target second sample image with the feature image of Gaussian noise output by the encoder to obtain a fused feature image; inputting the fused feature image into a decoder of the diffusion model, and outputting it to obtain the predicted denoised noise output by the diffusion model.

[0052] A step number threshold can be preset to allow the training of the diffusion model to be divided into two stages. When the number of noise addition steps is less than the preset step number threshold, a feature image of Gaussian noise is added during the training of the diffusion model so that the diffusion model can better learn to predict the noise.

[0053] Step S105 , obtaining a denoised image of the target second sample image according to the denoised image of the second sample image and the predicted denoised noise.

[0054] The computer device can use the predicted denoising noise to denoise the noisy image of the target second sample image, and directly predict the denoised image of the noisy image of the target second sample image based on the diffusion model. For example, the computer device can use the predicted t2-step denoised image of the target second sample image to denoise to obtain the predicted denoising noise required for the t2-1-step denoised image, obtain the t2-1-step denoised image of the target second sample image from the t2-step denoised image of the target second sample image, and directly predict the denoised image of the noisy image of the second sample image from the t2-1-step denoised image of the target second sample image based on the diffusion model.

[0055] Optionally, a method such as low-pass filtering may be introduced into the denoising process to enhance the reconstruction of content information.

[0056] Step S106 , training a diffusion model according to the style loss between the denoised image of the target second sample image and the plurality of first sample images, and the diffusion loss between the predicted added noise and the preset noise.

[0057] The computer device may obtain the predicted noise of the sampled target first sample image, and compare it with the preset noise to obtain the difference between the predicted noise and the preset noise. For example, the computer device predicts the noise added when the first noise adding step number t1 is obtained, that is, the predicted noise ε θ , and compared with the preset noise ε added at the first noise adding step number t1, so as to obtain the difference between the predicted noise adding noise and the preset noise. Optionally, the difference between the predicted noise adding noise and the preset noise can be the L2 distance between the predicted noise adding noise and the preset noise. Optionally, the L2 distance between the predicted noise adding noise and the preset noise can be called diffusion loss, expressed as L dif .

[0058] The computer device can extract the image style of the denoised image of the noisy image of the target second sample image and the image style of the multiple first sample images based on the VGG (Visual Geometry Group) network. For example, the computer device can use the VGG network to extract the denoised image of the noisy image of the second sample image and the depth features of the first sample image, and calculate the Gram matrix for the extracted depth features respectively, to obtain the image style matrix of the denoised image of the noisy image of the target second sample image and the image style matrix of the multiple first sample images.

[0059] The computer device may also obtain the difference between the image style of the denoised image of the noisy image of the target second sample image and the image style of the plurality of first sample images. For example, the difference between the image style of the denoised image of the noisy image of the target second sample image and the image style of the plurality of first sample images may be the L2 distance between the Gram matrix of the depth feature of the denoised image of the noisy image of the target second sample image and the Gram matrix of the depth feature of the plurality of first sample images. Optionally, the L2 distance between the Gram matrices representing the image style may be referred to as the style loss, expressed as L style , where L style It can be expressed as a weighted sum of the L2 distances between the Gram matrices of the depth features of the denoised image of the noisy image of the target second sample image and the Gram matrices of the depth features of each of the multiple first sample images.

[0060] Optionally, step S106 may include: based on the first number of noisy steps, obtaining a weight of the diffusion loss, the weight of the diffusion loss is negatively correlated with the first number of noisy steps; when the second number of noisy steps is greater than or equal to a preset step threshold, obtaining a weight of the style loss based on the second number of noisy steps, the weight of the style loss is negatively correlated with the second number of noisy steps; based on the weight of the diffusion loss and the weight of the style loss, performing weighted summation of the diffusion loss and the style loss to obtain the total loss of the diffusion model; training the diffusion model with the reduction of the total loss as the training goal to obtain a trained diffusion model.

[0061] For example, the computer device can obtain the diffusion loss L based on the predicted noise and the preset noise. dif The computer device may obtain a style loss L based on the image style of the denoised image and the image styles of the plurality of first sample images. style Based on the first noise adding step number t1, the diffusion loss weight w(t1) is obtained. The diffusion loss weight is negatively correlated with the first noise adding step number, for example, it can be When the second noise adding step number is greater than the first preset step number threshold T s When , based on the second noise adding step number t2, the weight of style loss w(t2) is obtained. The weight of style loss is negatively correlated with the second noise adding step number, for example, it can be Based on the weight of the diffusion loss and the weight of the style loss, the diffusion loss and the style loss are weighted and summed to obtain the total loss of the diffusion model. For example, the final total loss can be L = w(t1)·L dif +m(t2)(1-w(t2))·L style ,in, is a Sigmoid function, which can indicate that when the second noise adding step number t2 is greater than the first preset value T sWhen the style loss L is introduced in the training of the diffusion model style ; Taking reducing the total loss as the training goal, the diffusion model is trained to obtain the trained diffusion model.

[0062] It can be found that training the diffusion model based on the target first sample image can enable the diffusion model to further learn the texture details of the target first sample image, and can enable the trained diffusion model to generate an image with image features of the texture details of the first sample image, making the generated image more realistic. In addition, introducing the learning of the styles of multiple first sample images in the training process of the diffusion model can make the image style of the image generated by the trained diffusion model closer to the image style of the first sample image.

[0063] Further, step S106 may also include: when the second denoising step number is greater than or equal to a preset step number threshold, determining the cross-domain directional consistency loss between the direction vector of the denoised image of the target second sample image and the reference direction vector of the denoised image of the target second sample image; determining the weight of the cross-domain directional consistency loss, the weight of the cross-domain directional consistency loss is a preset value; based on the weight of the diffusion loss and the weight of the style loss, performing weighted summation of the diffusion loss and the style loss to obtain the total loss of the diffusion model, including: based on the weight of the diffusion loss, the weight of the style loss and the weight of the cross-domain directional consistency loss, performing weighted summation of the diffusion loss, the style loss and the cross-domain directional consistency loss to obtain the total loss of the diffusion model.

[0064] The computer device may perform feature extraction on the plurality of second sample images based on a CLIP (Contrastive Language Image Pretraining) model to obtain direction vectors of the plurality of second sample images, and further obtain an average direction vector of the plurality of second sample images.

[0065] The computer device may perform feature extraction on the plurality of first sample images based on the CLIP model to obtain direction vectors of the plurality of first sample images, and further obtain an average direction vector of the plurality of first sample images.

[0066] The computer device can perform feature extraction on the multiple second sample images to obtain the direction vectors of the multiple second sample images. Similarly, the computer device can perform feature extraction on the multiple first sample images to obtain the direction vectors of the multiple first sample images. Further, the computer device can obtain the center of the direction vectors of the multiple second sample images, that is, the average direction vector of the multiple second sample images, and the center of the direction vectors of the multiple first sample images, that is, the average direction vector of the multiple second sample images. By calculating the vector difference between the center of the direction vectors of the multiple first sample images and the center of the direction vectors of the multiple second sample images, the computer device can obtain the cross-domain direction vectors between the multiple second sample images and the multiple first sample images.

[0067] The computer device may add the direction vector of the target second sample image to the cross-domain direction vector, thereby obtaining a reference direction vector of the denoised image of the noisy image of the target second sample image.

[0068] The computer device may use the CLIP model to perform feature extraction on the denoised image of the noisy image of the target second sample image to obtain a direction vector of the denoised image of the noisy image of the target second sample image.

[0069] The computer device may also obtain a difference between a direction vector of a denoised image of a noisy image of the target second sample image and a reference direction vector. For example, the difference between a direction vector of a denoised image of a noisy image of the target second sample image and a reference direction vector may be an L2 distance between a direction vector of a denoised image of a noisy image of the target second sample image and a vector of a reference direction vector. Optionally, the L2 distance between a direction vector of a denoised image of a noisy image of the target second sample image and a vector of a reference direction vector may be referred to as a cross-domain direction consistency loss, expressed as L DDC .

[0070] The weight of the cross-domain direction consistency loss may be a preset value, for example, 1.

[0071] The computer device performs weighted summation of the diffusion loss and the style loss based on the weight of the diffusion loss, the weight of the style loss, and the weight of the cross-domain directional consistency loss to obtain the total loss of the diffusion model. For example, the final total loss can be L = w(t1)·L dif +m(t2)(1-w(t2))·L style +m(t2)·L DDC ,in, is a Sigmoid function, which can indicate that when the second noise adding step number t2 is greater than the first preset value T s When the style loss L is introduced in the training of the diffusion model styleand cross-domain consistency loss; taking reducing the total loss as the training goal, the diffusion model is trained to obtain the trained diffusion model.

[0072] It can be found that training the diffusion model based on the target first sample image can enable the diffusion model to further learn the texture details of the target first sample image, and can enable the trained diffusion model to generate an image with image features of the texture details of the first sample image, making the generated image more realistic. Moreover, introducing the learning of the styles of multiple first sample images in the training process of the diffusion model can make the image style of the image generated by the trained diffusion model closer to the image style of the first sample image. In addition, in the training process of the diffusion model, the constraint on the feature direction is emphasized, and the direction of the features of the generated image in the feature space is closer to the direction of the features of the original image in the feature space, which can better avoid overfitting of the diffusion model and avoid image distortion.

[0073] Alternatively, a more complex and effective network design, such as Transformer, can be used to capture global feature information.

[0074] See also Figure 2 , Figure 2 The diagram is an example of a training framework of a diffusion model training method provided in the present application.

[0075] like Figure 2 As shown, the diagram of the example of the training framework illustrates the training method of the diffusion model.

[0076] Optionally, the diffusion model may be Unet, including a Unet encoder and a Unet decoder.

[0077] First, the computer device can sample at least one first sample image x from the cartoon avatar sample (ie, the first sample) B , sample the noise step number t from the uniform distribution of U(0,T), and convert the sampled image x B After adding noise for t steps, it is input into the Unet of the diffusion model, the added noise is predicted, and the L2 loss of the added noise and the predicted noise is calculated as the diffusion loss L dif .

[0078] Next, the computer device may sample at least one second sample image x from the common person portrait sample (ie, the second sample). A , and the sampled image x is also transformed using the number of noise steps t sampled from the uniform distribution of U(0,T). A After adding noise for t steps, it is input into the Unet of the diffusion model to predict the noise ∈ used for denoising. The denoising results are According to the one-step denoising formula p of the diffusion model θ (x0|xt-1 ) calculate the corresponding denoised image And calculate the style loss L style and cross-domain direction consistency loss L DDC The style loss L style Used to constrain denoised images Style difference with cartoon avatars, cross-domain direction consistency loss L DDC It is used to constrain the difference between the directions of the features of the generated image in the feature space and the directions of the features of the original image in the feature space, so as to avoid image distortion caused by overfitting of the diffusion model.

[0079] By using the Sigmoid function m(t), the stage feature fusion can be regulated based on the preset step threshold. When the noise step number t is greater than the preset step threshold (i.e., the step number is large), the content features of the character portrait samples are fused to learn the content of the character portrait samples. When the noise step number t is less than the preset step threshold (i.e., the step number is small), the local texture features of the cartoon portrait samples are fused to learn the local details of the cartoon character portraits.

[0080] Finally, according to the number of noise addition steps t, the diffusion loss L is assigned dif , style loss L style and cross-domain direction consistency loss L DDC The corresponding weights are further constrained to mainly learn the style of cartoon avatars and the content of ordinary person avatars when the number of noise adding steps t is large, and mainly learn the local details and textures of cartoon avatars when the number of noise adding steps t is small, and adjust the parameters of the diffusion model Unet to complete the training of the diffusion model.

[0081] Based on the trained diffusion model, the image processing method of the present application can be applied.

[0082] See also Figure 3 , Figure 3 1 is a flowchart of an image processing method provided by the present application. The execution subject in the embodiment of the present application may be a computer device, which may include, for example, a server and a terminal device. Figure 3 As shown, the image processing method may include:

[0083] Step S301, obtaining the original image.

[0084] The computer device can obtain the original image in the hope of obtaining the target image through the original image. Step S302: performing noise processing on the original image to obtain a noise-added image of the original image.

[0085] The computer device can perform noise processing on the original image to obtain a noised image of the original image, so as to facilitate style transfer of the original image and obtain the target image.

[0086] Step S303: input the noisy image of the original image into the trained diffusion model to obtain the denoised noise output by the trained diffusion model.

[0087] The trained diffusion model is obtained by training using multiple first sample images, multiple second sample images, diffusion loss and style loss. The image styles of the multiple second sample images are inconsistent with the image styles of the multiple first sample images. The diffusion loss is used to characterize the difference between the predicted denoising noise output by the diffusion model based on the input denoised image of the target first sample image and the preset noise used to obtain the denoised image of the target first sample image. The style loss is used to characterize the style difference between the denoised image of the target second sample image and the multiple first sample images. The denoised image of the target second sample image is obtained by the diffusion model based on the input denoised image of the target second sample image and the predicted denoising noise output by the diffusion model. The target first sample image is one of the multiple first sample images, and the target second sample image is one of the multiple second sample images. The computer device uses the trained diffusion model to denoise the image after denoising, and predicts the denoising noise required for denoising the denoised image of the original image in the process of generating the target image.

[0088] Step S304, performing denoising processing on the noisy image and the denoised noise of the original image to obtain a target image.

[0089] The computer device uses the denoising noise of the noisy image of the original image predicted by the trained diffusion model to denoise the noisy image of the original image. During the denoising process, the image style is transferred and transformed into a target style, and finally a target image with the target style is generated.

[0090] The present application can obtain an original image; perform noise processing on the original image to obtain a noisy image of the original image; input the noisy image of the original image into a trained diffusion model to obtain denoised noise output by the trained diffusion model, wherein the trained diffusion model is trained using a plurality of first sample images, a plurality of second sample images, diffusion loss and style loss, the image styles of the plurality of second sample images are inconsistent with the image styles of the plurality of first sample images, the diffusion loss is used to characterize the difference between the predicted denoised noise output by the diffusion model based on the input noisy image of the target first sample image and the preset noise used to obtain the noisy image of the target first sample image, the style loss is used to characterize the style difference between the denoised image of the target second sample image and the plurality of first sample images, the denoised image of the target second sample image is obtained by the diffusion model based on the input noisy image of the target second sample image and the predicted denoised noise output by the diffusion model, the target first sample image is one of the plurality of first sample images, and the target second sample image is one of the plurality of second sample images; denoising is performed based on the denoised image and the denoised noise of the original image to obtain the target image. It can be seen that in the method proposed in the present application, since the trained diffusion model used has learned the image style and texture details of the target style image, in the process of denoising the noisy image of the original image, the image style of the original image can be transferred to the target style, and the texture information of the target image can be added, thereby improving the image quality of the generated image and generating a more realistic target image.

[0091] In real life and industrial production, industrial defect detection aims to detect parts manufactured in industrial assembly lines, discover and point out possible defects, and prevent them from having a negative impact on the quality of the final product, production efficiency, safety, and user experience. Since there are often fewer industrial defective parts in actual production lines, most of the existing industrial defect detection methods rely on unsupervised learning methods, and the accuracy of such methods in defect location needs to be improved. In addition, some methods use image processing technology to artificially define rules to simulate industrial defects and generate a large number of specific defect samples to assist the model in supervised learning. However, the defect samples generated by such methods are artificially pre-defined, and are relatively insufficient in type and diversity. In addition, the authenticity of the synthetic images is often poor, resulting in a large difference between the synthetic defect sample domain and the real defect sample domain.

[0092] By using the method of the present application, a large number of defective samples with high authenticity can be generated based on a small number of defective samples (for example, less than 10) and a large number of normal samples.

[0093] In the above embodiment, the first sample image may be an image of a defective sample, and the second sample image may be an image of a normal sample. Since the defective sample has defects such as breakage, cracking, and missing, the image of the defective sample has obvious defects compared with the image of the normal sample.

[0094] Therefore, the image style of the defective sample image is inconsistent with the image style of the normal sample. In other words, the image style of the first sample image can represent that the first sample image has a specific defect, and the image style of the second sample image can represent that the second sample image is normal without any defect.

[0095] Based on this, in the process of training the diffusion model, the diffusion loss is obtained based on the difference between the predicted denoising noise output by the diffusion model according to the input denoised image of the target first sample image and the preset noise used to obtain the denoised image of the target first sample image. The style loss is obtained based on the style difference between the denoised image of the target second sample image and multiple first sample images. The denoised image of the target second sample image is obtained by the diffusion model according to the input denoised image of the target second sample image and the predicted denoising noise output by the diffusion model. The target first sample image is one of the multiple first sample images, and the target second sample image is one of the multiple second sample images. Then, the diffusion model is trained based on the diffusion loss and the style loss. Through the above training method, the image style of the target image obtained by the trained diffusion model can be close to the image style of the first sample image, that is, the defects of the defective sample image constructed based on the normal sample image can be close to the defects of the second sample image.

[0096] Using the method of the present application, a large number of normal sample images are used to train the diffusion model, and a small number of defective samples are combined to further train the diffusion model. When the number of steps is large, the content of the normal samples and the style information of the defective samples are learned, and when the number of steps is small, the local details of the defective samples are learned, thereby improving the capture of content, style and local details. In addition, during the model training process, based on the direction vector and cross-domain direction consistency loss, the consistency between the generated image distribution and the normal sample cloth image distribution can be effectively and stably guaranteed to prevent the diffusion model from overfitting. Finally, the trained diffusion model can be used to generate a large number of defective samples with high authenticity based on a small number of defective samples and a large number of normal samples. For example, Figure 4 shown.

[0097] See also Figure 5 , Figure 5is a schematic diagram of the structure of an image processing device 500 provided in the present application. The image processing device may be a computer program (including program code) running in a computer device, for example, the image processing device is an application software, and the image processing device may be used to execute the corresponding steps of the method provided in the embodiment of the present application. Figure 5 As shown, the image processing device 500 may include: an acquisition module 501 and a processing module 502 .

[0098] The acquisition module 501 is used to acquire the original image.

[0099] The processing module 502 is used to perform noise processing on the original image to obtain a noisy image of the original image.

[0100] The processing module 502 is further used to input the noisy image of the original image into the trained diffusion model to obtain the denoised noise output by the trained diffusion model, wherein the trained diffusion model is obtained by training using multiple first sample images, multiple second sample images, diffusion loss and style loss, the image styles of the multiple second sample images are inconsistent with the image styles of the multiple first sample images, the diffusion loss is used to characterize the difference between the predicted denoised noise output by the diffusion model based on the noisy image of the input target first sample image and the preset noise used to obtain the noisy image of the target first sample image, the style loss is used to characterize the style difference between the denoised image of the target second sample image and the multiple first sample images, the denoised image of the target second sample image is obtained by the diffusion model based on the noisy image of the input target second sample image and the predicted denoised noise output by the diffusion model, the target first sample image is one of the multiple first sample images, and the target second sample image is one of the multiple second sample images.

[0101] The processing module 502 is further used to perform denoising processing based on the noisy image and the denoised noise of the original image to obtain a target image.

[0102] Optionally, the target image is obtained by calling a diffusion model for processing, and the training method of the diffusion model includes: performing denoising processing on the target first sample image according to a first denoising step number and a preset noise to obtain a denoised image of the target first sample image; inputting the denoised image of the target first sample image into the diffusion model to obtain a predicted denoised noise output by the diffusion model; performing denoising processing on the target second sample image according to a second denoising step number to obtain a denoised image of the target second sample image; inputting the denoised image of the target second sample image into the diffusion model to obtain a predicted denoised noise output by the diffusion model; obtaining a denoised image of the target second sample image according to the denoised image of the second sample image and the predicted denoised noise; and training the diffusion model according to the style loss between the denoised image of the target second sample image and multiple first sample images, and the diffusion loss between the predicted denoised noise and the preset noise.

[0103] Optionally, in the training method of the diffusion model, the noisy image of the target second sample image is input into the diffusion model to obtain the predicted denoised noise output by the diffusion model, including: inputting the noisy image of the target second sample image into an encoder in the diffusion model to obtain a feature image of the noisy image of the target second sample image; when the second noisy step number is greater than or equal to a preset step number threshold, fusing the feature image of the noisy image of the target second sample image with the feature image of the target second sample image output by the encoder to obtain a fused feature image; and inputting the fused feature image into a decoder of the diffusion model, and outputting it to obtain the predicted denoised noise output by the diffusion model.

[0104] Optionally, in the training method of the diffusion model, the noisy image of the target second sample image is input into the diffusion model to obtain the predicted denoised noise output by the diffusion model, including: inputting the noisy image of the target second sample image into an encoder in the diffusion model to obtain a feature image of the noisy image of the target second sample image; when the second noisy step number is less than a preset step number threshold, fusing the feature image of the noisy image of the target second sample image with the feature image of Gaussian noise output by the encoder to obtain a fused feature image; and inputting the fused feature image into a decoder of the diffusion model, and outputting it to obtain the predicted denoised noise output by the diffusion model.

[0105] Optionally, the training method of the diffusion model further includes: sampling from the first step number interval to obtain a first noise adding step number.

[0106] Optionally, the training method of the diffusion model further includes: sampling from the second step number interval to obtain a second noisy step number. Optionally, in the training method of the diffusion model, according to the denoised image of the target second sample image and the style loss between the multiple first sample images, and the diffusion loss between the predicted noisy noise and the preset noise, the diffusion model is trained, including: based on the first noisy step number, obtaining the weight of the diffusion loss, the weight of the diffusion loss is negatively correlated with the first noisy step number; when the second noisy step number is greater than or equal to the preset step number threshold, based on the second noisy step number, obtaining the weight of the style loss, the weight of the style loss is negatively correlated with the second noisy step number; based on the weight of the diffusion loss and the weight of the style loss, weighted summing the diffusion loss and the style loss to obtain the total loss of the diffusion model; with the reduction of the total loss as the training goal, the diffusion model is trained to obtain the trained diffusion model.

[0107] Optionally, the training method of the diffusion model also includes: when the second noisy step number is greater than or equal to a preset step number threshold, determining the cross-domain directional consistency loss between the direction vector of the denoised image of the target second sample image and the reference direction vector of the denoised image of the target second sample image; determining the weight of the cross-domain directional consistency loss, the weight of the cross-domain directional consistency loss is a preset value; based on the weight of the diffusion loss and the weight of the style loss, performing weighted summation of the diffusion loss and the style loss to obtain the total loss of the diffusion model, including: based on the weight of the diffusion loss, the weight of the style loss and the weight of the cross-domain directional consistency loss, performing weighted summation of the diffusion loss, the style loss and the cross-domain directional consistency loss to obtain the total loss of the diffusion model.

[0108] Optionally, in the training method of the diffusion model, the cross-domain directional consistency loss between the direction vector of the denoised image of the target second sample image and the reference direction vector of the denoised image of the target second sample image is determined, including: performing feature extraction on multiple second sample images to obtain the average direction vector of the multiple second sample images; performing feature extraction on multiple first sample images to obtain the average direction vector of the multiple first sample images; based on the average direction vector of the multiple second sample images and the average direction vector of the multiple first sample images, obtaining the cross-domain direction vector between the multiple second sample images and the multiple first sample images; for any second sample image in the target second sample images, adding the direction vector of any second sample image to the cross-domain direction vector to obtain the reference direction vector of the denoised image of any second sample image; based on the direction vectors of the denoised images corresponding to each second sample image obtained by sampling and the reference direction vector of the corresponding denoised image, obtaining the cross-domain directional consistency loss.

[0109] According to one embodiment of the present application, Figure 3 The steps involved in the image processing method shown can be represented by Figure 5 The various modules in the image processing device 500 shown in the figure are executed. For example, Figure 3 The step S301 shown in FIG. 1 can be performed by Figure 5 The acquisition module 501 in is executed, Figure 3 The steps S302-S304 shown in FIG. 1 can be performed by Figure 5 The processing module 502 in is executed.

[0110] The present application can obtain an original image; perform noise processing on the original image to obtain a noisy image of the original image; input the noisy image of the original image into a trained diffusion model to obtain denoised noise output by the trained diffusion model, wherein the trained diffusion model is trained using a plurality of first sample images, a plurality of second sample images, diffusion loss and style loss, the image styles of the plurality of second sample images are inconsistent with the image styles of the plurality of first sample images, the diffusion loss is used to characterize the difference between the predicted denoised noise output by the diffusion model based on the input noisy image of the target first sample image and the preset noise used to obtain the noisy image of the target first sample image, the style loss is used to characterize the style difference between the denoised image of the target second sample image and the plurality of first sample images, the denoised image of the target second sample image is obtained by the diffusion model based on the input noisy image of the target second sample image and the predicted denoised noise output by the diffusion model, the target first sample image is one of the plurality of first sample images, and the target second sample image is one of the plurality of second sample images; denoising is performed based on the denoised image and the denoised noise of the original image to obtain the target image. It can be seen that in the method proposed in the present application, in the process of denoising the noisy image of the original image, the image style of the original image can be transferred to the target style, and the texture information of the target image can be added, thereby improving the image quality of the generated image and generating a more realistic target image.

[0111] According to one embodiment of the present application, Figure 5 The various modules in the image processing device 500 shown can be separately or all combined to be composed of one or more units, or one (some) of the units can be further divided into multiple smaller sub-units in function, and the same operation can be achieved without affecting the realization of the technical effects of the embodiments of the present application. The above modules are divided based on logical functions. In practical applications, the functions of a module can also be implemented by multiple units, or the functions of multiple modules can be implemented by one unit. In other embodiments of the present application, the image processing device 500 may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0112] According to one embodiment of the present application, the program can be executed by running on a general-purpose computer device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 3 A computer program (including program code) for each step involved in the corresponding method shown in Figure 5 The image processing apparatus 500 shown in the figure can be used to implement the image processing method of the embodiment of the present application. The above computer program can be recorded on, for example, a computer-readable storage medium, and loaded into the above computing device through the computer-readable storage medium, and run therein.

[0113] See also Figure 6 , Figure 6 600 is a schematic diagram of the structure of a computer device 600 provided by the present application. Figure 6 As shown, the computer device 600 may include: a processor 601, a network interface 604 and a memory 605. In addition, the computer device 600 may also include: a user interface 603, and at least one communication bus 602. The communication bus 602 is used to realize the connection and communication between these components. The user interface 603 may include a display screen (Display), a keyboard (Keyboard), and the user interface 603 may optionally include a standard wired interface and a wireless interface. The network interface 604 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 605 may be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage. The memory 605 may optionally be at least one storage device located away from the aforementioned processor 601. As shown in FIG. Figure 6 As shown, the memory 605 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0114] exist Figure 6 In the computer device 600 shown in the figure, the network interface 604 can provide a network communication function; the user interface 603 is mainly used to provide an input interface for the user; and the processor 601 can be used to call the device control application stored in the memory 605 to execute the above Figure 3 The description of the above-mentioned image processing method in the corresponding embodiment can also be performed Figure 5 The description of the image processing device 500 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0115] In addition, it should be pointed out that the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the image processing device 500 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 3 The description of the image processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0116] As an example, the above program instructions may be deployed on a computer device for execution, or deployed on multiple computer devices located at one location for execution, or executed on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain network.

[0117] The computer-readable storage medium may be an internal storage unit of the image processing apparatus or the computer device provided in any of the aforementioned embodiments, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (smart media card, SMC), a secure digital (secure digital, SD) card, a flash card (flash card), etc. equipped on the computer device. Further, the computer-readable storage medium may also include both an internal storage unit of the computer device and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has been output or is to be output.

[0118] The present application provides a computer program product or a computer program, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above Figure 3 The description of the above-mentioned image processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of adopting the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0119] The terms "first", "second", etc. in the description, claims, and drawings of the embodiments of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the term "comprising" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, device, product, or equipment that includes a series of steps or units is not limited to the listed steps or modules, but optionally includes steps or modules that are not listed, or optionally includes other step units inherent to these processes, methods, devices, products, or equipment.

[0120] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0121] The method and related apparatus provided by the embodiment of the present application are described with reference to the method flow chart and / or structural diagram provided by the embodiment of the present application. Specifically, each process and / or box in the method flow chart and / or structural diagram, as well as the combination of the processes and / or boxes in the flow chart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the functions specified in the process. Figure 1 A process or multiple processes and / or structures Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A flow or multiple flows and / or structures illustrate the steps of the functions specified in one block or multiple blocks.

[0122] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Get the original image; Performing noise processing on the original image to obtain a noisy image of the original image; Inputting the noisy image of the original image into a trained diffusion model to obtain denoised noise output by the trained diffusion model, wherein the trained diffusion model is trained using a plurality of first sample images, a plurality of second sample images, a diffusion loss and a style loss, the image styles of the plurality of second sample images are inconsistent with the image styles of the plurality of first sample images, the diffusion loss is used to characterize the difference between the predicted denoised noise output by the diffusion model based on the noisy image of the input target first sample image and the preset noise used to obtain the noisy image of the target first sample image, the style loss is used to characterize the style difference between the denoised image of the target second sample image and the plurality of first sample images, the denoised image of the target second sample image is obtained by the diffusion model based on the noisy image of the input target second sample image and the predicted denoised noise output by the diffusion model, the target first sample image is one of the plurality of first sample images, and the target second sample image is one of the plurality of second sample images; De-noising is performed based on the noisy image of the original image and the denoised noise to obtain a target image.

2. The image generation method according to claim 1, characterized in that: The method further comprises: Performing noise adding processing on the target first sample image according to a first noise adding step number and a preset noise, to obtain a noisy image of the target first sample image; Inputting the noisy image of the target first sample image into a diffusion model to obtain a predicted noisy noise output by the diffusion model; Performing noise adding processing on the target second sample image according to a second noise adding step number to obtain a noisy image of the target second sample image; Inputting the noisy image of the target second sample image into the diffusion model to obtain the predicted denoised noise output by the diffusion model; Obtaining a denoised image of the target second sample image according to the denoised image of the second sample image and the predicted denoised noise; The diffusion model is trained according to the style loss between the denoised image of the target second sample image and the plurality of first sample images, and the diffusion loss between the predicted denoising noise and the preset noise.

3. The image generation method according to claim 2, characterized in that: The step of inputting the noisy image of the target second sample image into the diffusion model to obtain the predicted denoised noise output by the diffusion model comprises: Inputting the noisy image of the target second sample image into an encoder in the diffusion model to obtain a feature image of the noisy image of the target second sample image; When the second noise adding step number is greater than or equal to a preset step number threshold, fusing the feature image of the noise adding image of the target second sample image with the feature image of the target second sample image output by the encoder to obtain a fused feature image; The fused feature image is input into the decoder of the diffusion model and output to obtain the predicted denoised noise output by the diffusion model.

4. The image generation method according to claim 2, characterized in that: The step of inputting the noisy image of the target second sample image into the diffusion model to obtain the predicted denoised noise output by the diffusion model comprises: Inputting the noisy image of the target second sample image into an encoder in the diffusion model to obtain a feature image of the noisy image of the target second sample image; When the second noisy step number is less than a preset step number threshold, fusing the feature image of the noisy image of the target second sample image with the feature image of the Gaussian noise output by the encoder to obtain a fused feature image; The fused feature image is input into the decoder of the diffusion model and output to obtain the predicted denoised noise output by the diffusion model.

5. The image generation method according to claim 2, characterized in that: The method further comprises: The first noise adding step number is obtained by sampling from the first step number interval.

6. The image generation method according to claim 2, characterized in that: The method further comprises: The second noise adding step number is obtained by sampling from the second step number interval.

7. The image generation method according to claim 3, characterized in that: The step of training the diffusion model according to the style loss between the denoised image of the target second sample image and the plurality of first sample images, and the diffusion loss between the predicted noise-added noise and the preset noise, comprises: Based on the first noise adding step number, obtaining the weight of the diffusion loss, wherein the weight of the diffusion loss is negatively correlated with the first noise adding step number; When the second number of steps for adding noise is greater than or equal to a preset number of steps threshold, obtaining a weight of the style loss based on the second number of steps for adding noise, wherein the weight of the style loss is negatively correlated with the second number of steps for adding noise; Based on the weight of the diffusion loss and the weight of the style loss, performing weighted summation on the diffusion loss and the style loss to obtain a total loss of the diffusion model; The diffusion model is trained with the reduction of the total loss as a training goal to obtain the trained diffusion model.

8. The image generation method according to claim 7, characterized in that: The method further comprises: When the second denoising step number is greater than or equal to a preset step number threshold, determining a cross-domain direction consistency loss between a direction vector of the denoised image of the target second sample image and a reference direction vector of the denoised image of the target second sample image; Determine a weight of the cross-domain direction consistency loss, where the weight of the cross-domain direction consistency loss is a preset value; Based on the weight of the diffusion loss and the weight of the style loss, weighted summing the diffusion loss and the style loss to obtain the total loss of the diffusion model includes: Based on the weight of the diffusion loss, the weight of the style loss and the weight of the cross-domain directional consistency loss, a weighted sum is performed on the diffusion loss, the style loss and the cross-domain directional consistency loss to obtain a total loss of the diffusion model.

9. The image generation method according to claim 8, characterized in that: The determining of the cross-domain direction consistency loss between the direction vector of the denoised image of the target second sample image and the reference direction vector of the denoised image of the target second sample image comprises: Performing feature extraction on the plurality of second sample images to obtain an average direction vector of the plurality of second sample images; Performing feature extraction on the plurality of first sample images to obtain an average direction vector of the plurality of first sample images; Based on the average direction vector of the plurality of second sample images and the average direction vector of the plurality of first sample images, acquiring a cross-domain direction vector between the plurality of second sample images and the plurality of first sample images; For any second sample image in the target second sample images, add the direction vector of the any second sample image to the cross-domain direction vector to obtain a reference direction vector of a denoised image of the any second sample image; The cross-domain direction consistency loss is obtained based on the direction vectors of the denoised images corresponding to the sampled second sample images and the reference direction vectors of the corresponding denoised images.

10. An image processing device, characterized in that: The device calls a diffusion model, including: An acquisition module, used for acquiring the original image; A processing module, used for performing noise processing on the original image to obtain a noisy image of the original image; The processing module is further used to input the noisy image of the original image into the trained diffusion model to obtain the denoised noise output by the trained diffusion model, wherein the trained diffusion model is obtained by training using a plurality of first sample images, a plurality of second sample images, a diffusion loss and a style loss, the image styles of the plurality of second sample images are inconsistent with the image styles of the plurality of first sample images, the diffusion loss is used to characterize the difference between the predicted denoised noise output by the diffusion model based on the noisy image of the input target first sample image and the preset noise used to obtain the noisy image of the target first sample image, the style loss is used to characterize the style difference between the denoised image of the target second sample image and the plurality of first sample images, the denoised image of the target second sample image is obtained by the diffusion model based on the noisy image of the input target second sample image and the predicted denoised noise output by the diffusion model, the target first sample image is one of the plurality of first sample images, and the target second sample image is one of the plurality of second sample images; The processing module is further used to perform denoising processing based on the noisy image of the original image and the denoised noise to obtain a target image.

11. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

12. A computer device, characterized in that: The invention comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the method according to any one of claims 1 to 9.

Images