Image processing method and device, electronic equipment, storage medium and program product
By using a noise prediction model and residual calculations, combined with weight parameters, the problem of controlling frequency information in image generation by diffusion models was solved, thus achieving high-quality visual generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA MOBILE JIUTIAN ARTIFICIAL INTELLIGENCE TECHNOLOGY (BEIJING) CO LTD
- Filing Date
- 2026-02-04
- Publication Date
- 2026-06-05
AI Technical Summary
The diffusion model has difficulty effectively controlling the frequency information of the data during image generation, resulting in low visual quality.
A noise prediction model is adopted, which uses an image prediction sub-network and residual form to predict the denoised image and combine it with weight parameters to achieve a deep correlation between the noise prediction result and the image features, thus preserving different frequency information of the image.
Effectively control the frequency information of the image, avoid detail loss and structural distortion during denoising using lightweight models, and improve the quality of visual generation.
Smart Images
Figure CN122155985A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of visual generation technology, and in particular to an image processing method, apparatus, electronic device, storage medium, and program product. Background Technology
[0002] Currently, diffusion models are commonly used for image and video generation. When optimizing diffusion models, parameterized noise prediction methods can be employed. These methods involve predicting noise in the image instead of predicting the original image during denoising. However, this approach makes it difficult for diffusion models to control the frequency information of the data, potentially leading to lower visual quality in the generated images. Summary of the Invention
[0003] This application provides an image processing method, apparatus, electronic device, storage medium, and program product to solve the technical problem that diffusion models have difficulty controlling the frequency information of data.
[0004] In a first aspect, embodiments of this application provide an image processing method, including: A first image is acquired; the first image represents a noise image at a first moment during the inference process of the diffusion model; the first image is input into a noise prediction model for noise prediction processing to obtain a first noise prediction result for the first image; wherein, the noise prediction model includes an image prediction sub-network, which is used to perform image prediction processing on the first image to obtain a first image prediction result, and the first noise prediction result is a weighted fusion of the first image and the first image prediction result; based on the first image and the first noise prediction result, a second image is determined, the second image represents an image at a second moment during the inference process of the diffusion model, and the second moment is the moment following the first moment.
[0005] In one embodiment, the noise prediction model further includes a first weight parameter and a second weight parameter; the first noise prediction result is the sum of a first product and a second product, wherein the first product is the product of the first image and the first weight parameter, and the second product is the product of the first image prediction result and the second weight parameter.
[0006] In one embodiment, determining a second image based on a first image and a first noise prediction result includes: Based on the first image, the first noise prediction result, the first predetermined parameter at the first time, and the second predetermined parameter at the first time, the image derivation result is determined. The image derivation result is the difference between the third product and the fourth product. The third product is the product of the first image and the first predetermined parameter, and the fourth product is the product of the first noise prediction result and the second predetermined parameter. Based on the first image and the image derivation results, the second image is determined.
[0007] In one embodiment, the training process for the noise prediction model includes: Obtain sample images; the sample images are obtained by adding Gaussian noise to the original images; The sample image is input into the initial noise prediction model for processing to obtain the initial noise prediction result; the initial noise prediction model includes a first initial parameter, an initial image prediction sub-network, and a second initial parameter. Based on the Gaussian noise added to the sample image and the initial noise prediction results, the first initial parameters, the initial image prediction sub-network, and the second initial parameters are updated to obtain the noise prediction model.
[0008] In one embodiment, the training process of the noise prediction model further includes: Obtain the initial image prediction results of the initial image prediction subnetwork for the sample image; The initial image prediction subnetwork is updated based on the initial image prediction results and the original image.
[0009] In one embodiment, the first weight parameter and the second weight parameter are floating-point parameters.
[0010] Secondly, embodiments of this application provide an image processing apparatus, comprising: The acquisition module is used to acquire the first image; the first image represents the noise image at the first moment during the diffusion model inference process. The processing module is used to input the first image into the noise prediction model for noise prediction processing to obtain the first noise prediction result of the first image; wherein, the noise prediction model includes an image prediction sub-network, which is used to perform image prediction processing on the first image to obtain the first image prediction result, and the first noise prediction result is a weighted fusion of the first image and the first image prediction result; The processing module is also used to determine a second image based on the first image and the first noise prediction result. The second image is used to represent the image at a second time point during the diffusion model inference process. The second time point is the time point following the first time point.
[0011] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the steps of the image processing method described in the first aspect.
[0012] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the image processing method described in the first aspect.
[0013] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the image processing method described in the first aspect.
[0014] The image processing method, apparatus, electronic device, storage medium, and program product provided in this application can predict the denoised image through the image prediction sub-network of the noise prediction model, and then accurately obtain the noise prediction result through residual calculation. This achieves a deep correlation between the features of the noise prediction result and the first image, effectively preserving different frequency information of the first image. This supports the diffusion model in effectively controlling the frequency information of the image, avoiding the loss of details and structural distortion during denoising by the lightweight model, and improving the quality of visual generation. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a schematic flowchart of the image processing method provided in the embodiments of this application.
[0017] Figure 2 This is a schematic diagram of the model structure provided in the embodiments of this application.
[0018] Figure 3 This is a schematic diagram of the training data provided in the embodiments of this application.
[0019] Figure 4 This is a schematic diagram of the experimental results provided in the embodiments of this application.
[0020] Figure 5 This is a schematic flowchart of the model training method provided in the embodiments of this application.
[0021] Figure 6 This is a schematic diagram of the structure of the image processing apparatus provided in the embodiments of this application.
[0022] Figure 7 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0024] Figure 1 This is a schematic flowchart illustrating the image processing method provided in an embodiment of this application. (Refer to...) Figure 1 This application provides an image processing method, which may include: Step 110: Obtain the first image.
[0025] The first image represents the noise image at the first moment during the inference process of the diffusion model. This noise image may also be called a latent variable or other names, without restriction.
[0026] The diffusion model reasoning process refers to the process from Begin gradually denoising until you get... This completes the mapping of image data distribution. Among them, It is random Gaussian noise, i.e., the noise image at time T. This is an image generated after the diffusion model inference process is completed.
[0027] For example, the first time point could be time T, meaning the first image could be... For example, the first time point can be time t, meaning the first image can be... Time t refers to a time that occurs after time T and is obtained from time t. The previous moment. Right now and The noise image between them.
[0028] Step 120: Input the first image into the noise prediction model for noise prediction processing to obtain the first noise prediction result of the first image.
[0029] The noise prediction model includes an image prediction sub-network. This sub-network performs image prediction processing on the first image to obtain a prediction result. The first image prediction result refers to the prediction result of the first image after noise removal. In other words, the image prediction sub-network is used to predict the first image after noise removal.
[0030] The first noise prediction result is a weighted fusion of the first image and the prediction results for the first image. The first noise prediction result refers to the prediction result for the noise present in the first image.
[0031] Specifically, the noise prediction model can perform image prediction on the input first image through the image prediction sub-network to obtain the first image prediction result, and further perform weighted fusion of the first image and the first image prediction result to obtain the first noise prediction result.
[0032] Based on this, the image prediction sub-network of the noise prediction model can be used to predict the denoised image first, and then the noise prediction result can be accurately obtained by calculating the residual form. This achieves a deep correlation between the noise prediction result and the features of the first image, effectively preserving the different frequency information of the first image. This supports the diffusion model in effectively controlling the frequency information of the image and avoids the loss of details and structural distortion when the lightweight model is denoised.
[0033] Step 130: Determine the second image based on the first image and the first noise prediction result.
[0034] The second image represents the image at the second time step during the diffusion model inference process. The second time step is the time step following the first time step.
[0035] Furthermore, the second image can be a noisy image or an image generated after the diffusion model inference process is completed.
[0036] For example, if the first time point is time T, then the second time point, which is time T-1, could be the second image. In this case, the second image is a noisy image.
[0037] For example, if the first time point is time t, then the second time point, i.e., time t-1, could be the second image. If t-1 > 0, then the second image is a noisy image. If t-1 = 0, then the second image is... That is, the image generated after the diffusion model inference process is completed.
[0038] Specifically, in one embodiment, when determining the second image based on the first image and the first noise prediction result, the image derivation result of the first image can be determined firstly based on the first image, the first noise prediction result, the first predetermined parameter at the first time, and the second predetermined parameter at the first time. Further, the second image can be determined based on the first image and the image derivation result.
[0039] The image derivation result is the difference between the third product and the fourth product. The third product is the product of the first image and the first predetermined parameters, and the fourth product is the product of the first noise prediction result and the second predetermined parameters. The image derivation result can be expressed by the following first formula: .
[0040] in, The result is derived from the image. This is the first image. For the first moment. This is the first noise prediction result. This is the first predetermined parameter for the first moment. This is the second predetermined parameter for the first moment. This is a real number used to control the magnitude of the inverse denoising at the first time step during the diffusion model inference process. For example, You can refer to the corresponding parameter settings in the Denoising Diffusion Probabilistic Models (DDPM).
[0041] Furthermore, based on the first image and the image derivation results, the second image can be determined using the following second formula: .
[0042] in, This is the second image. It's the second moment. This is the first image. For the first moment. The above image is the derivation result. It is standard Gaussian noise. , and All coefficients are derived from maximum likelihood estimation and probability theory.
[0043] Based on this, the image derivation result can be obtained through predetermined parameters, and then combined with the original first image to optimize the denoising calculation, reducing the information loss in the denoising process. This ensures that each iteration of the reverse denoising has clear parameter constraints, guaranteeing the stability and consistency of image restoration during the diffusion model inference process, and effectively preserving the frequency information of the image.
[0044] Furthermore, the second formula mentioned above is also the formula used to implement the inference process of the diffusion model. (In related technologies...) It is obtained directly from the image prediction neural network, that is, the first image is directly input into the image prediction neural network for prediction processing, and thus... This approach easily loses frequency information from the first image. In contrast, this application first obtains the first noise prediction result using a noise prediction model in a residual manner, effectively preserving the frequency information in the first image, and then further derives... It can enable the diffusion model to effectively control the frequency information of the image.
[0045] In one embodiment, the noise prediction model further includes a first weight parameter and a second weight parameter. The first noise prediction result is the sum of a first product and a second product. The first product is the product of a first image and the first weight parameter. The second product is the product of the first image prediction result and the second weight parameter.
[0046] Based on this, a weighted fusion logic for noise prediction results can be constructed by introducing dual-weight parameters, balancing the noisy features of the first image with the denoised features of the image prediction results, effectively preserving the frequency information of the image. Furthermore, the network structure of the noise prediction model is relatively simple, ensuring both noise prediction performance and lightweight design with efficient inference.
[0047] In one embodiment, the first weight parameter and the second weight parameter are floating-point parameters, which support accurate noise prediction through training.
[0048] In one possible example, such as Figure 2 The diagram shown is a schematic representation of the model structure provided in an embodiment of this application. Related technologies typically employ methods such as... Figure 2 (a) shows a method that directly uses an image prediction neural network to predict noisy images to obtain image prediction results, or uses methods such as... Figure 2 (b) shows the method of obtaining noise prediction results by directly predicting the noise image using a noise prediction neural network.
[0049] However, noise is a chaotic high-frequency signal, making it difficult to directly predict noise information using neural networks. Nevertheless, methods for predicting noise often perform better than methods for predicting images. Based on this, the parameterization method proposed in this application modifies the image prediction method using residuals to achieve the following: Figure 2 (c) shows a method for predicting noise in residual form, which involves weighted fusion of the input noisy image and the output of the image prediction subnetwork to obtain a noise prediction result in residual form. That is, the first image prediction result is obtained by predicting the image through the image prediction subnetwork, and then the product of the first weight parameter and the first image, and the product of the second weight parameter and the first image prediction result are added together to obtain a noise prediction result in residual form.
[0050] Furthermore, the theoretical basis for the residual modification of the predicted image in the embodiments of this application is introduced below. According to the principle of the diffusion model, the diffusion process is expressed by the following third formula: .
[0051] in, The noise image at time t (i.e., the latent variable). This is the original image. It is standard Gaussian noise. This can be considered a real number, used to control the positive noise addition amplitude at time t during the diffusion process in the diffusion model. For details, please refer to the relevant introduction to DDPM; further explanation is omitted here. A simple transformation of the third formula yields the following fourth formula: .
[0052] The fourth formula shows that the standard Gaussian noise added to the original image during the diffusion process of the diffusion model... , is the latent variable after adding noise. With the original image Obtained by linear combination.
[0053] Furthermore, the predicted noise and the predicted image can also have the same relationship as shown in the fifth formula: .
[0054] in, For noise prediction neural network pairs The noise prediction results obtained from the prediction. Image prediction neural network pairs The image prediction result obtained from the prediction.
[0055] therefore, Right now and The linear combination relationship is established. In this embodiment, the coefficients of the linear combination are represented by the aforementioned learnable first and second weight parameters, thereby realizing a noise prediction model modified in the form of residuals.
[0056] Based on experimental results, the following section describes the shortcomings of diffusion models based on noise prediction neural networks in capturing frequency information, and how the diffusion model based on the noise prediction model provided in the embodiments of this application can overcome these shortcomings.
[0057] Figure 3 A schematic diagram of the training data provided in the embodiments of this application. Figure 3 The experiment shows several sinusoidal signals of different frequencies used as training data. Figure 3 The horizontal axis represents time, and the vertical axis represents amplitude. Furthermore, the number of sampling points is set to 64, and the frequency follows a uniform distribution from 1 to 3. The training objective is to enable the diffusion model to generate sinusoidal signals.
[0058] To demonstrate that the noise prediction model provided in this application embodiment can also achieve good results in a lightweight model, the diffusion model was trained using a 4-layer perceptron in the experiment. Figure 4 A schematic diagram illustrating the experimental results provided in the embodiments of this application. Figure 4The diagrams show a sinusoidal signal, a first signal generated by training a diffusion model based on the noise prediction model provided in this application, and a second signal generated by training a diffusion model based on a noise prediction neural network. The noise prediction neural network can be configured with reference to DDPM.
[0059] first, Figure 4 The upper part of the diagram, with time on the horizontal axis and amplitude on the vertical axis, displays the time-domain waveforms of a standard sinusoidal signal, the first signal, and the second signal. To accurately analyze the mechanism of signal generation, Fourier transforms were performed on these signals, and the results are shown below. Figure 4 The lower half is shown. Figure 4 In the lower half of the graph, the horizontal axis represents frequency, and the vertical axis represents spectral amplitude. A standard sinusoidal signal has two frequency peaks, one positive and one negative. The second signal, however, has multiple frequency peaks. This is because the noise prediction neural network struggles to grasp the frequency information of the noise when predicting it, resulting in multiple frequency peaks. In contrast, the first signal, like the sinusoidal signal, has two frequency peaks. This indicates that the diffusion model based on the noise prediction model provided in this application can accurately learn the frequency characteristics of the signal.
[0060] Therefore, compared with the parameterized method of predicting noise based on a noise prediction neural network, the noise prediction model modified in the form of residuals provided in this application embodiment can accurately learn the frequency information of the data.
[0061] Furthermore, compared to inference acceleration methods such as Denoising Diffusion Implicit Models (DDIM) and Diffusion Probabilistic Model Solver (DPM-Solver), the method provided in this application can optimize the lightweight model by modifying the model structure in the form of residuals.
[0062] In addition, the network modification method provided in this application is based on the mechanism of the diffusion model. The residual part represents the noisy image, and the backbone network part represents the original image. The additive relationship between these is closely consistent with the relationship between the latent space variables and noise in the diffusion model, and has strong interpretability.
[0063] Figure 5 This is a schematic flowchart illustrating the model training method provided in an embodiment of this application. (Refer to...) Figure 5 The training process for a noise prediction model may include: Step 510: Obtain sample images.
[0064] The sample images are obtained by adding Gaussian noise to the original images, that is, the noisy images obtained by adding Gaussian noise to the original images during the training of the diffusion model.
[0065] During the training of the diffusion model, Gaussian noise is progressively added to the original images used for training. This results in multiple images contaminated with varying degrees of Gaussian noise. For example, 1000 steps of Gaussian noise can be added. The noisy image obtained after adding t steps of Gaussian noise is the noisy image at time t. The sample images can be images contaminated with noise at any time during the training of the diffusion model.
[0066] These noise-contaminated images are also called latent variables. Specifically, during the noise addition process, it is ensured that the images contaminated for 1000 steps are pure Gaussian noise. The training objective of the diffusion model is to use a neural network to reconstruct the original images from these noise-contaminated images.
[0067] Step 520: Input the sample image into the initial noise prediction model for processing to obtain the initial noise prediction result.
[0068] The initial noise prediction model includes a first initial parameter, an initial image prediction subnetwork, and a second initial parameter.
[0069] Step 530: Based on the Gaussian noise added to the sample image and the initial noise prediction result, update the first initial parameters, the initial image prediction sub-network, and the second initial parameters to obtain the noise prediction model.
[0070] Specifically, in one embodiment, a noise loss value can be calculated based on the Gaussian noise added to the sample image and the initial noise prediction result. Further, the first initial parameters, the initial image prediction sub-network, and the second initial parameters can be updated based on the noise loss value until a noise prediction model that meets the requirements is obtained.
[0071] Based on this, the first and second initial parameters of the noise prediction model can be integrated with the image prediction sub-network into a unified optimization loop, enabling coordinated parameter updates throughout the noise prediction process. This avoids disconnections in certain optimization steps and improves the accuracy of noise prediction. Furthermore, the convergence efficiency of the diffusion model's diffusion process can be improved by parameterizing the predicted noise in the form of residuals.
[0072] In one embodiment, during the training of the noise prediction model, the initial image prediction result of the initial image prediction sub-network for the sample image can be obtained, and an image loss value can be calculated based on the initial image prediction result and the original image. For example, the output of the initial image prediction sub-network can be compared with the original image to calculate the L2 loss. Furthermore, the parameters of the initial image prediction sub-network can be updated based on the image loss value. In this way, the ability of the noise prediction model to restore image features can be further improved, thereby improving the quality of visual generation.
[0073] The image processing apparatus provided in the embodiments of this application will be described below. The image processing apparatus described below can be referred to in correspondence with the image processing method described above.
[0074] Figure 6 This is a schematic flowchart of an image processing apparatus provided in an embodiment of this application. (Refer to...) Figure 6 The image processing apparatus may include: The acquisition module 610 is used to acquire the first image. The first image represents the noise image at the first moment during the diffusion model inference process.
[0075] The processing module 620 is used to input the first image into the noise prediction model for noise prediction processing to obtain a first noise prediction result for the first image. The noise prediction model includes an image prediction sub-network, which performs image prediction processing on the first image to obtain a first image prediction result. The first noise prediction result is a weighted fusion of the first image and the first image prediction result.
[0076] The processing module 620 is also used to determine a second image based on the first image and the first noise prediction result. The second image is used to represent the image at a second moment in the diffusion model inference process. The second moment is the moment after the first moment.
[0077] In one embodiment, the noise prediction model further includes a first weight parameter and a second weight parameter; the first noise prediction result is the sum of a first product and a second product, wherein the first product is the product of the first image and the first weight parameter, and the second product is the product of the first image prediction result and the second weight parameter.
[0078] In one embodiment, the processing module 620 is specifically used for: Based on the first image, the first noise prediction result, the first predetermined parameter at the first time, and the second predetermined parameter at the first time, the image derivation result is determined. The image derivation result is the difference between the third product and the fourth product. The third product is the product of the first image and the first predetermined parameter, and the fourth product is the product of the first noise prediction result and the second predetermined parameter. Based on the first image and the image derivation results, the second image is determined.
[0079] In one embodiment, the training process for the noise prediction model includes: Obtain sample images; the sample images are obtained by adding Gaussian noise to the original images; The sample image is input into the initial noise prediction model for processing to obtain the initial noise prediction result; the initial noise prediction model includes a first initial parameter, an initial image prediction sub-network, and a second initial parameter. Based on the Gaussian noise added to the sample image and the initial noise prediction results, the first initial parameters, the initial image prediction sub-network, and the second initial parameters are updated to obtain the noise prediction model.
[0080] In one embodiment, the training process of the noise prediction model further includes: Obtain the initial image prediction results of the initial image prediction subnetwork for the sample image; The initial image prediction subnetwork is updated based on the initial image prediction results and the original image.
[0081] In one embodiment, the first weight parameter and the second weight parameter are floating-point parameters.
[0082] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call a computer program stored in the memory 730 to execute the steps of the XXXX method, such as including: A first image is acquired. This first image represents the noise image at the first time step during the diffusion model inference process. The first image is input into a noise prediction model for noise prediction processing, yielding a first noise prediction result for the first image. The noise prediction model includes an image prediction sub-network, which performs image prediction processing on the first image to obtain a first image prediction result. The first noise prediction result is a weighted fusion of the first image and the first image prediction result. Based on the first image and the first noise prediction result, a second image is determined. This second image represents the image at the second time step during the diffusion model inference process; the second time step is the time step following the first time step.
[0083] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0084] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the image processing methods provided in the above embodiments, such as including: A first image is acquired. This first image represents the noise image at the first time step during the diffusion model inference process. The first image is input into a noise prediction model for noise prediction processing, yielding a first noise prediction result for the first image. The noise prediction model includes an image prediction sub-network, which performs image prediction processing on the first image to obtain a first image prediction result. The first noise prediction result is a weighted fusion of the first image and the first image prediction result. Based on the first image and the first noise prediction result, a second image is determined. This second image represents the image at the second time step during the diffusion model inference process; the second time step is the time step following the first time step.
[0085] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the methods provided in the above embodiments, such as including: A first image is acquired. This first image represents the noise image at the first time step during the diffusion model inference process. The first image is input into a noise prediction model for noise prediction processing, yielding a first noise prediction result for the first image. The noise prediction model includes an image prediction sub-network, which performs image prediction processing on the first image to obtain a first image prediction result. The first noise prediction result is a weighted fusion of the first image and the first image prediction result. Based on the first image and the first noise prediction result, a second image is determined. This second image represents the image at the second time step during the diffusion model inference process; the second time step is the time step following the first time step.
[0086] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).
[0087] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0088] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0089] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. An image processing method, characterized in that, include: Get the first image; The first image is used to represent the noise image at the first moment during the inference process of the diffusion model; The first image is input into a noise prediction model for noise prediction processing to obtain a first noise prediction result for the first image; wherein, the noise prediction model includes an image prediction sub-network, which is used to perform image prediction processing on the first image to obtain a first image prediction result, and the first noise prediction result is a weighted fusion of the first image and the first image prediction result; Based on the first image and the first noise prediction result, a second image is determined. The second image is used to represent the image at a second moment during the inference process of the diffusion model, and the second moment is the moment following the first moment.
2. The image processing method according to claim 1, characterized in that, The noise prediction model further includes a first weight parameter and a second weight parameter; the first noise prediction result is the sum of a first product and a second product, wherein the first product is the product of the first image and the first weight parameter, and the second product is the product of the first image prediction result and the second weight parameter.
3. The image processing method according to claim 2, characterized in that, Determining the second image based on the first image and the first noise prediction result includes: Based on the first image, the first noise prediction result, the first predetermined parameter at the first time, and the second predetermined parameter at the first time, an image derivation result is determined. The image derivation result is the difference between the third product and the fourth product. The third product is the product of the first image and the first predetermined parameter, and the fourth product is the product of the first noise prediction result and the second predetermined parameter. The second image is determined based on the first image and the image derivation result.
4. The image processing method according to claim 2, characterized in that, The training process of the noise prediction model includes: Obtain a sample image; the sample image is obtained by adding Gaussian noise to the original image; The sample image is input into the initial noise prediction model for processing to obtain the initial noise prediction result; the initial noise prediction model includes a first initial parameter, an initial image prediction sub-network, and a second initial parameter. Based on the Gaussian noise added to the sample image and the initial noise prediction result, the first initial parameter, the initial image prediction sub-network, and the second initial parameter are updated to obtain the noise prediction model.
5. The image processing method according to claim 4, characterized in that, The training process of the noise prediction model also includes: Obtain the initial image prediction results of the initial image prediction subnetwork for the sample image; The initial image prediction subnetwork is updated based on the initial image prediction result and the original image.
6. The image processing method according to claim 2, characterized in that, The first weight parameter and the second weight parameter are floating-point parameters.
7. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire the first image; The first image is used to represent the noise image at the first moment during the inference process of the diffusion model; The processing module is used to input the first image into a noise prediction model for noise prediction processing to obtain a first noise prediction result of the first image; wherein, the noise prediction model includes an image prediction sub-network, the image prediction sub-network is used to perform image prediction processing on the first image to obtain a first image prediction result, and the first noise prediction result is a weighted fusion of the first image and the first image prediction result. The processing module is further configured to determine a second image based on the first image and the first noise prediction result, the second image representing an image at a second moment during the diffusion model inference process, the second moment being the moment following the first moment.
8. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the image processing method according to any one of claims 1 to 6.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 6.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the image processing method according to any one of claims 1 to 6.