A data processing method and apparatus

Abstract

A data processing method, applied in the field of artificial intelligence, includes: acquiring noisy data; performing denoising on the noisy data in multiple steps sequentially using a first diffusion model to obtain a denoising result corresponding to each step, wherein the first step is either the first step or an intermediate step; using the first diffusion model and a second diffusion model to predict noise in the denoised result corresponding to the first step, respectively, to obtain a first prediction result and a second prediction result; the maximum number of denoising steps of the first diffusion model is less than the maximum number of denoising steps of the second diffusion model; and updating the first diffusion model based on the difference between the first prediction result and the second prediction result. This application can improve the training efficiency of student models, reduce training time overhead, and enable student models to have better model performance.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a data processing method and apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] Diffusion models, as a novel generative model, have attracted increasing attention due to their ability to model complex data distributions and are gradually replacing generative adversarial models in various downstream tasks. However, diffusion models require tens or even hundreds of iterative samplings during inference to refine the model's output, resulting in high latency and computational costs, posing a significant challenge to the model's deployment and application. To address this issue, existing methods compress the sampling path of diffusion models by redesigning the diffusion process, using high-order numerical solvers, and employing knowledge distillation techniques.

[0004] Among these methods, knowledge distillation is currently the mainstream approach to accelerate diffusion model sampling because it has the potential to accelerate sampling efficiency while maintaining model performance. Furthermore, this technique can be generalized to different diffusion models, exhibiting strong generalization ability. However, in existing knowledge distillation techniques for diffusion models, the student model can process the noisy data, and both the teacher and student models need to process the noisy result obtained by the student model and align the outputs of the teacher and student models. However, this approach struggles to fully transfer the teacher model's prior knowledge to the student model. This results in lower quality output data for the distilled student model, affecting its performance, and also leads to low efficiency in the distillation process, resulting in significant training time overhead. Summary of the Invention

[0005] This application provides a data processing method to improve the training efficiency of diffusion models.

[0006] In a first aspect, this application provides a data processing method, the method comprising: acquiring noisy data; performing denoising on the noisy data in multiple steps sequentially using a first diffusion model to obtain a denoising result corresponding to each step, wherein the first step is the first step or an intermediate step among the multiple steps; performing noise prediction on the denoising result corresponding to the first step using a second diffusion model and a third diffusion model respectively to obtain a first prediction result and a second prediction result; wherein the maximum number of denoising steps of the first diffusion model is less than the maximum number of denoising steps of the second diffusion model; and updating the first diffusion model and the third diffusion model according to the difference between the first prediction result and the second prediction result.

[0007] The approach of this application involves adding noise to the output of the student model (corresponding to the first diffusion model) in a multi-step denoising process, specifically in the steps preceding the last step (i.e., the first or intermediate steps). Noise prediction is then performed on the noise-added results using a model corresponding to the student model (corresponding to the third diffusion model) and a teacher model (corresponding to the second diffusion model). Knowledge distillation is achieved by aligning the noise prediction results of the student and teacher models. This application aligns the denoising prediction information of the student model and teacher model for the output of the student model in the steps preceding the last step. This allows the student model to learn the denoising path used by the teacher model in generating the data, improving the supervision signal during training, making the learning process easier, increasing the training efficiency of the student model, reducing training time overhead, and enabling the student model to achieve superior model performance.

[0008] Furthermore, the knowledge distillation process described above only needs to process the data generated by the student model, without needing to process the labeled training data, thus improving the flexibility and usability of the solution.

[0009] The updating of the first diffusion model and the third diffusion model can be done alternately. For example, the first diffusion model is updated in one iteration, and the third diffusion model is updated in the next iteration.

[0010] The noise addition process involves adding noise in a certain number of steps. The number of steps determines the degree of noise addition. The teacher model often has a large number of denoising steps. In order to ensure that the student model can learn the noise prediction ability of the teacher model in the middle steps, during the knowledge distillation process, the number of steps in the noise addition process for the denoising result corresponding to the first step can be randomly selected from the number of denoising steps of the teacher model.

[0011] In one possible implementation, the second step can be a step following the first step among multiple steps. The second step can be an intermediate step or the last step. A third diffusion model is used to predict noise in the denoised result corresponding to the second step, resulting in a third prediction result. Similarly, the second diffusion model is used to predict noise in the denoised result corresponding to the second step, resulting in a fourth prediction result. The first and third diffusion models are updated based on the difference between the third and fourth prediction results. Specifically, the denoised result corresponding to the first step is obtained after the third step, and the denoised result corresponding to the second step is obtained after the fourth step, with the fourth step having a lower level of noise addition than the third step. This method ensures that when adding noise to the output of the student model at different steps, the later the step, the fewer the steps (i.e., the lower the level of noise addition).

[0012] In one possible implementation, the third step is selected from the first numerical interval, and the fourth step is selected from the second numerical interval. The first and second numerical intervals do not intersect, and the values ​​in the second numerical interval are less than the values ​​in the first numerical interval.

[0013] In one possible implementation, the values ​​included in the first numerical range are no greater than the maximum number of denoising steps of the second diffusion model.

[0014] By using the above method, the noise added to the output of the student model at different steps is equivalent to decoupling the learning objectives with different sampling steps, so that the student model can support both high-quality deterministic sampling and flexible adjustment of the number of steps.

[0015] In one possible implementation, the method further includes: acquiring input data, which is an image and / or text; performing multiple steps of denoising on the noisy data sequentially through a first diffusion model to obtain a denoising result corresponding to each step, including: using the input data as a conditional input to the first diffusion model, performing multiple steps of denoising on the noisy data sequentially to obtain a denoising result corresponding to each step; the denoising result is an image.

[0016] In one possible implementation, the noise data is image data.

[0017] Secondly, this application provides a data processing method, the method comprising: acquiring first data and noisy data; using the first data as a conditional input to an updated first diffusion model obtained by the method of the first aspect, and performing denoising on the noisy data in multiple steps to obtain second data.

[0018] In one possible implementation, the first data is text and / or an image; the second data is an image generated based on either text or an image.

[0019] Thirdly, this application provides a data processing apparatus, the apparatus comprising:

[0020] The acquisition module is used to acquire noise data;

[0021] The denoising module is used to perform multiple steps of denoising on the noisy data sequentially through the first diffusion model to obtain the denoising result corresponding to each step, wherein the first step is either the first step or an intermediate step;

[0022] The noise prediction module is used to predict the noise addition result of the denoising result corresponding to the first step number using the second diffusion model and the third diffusion model, respectively, to obtain the first prediction result and the second prediction result; the maximum number of denoising steps of the first diffusion model is less than the maximum number of denoising steps of the second diffusion model.

[0023] The update module is used to update the first diffusion model and the third diffusion model based on the difference between the first prediction result and the second prediction result.

[0024] In one possible implementation, the second step is one of a plurality of steps following the first step. The noise prediction module is also used for:

[0025] The noise prediction is performed on the noise addition result of the denoising result corresponding to the second step using the third diffusion model, and the third prediction result is obtained.

[0026] Using the second diffusion model, noise prediction is performed on the noise addition result of the denoising result corresponding to the second step to obtain the fourth prediction result;

[0027] The update module is also used to update the first diffusion model and the third diffusion model based on the difference between the third and fourth prediction results; wherein,

[0028] The noise addition result corresponding to the first step is obtained after the noise addition in the third step, and the noise addition result corresponding to the second step is obtained after the noise addition in the fourth step. The noise addition level in the fourth step is lower than that in the third step.

[0029] In one possible implementation, the third step is selected from the first numerical interval, and the fourth step is selected from the second numerical interval. The first and second numerical intervals do not intersect, and the values ​​in the second numerical interval are less than the values ​​in the first numerical interval.

[0030] In one possible implementation, the values ​​in the first numerical range are no greater than the maximum number of denoising steps of the second diffusion model.

[0031] In one possible implementation, the acquisition module is also used to acquire input data, which is an image and / or text;

[0032] The noise reduction module is specifically used for:

[0033] The input data is used as the conditional input of the first diffusion model, and the noise data is denoised in multiple steps in sequence to obtain the denoising result corresponding to each step; the denoising result is an image.

[0034] In one possible implementation, the noise data is image data.

[0035] Fourthly, this application provides a data processing apparatus, the apparatus comprising:

[0036] The acquisition module is used to acquire the first data and the noisy data;

[0037] The generation module is used to take the first data as a conditional input to the updated first diffusion model obtained by the method of the first aspect, and perform denoising on the noisy data in multiple steps to obtain the second data.

[0038] In one possible implementation, the first data is text and / or an image; the second data is an image generated based on either text or an image.

[0039] Fifthly, embodiments of this application provide a computing device that may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect and any optional methods thereunder, as well as the methods described in the second aspect and any optional methods thereunder.

[0040] In a sixth aspect, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any optional methods thereof, as well as the methods described in the second aspect and any optional methods thereof.

[0041] In a seventh aspect, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any optional methods thereof, as well as the methods described in the second aspect and any optional methods thereof.

[0042] Eighthly, this application provides a chip system including a processor for executing data processing devices to realize the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the application system framework of the present invention;

[0044] Figure 2 This is a schematic diagram of a system architecture according to this application;

[0045] Figure 3 A process for providing a cloud service;

[0046] Figure 4 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0047] Figure 5 This application provides an example of an application architecture.

[0048] Figure 6 A schematic diagram of a data processing method provided in an embodiment of this application;

[0049] Figure 7 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application;

[0050] Figure 8 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0051] Figure 9 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0052] Figure 10 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0053] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0054] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0055] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0056] The terms “substantially,” “about,” and similar terms used herein are used as approximations rather than as terms of degree, and are intended to take into account the inherent biases of measurements or calculations known to those skilled in the art. Furthermore, the use of “may” in describing embodiments of the invention refers to “one or more possible embodiments.” The terms “use,” “using,” and “used” used herein are to be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Additionally, the term “exemplary” is intended to refer to an instance or illustration.

[0057] This application can be applied to the field of natural language processing in the field of artificial intelligence. The following will introduce several application scenarios that have been implemented in products, taking natural language processing as an example.

[0058] First, we will introduce the application scenarios of this application. This application can be applied, but is not limited to, to applications with image generation capabilities (hereinafter referred to as generation applications) or cloud services provided by cloud-side servers, etc., which will be introduced separately below:

[0059] I. Generating class applications

[0060] The product form of this application embodiment can be a generative application. Generative applications can run on terminal devices or cloud servers.

[0061] In one possible implementation, a generative application can perform image generation tasks and obtain the processed results. For example, text-based image generation tasks. Another example is image enhancement tasks based on images (including but not limited to super-resolution, denoising, and dehazing).

[0062] For example, generative applications can implement image generation tasks that include at least diffusion-based methods, but are not limited to this.

[0063] In one possible implementation, a user can open a generative application installed on a terminal device and input image data and text data (the text may be triggered by an instruction, not necessarily actively input by the user). The generative application can process the image and text using a model trained by the method provided in the embodiments of this application, or by the method provided in the embodiments of this application, and present the processing result to the user (the presentation method may include, but is not limited to, displaying, playing, saving, uploading to the cloud, etc.).

[0064] In one possible implementation, a user can open a generative application installed on a terminal device and input image and text data. The generative application can then send the image and text data to a cloud-based server. The cloud-based server processes the image or text using a model trained by the method provided in this application embodiment and sends the processing result back to the terminal device. The terminal device can then present the processing result to the user (the presentation method may include, but is not limited to, displaying, playing, saving, or uploading to the cloud).

[0065] The following sections will introduce the generated application class in this application embodiment from the perspectives of functional architecture and product architecture that implements the functions.

[0066] Reference Figure 1 , Figure 1 This is a schematic diagram of the functional architecture of the generated application class in the embodiments of this application:

[0067] In one possible implementation, such as Figure 1 As shown, the generative application 102 can receive input parameters 101 (e.g., including images or text) and generate processing results 103. The generative application 102 can be executed on at least one computer system (for example) and includes computer code that, when executed by one or more computers, causes the computers to execute a model trained by the methods provided in the embodiments of this application.

[0068] It should be understood that the steps related to the model inference process in the embodiments of this application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and server is not limited to the processor-memory architecture described above. The following section will further explain... Figure 2 The system architecture provided in the embodiments of this application will be described in detail.

[0069] Figure 2 This is a schematic diagram of the system architecture provided for an embodiment of this application. Figure 2 As shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0070] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, while the preprocessing modules 513 and 514 are optional.

[0071] The execution device 510 can be a terminal device or server that runs the above-mentioned generation application, or image classification or retrieval application.

[0072] The data acquisition device 560 is used to collect training samples. Training samples can be image data or text data, etc. After collecting the training samples, the data acquisition device 560 stores these training samples in the database 530.

[0073] The training device 520 can maintain training samples in the database 530 and obtain the target model / rule 501 from the neural network to be trained (e.g., the neural network model in the embodiments of this application (e.g., including encoder, generative model (e.g., diffusion model)).

[0074] It should be understood that the training device 520 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model based on the pre-training.

[0075] It should be noted that in practical applications, the training samples maintained in database 530 may not all come from the data acquisition device 560; they may also be received from other devices. Furthermore, it should be noted that training device 520 may not necessarily train the target model / rule 501 entirely based on the training samples maintained in database 530; it may also obtain training samples from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0076] The target model / rule 501 trained using training device 520 can be applied to different systems or devices, such as... Figure 2 The execution device 510 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server, etc.

[0077] Specifically, the training device 520 can transfer the trained model to the execution device 510.

[0078] exist Figure 2In the execution device 510, an input / output (I / O) interface 512 is configured for data interaction with external devices. Users can input data (such as image data or text data in this embodiment) into the I / O interface 512 through the client device 540.

[0079] Preprocessing modules 513 and 514 are used to preprocess the input data received from the I / O interface 512. It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can be used directly to process the input data.

[0080] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0081] Finally, the I / O interface 512 provides the processing result to the client device 540, thereby providing it to the user.

[0082] exist Figure 2 In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various forms such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0083] It is worth noting that, Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2In this context, the data storage system 550 is an external storage device relative to the execution device 510. However, in other cases, the data storage system 550 may also be placed within the execution device 510. It should be understood that the aforementioned execution device 510 may be deployed within the client device 540.

[0084] III. Cloud services providing image compositing functionality:

[0085] In one possible implementation, the server can provide image compositing services to the client side via an application programming interface (API).

[0086] In this process, the terminal device can send relevant parameters (such as images, text, and other data) to the server through the API provided by the cloud. The server can then obtain the processing results based on the received parameters and return the processing results to the terminal.

[0087] The description of the terminal and server can be found in the above embodiments, and will not be repeated here.

[0088] like Figure 3 The process of using an image compositing function cloud service provided by a cloud platform is illustrated.

[0089] 1. Activate and purchase image compositing services.

[0090] 2. Users can download the software development kit (SDK) corresponding to the image compositing service. Cloud platforms usually provide multiple development versions of the SDK for users to choose from according to their development environment needs, such as JAVA version SDK, Python version SDK, PHP version SDK, Android version SDK, etc.

[0091] 3. After downloading the corresponding version of the SDK to their local machine according to their needs, users can import the SDK project into their local development environment, configure and debug it in the local development environment, and develop other functions in the local development environment to form an application that integrates image synthesis capabilities.

[0092] 4. When an application needs to perform image compositing, it can trigger an API call for the image compositing function. When the application triggers the image compositing function, it sends an API request to the running instance of the image compositing service in the cloud environment. The API request carries an image or text, which is then processed by the running instance in the cloud environment to obtain the result.

[0093] 5. The cloud environment returns the processing results to the application, thus completing one image compositing function call.

[0094] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0095] (1) Neural Network

[0096] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0097]

[0098] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0099] (2) Deep Neural Networks

[0100] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0101] (3) Loss Function

[0102] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0103] (4) Backpropagation algorithm

[0104] Backpropagation (BP) can be used during training to correct the parameters in the initial model, thereby reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters in the initial model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining optimal model parameters, such as the weight matrix.

[0105] (5) Diffusion model

[0106] A diffusion model is a generative model used to generate data such as images and text. The core idea of ​​a diffusion model is to diffuse noise into the data and then gradually remove the noise to recover the original data. The diffusion model consists of two stages: a forward process (noise diffusion) and a reverse process (noise removal and recovery).

[0107] Knowledge distillation is currently the mainstream approach for accelerating diffusion model sampling. However, existing techniques struggle to fully transfer the prior knowledge of the teacher model to the student model. This results in lower quality output data from the distilled student model, impacting its performance, and also leads to lower efficiency in the distillation process, resulting in significant training time overhead.

[0108] The data processing method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0109] Reference Figure 4 , Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application, such as... Figure 4 As shown in the embodiment of this application, a data processing method may include steps 401 to 404, which are described in detail below.

[0110] 401. Obtain noise data.

[0111] The noise data can be randomly generated noise, such as random Gaussian noise. The noise data can also be image data (or, as it may be called, image-formatted data), such as data consisting of multiple pixels, each pixel corresponding to a pixel value.

[0112] Figure 4 A corresponding implementation could be the training process of a model (e.g., a diffusion model), such as model pre-training or model fine-tuning.

[0113] Specifically, the training process can be knowledge distillation. For a given diffusion model (i.e., the teacher model, such as the second diffusion model in this embodiment), the teacher model can be used to distill the student model (such as the first diffusion model in this embodiment). The first and second diffusion models can have the same or similar network structures. In this embodiment, the student model is the first diffusion model, and the teacher model is the second diffusion model.

[0114] In this model, the diffusion model can generate data based on the input data as a condition. For example, the input data can be the original input of the training samples; for instance, when the processing task is image super-resolution, the input data can be low-resolution images. The diffusion model can also be a text-to-image model, where the input data can be text, and the diffusion model can generate images that include the semantic content indicated by the input text. Furthermore, the input data can also be a video containing multiple image frames.

[0115] When generating data, the diffusion model can use the input data as a condition to denoise noisy data (such as randomly generated Gaussian noise) in multiple steps until the generated data is obtained.

[0116] Knowledge distillation of the diffusion model refers to transferring the performance of the diffusion model, which serves as the teacher model, to the student model. During the knowledge distillation process, the student model uses far fewer denoising steps than the teacher model, enabling the student model to obtain better output with fewer denoising steps.

[0117] In one possible implementation, the specific structure of the diffusion model is not limited in the embodiments of this application.

[0118] In this application, denoising can be understood as noise prediction and denoising based on the predicted noise.

[0119] 402. Using the first diffusion model, the noise data is denoised in multiple steps to obtain the denoising result corresponding to each step, where the first step is either the first step or an intermediate step.

[0120] In this embodiment of the application, the first diffusion model can be used as the student model and the second diffusion model can be used as the teacher model. The maximum number of denoising steps of the student model is less than (for example, much less than) the maximum number of denoising steps of the teacher model.

[0121] The maximum number of denoising steps here can be understood as the number of denoising steps used during training. For example, the maximum number of denoising steps for the teacher model is 1000, while the maximum number of denoising steps for the student model is 2, 4, 6, or 8, etc.

[0122] In one possible implementation, the noise data can be denoised in multiple steps sequentially using a first diffusion model based on the input data (for example, the input data can be used as a conditional input to denoise the noise data in multiple steps sequentially), to obtain the denoising result corresponding to each step.

[0123] In step 402, the noisy data can be denoised in multiple steps using the first diffusion model to obtain the denoising result for each step. The denoising result of the last step in the multiple steps can be the final output obtained by the first diffusion model, while the intermediate steps in the multiple steps can obtain intermediate outputs (the denoising process of multiple steps can be understood as a progressive denoising, where each step further denoises based on the denoising of the previous step until the denoising process of the last step can obtain the final output).

[0124] It should be understood that in the embodiments of this application, the number of steps gradually increases during the noise addition process, and the number of steps gradually decreases during the noise removal process. For example, the noise addition process is from step 0 to step T, and the noise removal process is from step T to step 0. T is the maximum number of steps, and T is a natural number greater than 0.

[0125] For example, when the number of steps is 2, the first diffusion model can denoise the input noise through the first step of the denoising process to obtain output 1, and then denoise output 1 through the second step of the denoising process to obtain output 2.

[0126] 403. Using the second diffusion model and the third diffusion model, noise prediction is performed on the noise addition result of the denoising result corresponding to the first step number, respectively, to obtain the first prediction result and the second prediction result; the maximum number of denoising steps of the first diffusion model is less than the maximum number of denoising steps of the second diffusion model.

[0127] The third diffusion model can be a model with the same network structure as the first diffusion model. During training, both the first and third diffusion models can be updated, resulting in different parameter sizes for the two models. Noise prediction using the second and third diffusion models refers to denoising each denoised result; the first and second prediction results are the denoised results.

[0128] The multi-step denoising process of the diffusion model can be understood as denoising through a data denoising path (or, it can be described as a data generation path). The denoising steps of each step can be understood as a path. The teacher model can obtain a better result by denoising the data according to a more reasonable path. In existing technologies, the final output of the student model (e.g., output 2 in the above example) is denoised, and the denoised result is used by both the student model and the teacher model to predict noise, and the noise prediction results are aligned. If the student model only learns the teacher model's ability to generate the final result, but does not learn the ability to generate data through the denoising path (that is, it does not learn the noise prediction ability of the intermediate steps of the teacher model), the training process of the student model will be inefficient.

[0129] The approach of this application is to add noise to the output of the steps preceding the last step (i.e., the first or intermediate steps) in a multi-step denoising process of the student model (corresponding to the first diffusion model). Noise prediction is then performed on the noise-added results using a model corresponding to the student model (corresponding to the third diffusion model) and a teacher model (corresponding to the second diffusion model). Knowledge distillation is achieved by aligning the noise prediction results of the student and teacher models. In this application, in addition to the output of the student model's last step, the denoising prediction information of the student and teacher models for the outputs of the steps preceding the student model's last step is also aligned. This allows the student model to learn the denoising path used by the teacher model in generating the data, improving the supervision signal during training, making the learning process easier, increasing the training efficiency of the student model, reducing training time overhead, and enabling the student model to achieve better model performance.

[0130] Furthermore, the knowledge distillation process described above only needs to process the data generated by the student model, without needing to process the labeled training data, thus improving the flexibility and usability of the solution.

[0131] Of course, in this embodiment of the application, when the number of multiple steps in step 402 is 2, the output of the first step can be used as the denoising result of the first step. When the number of multiple steps in step 402 is greater than 2, the output of one of the middle steps can be used as the denoising result of the first step.

[0132] As the third diffusion model is trained, it can achieve noise prediction capabilities that are the same as or close to those of the first diffusion model.

[0133] The following describes the noise addition process for the noise reduction result corresponding to the first step:

[0134] The noise addition process involves adding noise in a certain number of steps. The number of steps determines the degree of noise addition. The teacher model often has a large number of denoising steps. In order to ensure that the student model can learn the noise prediction ability of the teacher model in the middle steps, during the knowledge distillation process, the number of steps in the noise addition process for the denoising result corresponding to the first step can be randomly selected from the number of denoising steps of the teacher model.

[0135] In one possible implementation, the second step can be a step following the first step among multiple steps. The second step can be an intermediate step or the last step. Noise prediction is performed on the denoised result corresponding to the second step using a first diffusion model and a second diffusion model, respectively, to obtain a third and fourth prediction result. The first diffusion model is updated based on the difference between the third and fourth prediction results. Specifically, the denoised result corresponding to the first step is obtained after the third step of noise addition, and the denoised result corresponding to the second step is obtained after the fourth step of noise addition. The noise addition level of the fourth step is lower than that of the third step. This method ensures that when adding noise to the output of the student model at different steps, the later the step, the fewer the steps (i.e., the lower the noise addition level).

[0136] In one possible implementation, the third step is selected from the first numerical interval (e.g., randomly), and the fourth step is selected from the second numerical interval (e.g., randomly). The first and second numerical intervals do not overlap (i.e., they are completely offset from each other), and the values ​​included in the second numerical interval are less than the values ​​included in the first numerical interval. For example, when the maximum number of denoising steps in the second diffusion model is T, the first numerical interval can be (T / 2, T], and the second numerical interval can be [0, T / 2].

[0137] In one possible implementation, in order to ensure that when adding noise to the output of the student model at different steps, the number of steps for adding noise to the output of later steps is smaller, the values ​​included in the first numerical range are not greater than the maximum number of denoising steps of the second diffusion model.

[0138] By using the above method, the noise added to the output of the student model at different steps is equivalent to decoupling the learning objectives with different sampling steps, so that the student model can support both high-quality deterministic sampling and flexible adjustment of the number of steps.

[0139] 404. Update the first diffusion model and the third diffusion model based on the difference between the first and second prediction results.

[0140] For example, the difference between the first and second prediction results can be used to construct a loss and determine the gradient through the loss. Then, the first and third diffusion models can be updated using the gradient.

[0141] Reference Figure 5 , Figure 5 This is a schematic diagram of an application architecture according to an embodiment of this application.

[0142] in, Figure 5 The middle school student model underwent two steps of denoising, the first diffusion model. Figure 5 The noisy data can be denoised twice, yielding two denoised results (1 and 2). Then, denoised result 1 and denoised result 2 can be denoised again. The number of steps in the denoising process for denoised result 1 is randomly selected from [T / 2, T), and the number of steps in the denoising process for denoised result 2 is randomly selected from [0, T / 2]. Then, noise prediction is performed on denoised result 1 and denoised result 2 using a first diffusion model and a second diffusion model, respectively. A loss is constructed based on the difference between prediction result 1 and prediction result 2. Figure 5 The diffusion loss shown can then be used to update the first diffusion model.

[0143] The data processing method in this application embodiment has been described above from the model training process. Next, the data processing method in this application embodiment will be described from the model inference process.

[0144] Reference Figure 6 , Figure 6 This is a flowchart illustrating a data processing method provided in an embodiment of this application, such as... Figure 6 As shown in the embodiment of this application, a data processing method may include steps 601 to 602, which are described in detail below.

[0145] 601. Obtain the first data and noise data.

[0146] In this model, the diffusion model can generate data based on the input data (first data). For example, the first data can be the original input of the training samples. For instance, when the processing task is image super-resolution, the first data can be a low-resolution image. The diffusion model can also be a text-to-image model, where the first data can be text, and the diffusion model can generate an image (second image) that includes the semantic content indicated by the input text.

[0147] 602. Take the first data as the data obtained through... Figure 4 The updated first diffusion model, obtained by the method in the corresponding embodiment, is used to denoise the noisy data in multiple steps to obtain the second data.

[0148] Reference Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application, such as... Figure 7 As shown in the embodiment of this application, a data processing apparatus 700 includes:

[0149] Acquisition module 701 is used to acquire noise data;

[0150] For a detailed description of the acquisition module 701, please refer to the description of step 401 in the above embodiments, which will not be repeated here.

[0151] The denoising module 702 is used to predict the noise added to the denoising result corresponding to the first step number using a first diffusion model and a second diffusion model, respectively, to obtain a first prediction result and a second prediction result; the maximum number of denoising steps of the first diffusion model is less than the maximum number of denoising steps of the second diffusion model.

[0152] For a detailed description of the noise reduction module 702, please refer to the description of step 402 in the above embodiment, which will not be repeated here.

[0153] The noise prediction module 703 is used to predict the noise added to the denoising result corresponding to the first step number using the second diffusion model and the third diffusion model, respectively, to obtain the first prediction result and the second prediction result; the maximum number of denoising steps of the first diffusion model is less than the maximum number of denoising steps of the second diffusion model.

[0154] For a detailed description of the noise prediction module 703, please refer to the description of step 403 in the above embodiment, which will not be repeated here.

[0155] The update module 704 is used to update the first diffusion model and the third diffusion model based on the difference between the first prediction result and the second prediction result.

[0156] For a detailed description of the update module 704, please refer to the description of step 404 in the above embodiments, which will not be repeated here.

[0157] In one possible implementation, the second step is a step after the first step among multiple steps. The noise prediction module 703 is further configured to: perform noise prediction on the noise addition result of the denoising result corresponding to the second step through the third diffusion model to obtain a third prediction result; and perform noise prediction on the noise addition result of the denoising result corresponding to the second step through the second diffusion model to obtain a fourth prediction result.

[0158] Update module 704 is also used to update the first diffusion model and the third diffusion model based on the difference between the third prediction result and the fourth prediction result; wherein,

[0159] The noise addition result corresponding to the first step is obtained after the noise addition in the third step, and the noise addition result corresponding to the second step is obtained after the noise addition in the fourth step. The noise addition level in the fourth step is lower than that in the third step.

[0160] In one possible implementation, the third step is selected from the first numerical interval, and the fourth step is selected from the second numerical interval. The first and second numerical intervals do not intersect, and the values ​​in the second numerical interval are less than the values ​​in the first numerical interval.

[0161] In one possible implementation, the values ​​in the first numerical range are no greater than the maximum number of denoising steps of the second diffusion model.

[0162] In one possible implementation, the acquisition module 701 is also used to acquire input data, which is an image and / or text;

[0163] The denoising module 702 is specifically used to: take the input data as the conditional input of the first diffusion model, and sequentially perform multiple steps of denoising on the noise data to obtain the denoising result corresponding to each step; the denoising result is an image.

[0164] In one possible implementation, the noise data is image data.

[0165] The following describes a terminal device provided in an embodiment of this application. Please refer to [link to relevant documentation]. Figure 8 , Figure 8 This is a schematic diagram of a terminal device provided in an embodiment of this application. The terminal device 800 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., and is not limited thereto. Specifically, the terminal device 800 includes: a receiver 801, a transmitter 802, a processor 803, and a memory 804 (wherein the terminal device 800 may have one or more processors 803). Figure 8 (Taking a processor as an example), the processor 803 may include an application processor 8031 ​​and a communication processor 8032. In some embodiments of this application, the receiver 801, transmitter 802, processor 803, and memory 804 may be connected via a bus or other means.

[0166] Memory 804 may include read-only memory and random access memory, and provides instructions and data to processor 803. A portion of memory 804 may also include non-volatile random access memory (NVRAM). Memory 804 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0167] The processor 803 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0168] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 803. Processor 803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 803 or by instructions in software form. Processor 803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 803 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 804. The processor 803 reads the information from memory 804 and, in conjunction with its hardware, completes the steps involved in the model training or model inference process described above.

[0169] Receiver 801 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 802 can be used to output digital or character information through the first interface; transmitter 802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 802 may also include a display device such as a display screen.

[0170] This application also provides a server; please refer to [link / reference]. Figure 9 , Figure 9 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 99 (e.g., one or more processors) and memory 932, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 942 or data 944. The memory 932 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 99 may be configured to communicate with the storage media 930 and execute the series of instruction operations in the storage media 930 on the server 900.

[0171] Server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input / output interfaces 958; or one or more operating systems 941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0172] In this embodiment, the central processing unit 99 is used to perform actions related to model training or model inference in the above embodiments.

[0173] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0174] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0175] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0176] For details, please refer to Figure 10 , Figure 10 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1000. The NPU 1000 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1003, which is controlled by the controller 1004 to extract matrix data from the memory and perform multiplication operations.

[0177] In some implementations, the arithmetic circuit 1003 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1003 is a two-dimensional pulsating array. The arithmetic circuit 1003 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1003 is a general-purpose matrix processor.

[0178] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1008.

[0179] Unified memory 1006 is used to store input and output data. Weight data is directly transferred to weight memory 1002 via Direct Memory Access Controller (DMAC) 1005. Input data is also transferred to unified memory 1006 via DMAC.

[0180] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1009.

[0181] The Bus Interface Unit (BIU) 1010 is used by the instruction fetch memory 1009 to fetch instructions from external memory, and also by the memory access controller 1005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0182] The DMAC is mainly used to move input data from external memory DDR to unified memory 1006, or to weight data to weight memory 1002, or to input data to input memory 1001.

[0183] The vector computation unit 1007 includes multiple processing units that further process the output of the computation circuit 1003 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0184] In some implementations, the vector computation unit 1007 can store the processed output vector in the unified memory 1006. For example, the vector computation unit 1007 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1003, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1003, for example, for use in subsequent layers of the neural network.

[0185] The instruction fetch buffer 1009 connected to the controller 1004 is used to store the instructions used by the controller 1004;

[0186] Unified memory 1006, input memory 1001, weight memory 1002, and instruction fetch memory 1009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0187] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0188] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0189] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0190] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0191] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, The method includes: Acquire noise data; The noise data is denoised in multiple steps using the first diffusion model to obtain the denoising result corresponding to each step, wherein the first step is the first step or an intermediate step of the multiple steps. The noise prediction results are obtained by using the second diffusion model and the third diffusion model to predict the noise addition results of the denoising results corresponding to the first step number, respectively, to obtain the first prediction result and the second prediction result; the maximum number of denoising steps of the first diffusion model is less than the maximum number of denoising steps of the second diffusion model; Based on the difference between the first prediction result and the second prediction result, update the first diffusion model and the third diffusion model.

2. The method according to claim 1, characterized in that, The second step is one of the multiple steps following the first step, and the method further includes: The noise prediction is performed on the noise addition result of the denoising result corresponding to the second step using the third diffusion model to obtain the third prediction result; Using the second diffusion model, noise prediction is performed on the noise addition result of the denoising result corresponding to the second step to obtain the fourth prediction result; Based on the difference between the third prediction result and the fourth prediction result, the first diffusion model and the third diffusion model are updated; wherein... The noise addition result corresponding to the first step is obtained after the noise addition in the third step, and the noise addition result corresponding to the second step is obtained after the noise addition in the fourth step. The noise addition degree of the fourth step is lower than that of the third step.

3. The method according to claim 2, characterized in that, The third step is selected from the first numerical range, and the fourth step is selected from the second numerical range. The first numerical range and the second numerical range have no intersection, and the value in the second numerical range is less than the value in the first numerical range.

4. The method according to claim 3, characterized in that, The values ​​in the first numerical range are not greater than the maximum number of denoising steps of the second diffusion model.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: Acquire input data, wherein the input data is an image and / or text; The step of performing multiple denoising steps on the noise data using a first diffusion model to obtain a denoising result for each step includes: The input data is used as the conditional input of the first diffusion model, and the noise data is denoised in multiple steps in sequence to obtain the denoising result corresponding to each step; the denoising result is an image.

6. The method according to any one of claims 1 to 5, characterized in that, The noise data is image data.

7. A data processing method, characterized in that, The method includes: Acquire the initial data and the data with added noise; The first data is used as a conditional input to the first diffusion model obtained by any one of claims 1 to 6, and the noisy data is denoised in multiple steps to obtain the second data.

8. The method according to claim 7, characterized in that, The first data is text and / or an image; the second data is an image generated based on the text or the image.

9. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire noise data; The denoising module is used to perform multiple steps of denoising on the noise data sequentially through a first diffusion model to obtain the denoising result corresponding to each step, wherein the first step is the first step or an intermediate step; The noise prediction module is used to predict the noise of the noise addition result of the denoising result corresponding to the first step number using the second diffusion model and the third diffusion model, respectively, to obtain the first prediction result and the second prediction result; the maximum number of denoising steps of the first diffusion model is less than the maximum number of denoising steps of the second diffusion model. An update module is used to update the first diffusion model and the third diffusion model based on the difference between the first prediction result and the second prediction result.

10. The apparatus according to claim 9, characterized in that, The second step is one of the multiple steps following the first step. The noise prediction module is further configured to: The noise prediction is performed on the noise addition result of the denoising result corresponding to the second step using the third diffusion model to obtain the third prediction result; Using the second diffusion model, noise prediction is performed on the noise addition result of the denoising result corresponding to the second step to obtain the fourth prediction result; Based on the difference between the third prediction result and the fourth prediction result, the first diffusion model and the third diffusion model are updated; wherein... The noise addition result corresponding to the first step is obtained after the noise addition in the third step, and the noise addition result corresponding to the second step is obtained after the noise addition in the fourth step. The noise addition degree of the fourth step is lower than that of the third step.

11. The apparatus according to claim 10, characterized in that, The third step is selected from the first numerical range, and the fourth step is selected from the second numerical range. The first numerical range and the second numerical range have no intersection, and the value in the second numerical range is less than the value in the first numerical range.

12. The apparatus according to claim 11, characterized in that, The values ​​in the first numerical range are not greater than the maximum number of denoising steps of the second diffusion model.

13. The apparatus according to any one of claims 9 to 12, characterized in that, The acquisition module is also used to acquire input data, which is an image and / or text; The noise reduction module is specifically used for: The input data is used as the conditional input of the first diffusion model, and the noise data is denoised in multiple steps in sequence to obtain the denoising result corresponding to each step; the denoising result is an image.

14. The apparatus according to any one of claims 9 to 13, characterized in that, The noise data is image data.

15. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the first data and the noisy data; A generation module is configured to take the first data as a conditional input to the first diffusion model obtained by the method described in any one of claims 1 to 6, and perform denoising on the noisy data in multiple steps to obtain the second data.

16. The apparatus according to claim 15, characterized in that, The first data is text and / or an image; the second data is an image generated based on the text or the image.

17. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1 to 8.

18. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 8.

19. A computing device, comprising at least one processor and at least one memory; the processor and the memory are connected via a communication bus and communicate with each other. The at least one memory is used to store code; The at least one processor is used to execute the code to perform the method as described in any one of claims 1 to 8.

20. A chip, comprising a processor, characterized in that, The processor is used by the data processing apparatus to implement the method as described in any one of claims 1 to 8.

Images