Data processing method and apparatus

Abstract

A data processing method, applied to the field of artificial intelligence. The method comprises: acquiring first data; on the basis of the first data, obtaining a first processing result and a second processing result by means of a first model and a second model, respectively, wherein the first model and the second model are diffusion models, the first model is a teacher model, and the second model is a student model; respectively adding noise to the first processing result and the second processing result to obtain a first noise-added result and a second noise-added result; by means of the first model, denoising the first noise-added result to obtain a first denoised result; by means of the first model, denoising the second noise-added result to obtain a second denoised result; and obtaining first loss on the basis of the first denoised result and the second denoised result, and updating the second model on the basis of the first loss. The present application can improve the quality of student models after distillation.

Description

Data processing method and apparatus thereof

[0001] The present application claims priority to the Chinese Patent Application No. 202411096304.2, filed on August 9, 2024, and entitled "A data processing method and apparatus thereof", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of artificial intelligence, and in particular to a data processing method and apparatus thereof. BACKGROUND

[0003] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, enabling machines to have perception, reasoning and decision-making functions.

[0004] As a new generation model, the diffusion model has received more and more attention due to its ability to model complex data distribution, and has gradually replaced the generative adversarial model and been applied to various downstream tasks. However, the diffusion model requires tens or even hundreds of iterations of sampling to refine the output of the model when reasoning, resulting in high latency and computational cost, which poses a great challenge to the landing and application of the model. To solve this problem, existing methods compress the sampling path of the diffusion model by redesigning the diffusion process, using high-order numerical solvers and knowledge distillation techniques.

[0005] Among these methods, the knowledge distillation-based technique is currently the mainstream solution to accelerate the sampling of the diffusion model, as it has the potential to accelerate the sampling efficiency of the model while ensuring that the performance of the model does not decrease, and this technique can be generalized to different diffusion models, with strong generalizability. However, in existing knowledge distillation techniques for diffusion models, the teacher model needs to process the noisy output of the student model to obtain the output of the teacher model, and align the output of the teacher model with the output of the student model. However, the output of the teacher model is highly similar to the real high-definition image, and the existing way will introduce meaningless gradient directions to the student model, making it difficult to effectively transfer the prior knowledge of the teacher model to the student model, resulting in low-quality output data of the distilled student model. SUMMARY

[0006] In a first aspect, the present application provides a data processing method, comprising: obtaining first data; obtaining a first processing result and a second processing result by a first model and a second model respectively according to the first data, wherein the first model and the second model are diffusion models, the first model is a teacher model, and the second model is a student model; adding noise to the first processing result and the second processing result respectively to obtain a first noise-added result and a second noise-added result; denoising the first noise-added result by the first model to obtain a first denoised result; denoising the second noise-added result by the first model to obtain a second denoised result; obtaining a first loss according to the first denoised result and the second denoised result, and updating the second model according to the first loss.

[0007] For the existing score distillation sampling method, the inherent bias may introduce meaningless gradient directions to the student model (because the teacher model processes the noise-added output of the student model, and the teacher model is difficult to accurately predict the noise added to the output of the student model), resulting in output ambiguity, oversaturation and other problems. For the existing distribution matching distillation method, a new diffusion model needs to be trained to learn the score function of the sample generated by the student model, and the generator is updated according to the difference between the score function predicted by the new model and the score function predicted by the teacher model. However, this method involves a complex training process, which requires alternating training between the student model and the new diffusion model.

[0008] In the embodiments of the present application, in order to simply and effectively eliminate the bias (existing in score distillation sampling) and provide a clear gradient direction for the optimization of the student model, in the embodiments of the present application, the diffusion model as the teacher model processes the noise-added result of the output of the student model, and at the same time, the teacher model processes the original image (consistent with the student model). Then, the teacher model processes the noise-added output of the teacher model, and the two results obtained by the teacher model are used to construct a loss and update the student model, so that meaningless gradient directions are not introduced to the student model without introducing a new model, thereby improving the quality of the distilled student model.

[0009] In a possible implementation, the first processing result and the second processing result are at least one of the following: denoised scores, data enhanced from the first data, or predicted noise.

[0010] In a possible implementation, the adding noise to the first processing result and the second processing result respectively to obtain a first noise-added result and a second noise-added result comprises: adding noise of a target step to the first processing result and the second processing result respectively to obtain the first noise-added result and the second noise-added result; and the denoising the first noise-added result by using the first model to obtain a first denoised result and denoising the second noise-added result by using the first model to obtain a second denoised result comprises: denoising the first noise-added result by using the first model to obtain the first denoised result, and denoising the second noise-added result by using the first model to obtain the second denoised result.

[0011] In a possible implementation, the target step is obtained by sampling in a preset step range, and steps included in the preset step range are all less than one fourth of a total denoising step.

[0012] By sampling noise at small time steps, the ability of the student model to generate high-frequency details of an image can be enhanced.

[0013] In a possible implementation, the method further comprises:

[0014] obtaining second data, the second data being ground truth data corresponding to the first data;

[0015] adding noise to the second data to obtain a third noise-added result;

[0016] obtaining a first feature and a second feature respectively by using an encoding network in the first model according to the third noise-added result and the second noise-added result;

[0017] obtaining a first discrimination result and a second discrimination result respectively by using a discriminator according to the first feature and the second feature;

[0018] obtaining a second loss according to the first discrimination result and the second discrimination result, and updating the second model according to the second loss.

[0019] In a possible implementation, the obtaining a first discrimination result and a second discrimination result respectively by using a discriminator according to the first feature and the second feature comprises:

[0020] mapping the target step by using a mapping network to obtain a first parameter;

[0021] fusing the first parameter with the first feature and the second feature respectively to obtain a first fusion result and a second fusion result;

[0022] According to the first fusion result and the second fusion result, a first discrimination result and a second discrimination result are obtained respectively by a discriminator.

[0023] In a possible implementation, the first feature includes a first sub-feature and a second sub-feature; the first sub-feature and the second sub-feature are features of different scales in the first feature;

[0024] The first parameter includes a first sub-parameter and a second sub-parameter; the first sub-parameter and the second sub-parameter are different;

[0025] The first parameter is fused with the first feature and the second feature respectively to obtain a first fusion result and a second fusion result, including:

[0026] The first sub-parameter is fused with the first sub-feature, and the second sub-parameter is fused with the second sub-feature, to obtain the first fusion result.

[0027] In a possible implementation, the second feature includes a third sub-feature and a fourth sub-feature; the third sub-feature and the fourth sub-feature are features of different scales in the second feature;

[0028] The first parameter includes a first sub-parameter and a second sub-parameter; the first sub-parameter and the second sub-parameter are different;

[0029] The first parameter is fused with the first feature and the second feature respectively to obtain a first fusion result and a second fusion result, including:

[0030] The first sub-parameter is fused with the third sub-feature, and the second sub-parameter is fused with the fourth sub-feature, to obtain the second fusion result.

[0031] Unlike the method of directly discriminating data with the features extracted by the diffusion model. Considering the strong correlation between the features (or outputs) extracted by the diffusion model and the time steps, a time modulation module is designed to map the time information into embedding and modulate the corresponding multi-scale features, and adaptively adjust the information strength derived from the features. Finally, according to the modulated features, the synthetic data and the real are discriminated, so as to improve the data processing accuracy of the trained student model.

[0032] In a possible implementation, according to the first discrimination result and the second discrimination result, a second loss is obtained, and the second model is updated according to the second loss, including:

[0033] According to the first discrimination result and the second discrimination result, a second loss is obtained, and the second model, the mapping network and the discriminator are updated according to the second loss.

[0034] In a possible implementation, the first data is an image, and the first processing result and the second processing result are enhanced images.

[0035] In a second aspect, the present application provides a data processing apparatus, the apparatus comprising:

[0036] An acquisition module configured to acquire first data;

[0037] A processing module configured to obtain a first processing result and a second processing result by a first model and a second model respectively according to the first data, wherein the first model and the second model are diffusion models, the first model is a teacher model, and the second model is a student model;

[0038] The first processing result and the second processing result are respectively added with noise to obtain a first noise-added result and a second noise-added result;

[0039] The first noise-added result is de-noised by the first model to obtain a first de-noised result;

[0040] The second noise-added result is de-noised by the first model to obtain a second de-noised result;

[0041] According to the first de-noised result and the second de-noised result, a first loss is obtained, and the second model is updated according to the first loss.

[0042] In a possible implementation, the first processing result and the second processing result are at least one of the following:

[0043] A de-noising score, data obtained by enhancing the first data, or predicted noise.

[0044] In a possible implementation, the processing module is specifically configured to:

[0045] The first processing result and the second processing result are respectively added with noise of a target step to obtain a first noise-added result and a second noise-added result;

[0046] The first noise-added result is de-noised by the first model with the target step to obtain a first de-noised result, and the second noise-added result is de-noised by the first model with the target step to obtain a second de-noised result.

[0047] In a possible implementation, the target step length is obtained by sampling in a preset step length range, and each step length in the preset step length range is less than one fourth of the total denoising step length.

[0048] In a possible implementation, the obtaining module is further configured to:

[0049] obtain second data, the second data being true value data corresponding to the first data;

[0050] The processing module is further configured to:

[0051] add noise to the second data to obtain a third noise-added result;

[0052] obtain a first feature and a second feature respectively by using an encoding network in the first model according to the third noise-added result and the second noise-added result;

[0053] obtain a first discrimination result and a second discrimination result respectively by using a discriminator according to the first feature and the second feature;

[0054] obtain a second loss according to the first discrimination result and the second discrimination result, and update the second model according to the second loss.

[0055] In a possible implementation, the processing module is specifically configured to:

[0056] map the target step length by using a mapping network to obtain a first parameter;

[0057] fuse the first parameter with the first feature and the second feature respectively to obtain a first fusion result and a second fusion result;

[0058] obtain a first discrimination result and a second discrimination result respectively by using a discriminator according to the first fusion result and the second fusion result.

[0059] In a possible implementation, the first feature includes a first sub-feature and a second sub-feature, and the first sub-feature and the second sub-feature are features of different scales in the first feature.

[0060] The first parameter includes a first sub-parameter and a second sub-parameter, and the first sub-parameter and the second sub-parameter are different.

[0061] The processing module is specifically configured to:

[0062] fuse the first sub-parameter with the first sub-feature and fuse the second sub-parameter with the second sub-feature to obtain a first fusion result.

[0063] In a possible implementation, the second feature includes a third sub-feature and a fourth sub-feature; the third sub-feature and the fourth sub-feature are features of different scales in the second feature.

[0064] The first parameter includes a first sub-parameter and a second sub-parameter; the first sub-parameter and the second sub-parameter are different.

[0065] The processing module is specifically configured to:

[0066] fuse the first sub-parameter and the third sub-feature, and fuse the second sub-parameter and the fourth sub-feature to obtain a second fusion result.

[0067] In a possible implementation, the processing module is specifically configured to:

[0068] obtain a second loss according to the first discrimination result and the second discrimination result, and update the second model, the mapping network, and the discriminator according to the second loss.

[0069] In a possible implementation, the first data is an image, and the first processing result and the second processing result are enhanced images.

[0070] In a third aspect, an embodiment of the present application provides a data processing apparatus, which can include a memory, a processor, and a bus system, wherein the memory is configured to store a program, and the processor is configured to execute the program in the memory to perform the method in the first aspect and any optional method in the first aspect.

[0071] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and when the computer program is executed on a computer, the computer program causes the computer to execute the method in the first aspect and any optional method in the first aspect, and the method in the second aspect and any optional method in the second aspect.

[0072] In a fifth aspect, an embodiment of the present application provides a computer program, which, when executed on a computer, causes the computer to execute the method in the first aspect and any optional method in the first aspect, and the method in the second aspect and any optional method in the second aspect.

[0073] In a sixth aspect, the present application provides a chip system, which includes a processor configured to support execution of a data processing apparatus to implement functions involved in the above aspects, for example, sending or processing data involved in the above methods; or, information. In a possible design, the chip system further includes a memory, and the memory is configured to save necessary program instructions and data for the execution device or the training device. The chip system can be composed of a chip, or can include a chip and other discrete devices. BRIEF DESCRIPTION OF DRAWINGS

[0074] FIG. 1A is a structural schematic diagram of an artificial intelligence subject framework;

[0075] FIG. 1B and FIG. 1C are schematic diagrams of application system frameworks of the present application;

[0076] FIG. 1D is a schematic diagram of an optional hardware structure of a terminal;

[0077] FIG. 2 is a structural schematic diagram of a server;

[0078] FIG. 3 is a schematic diagram of a system architecture of the present application;

[0079] FIG. 4 is a flow of a cloud service;

[0080] FIG. 5 is a schematic diagram of a data processing method provided by an embodiment of the present application;

[0081] FIG. 6 is a processing schematic of a data processing method provided by an embodiment of the present application;

[0082] FIG. 7 is a processing schematic of a data processing method provided by an embodiment of the present application;

[0083] FIG. 8 is a processing schematic of a data processing method provided by an embodiment of the present application;

[0084] FIG. 9A and FIG. 9B are schematic diagrams of architectures of methods provided by an embodiment of the present application;

[0085] FIG. 10 is a structural schematic diagram of a data processing apparatus provided by an embodiment of the present application;

[0086] FIG. 11 is a structural schematic diagram of an execution device provided by an embodiment of the present application;

[0087] FIG. 12 is a structural schematic diagram of a training device provided by an embodiment of the present application;

[0088] FIG. 13 is a structural schematic diagram of a chip provided by an embodiment of the present application. DETAILED DESCRIPTION

[0089] The embodiments of the present application are described below with reference to the accompanying drawings. The terms used in the embodiment part of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0090] The embodiments of the present application are described below with reference to the accompanying drawings. The skilled in the art can know that, with the development of technology and the appearance of new scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0091] The terms "first", "second", and the like, as used in the specification and in the claims of the application, and in the above Description of Embodiments, unless otherwise specified, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the application described herein are, for example, capable of use in either order. It is therefore intended that the following claims not be limited to the described order or sequence, except where the description below claims otherwise.

[0092] The terms "substantially", "about", and like terms as used herein are used in a similarly relative sense and are intended to account for variations in measurement values or calculated values that would be recognized by those of ordinary skill in the art. Further, the use of "may" when describing embodiments of the application means, for example, one or more embodiments. The terms "use", "using", and "used" as used herein can be taken in their broadest possible context as being synonymous with the terms "utilize", "utilizing", and "utilized", respectively. Additionally, the term "exemplary" is intended to refer to an instance or illustration.

[0093] First, the overall workflow of the artificial intelligence system is described. Please refer to FIG. 1A, which shows a structural schematic diagram of an artificial intelligence subject framework. The following describes the artificial intelligence subject framework from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the bottom infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0094] (1) Infrastructure

[0095] Infrastructure provides computing power support for artificial intelligence systems, realizes communication with the external world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network-related platform support and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication obtain data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for calculation.

[0096] (2) Data

[0097] The data of the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. The data relates to graphics, images, voice, text, and also relates to the Internet of Things data of traditional devices, including the business data of existing systems and the sensing data of force, displacement, liquid level, temperature, humidity, etc.

[0098] (3) Data processing

[0099] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision, etc.

[0100] Among them, machine learning and deep learning can model, extract, preprocess, train, etc. Symbolic and formalized intelligent information.

[0101] Reasoning refers to the process of simulating human intelligent reasoning methods in computers or intelligent systems, using formalized information to perform machine thinking and solve problems according to reasoning control strategies, and the typical function is search and matching.

[0102] Decision-making refers to the process of decision-making after intelligent information is reasoned, which usually provides functions such as classification, sorting, prediction, etc.

[0103] (4) General capabilities

[0104] After the data is processed as mentioned above, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0105] (5) Intelligent products and industry applications

[0106] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields, which are the packaging of the overall solution of artificial intelligence, and realize the application of intelligent information decision-making products. Its application fields mainly include: intelligent terminals, intelligent transportation, intelligent medical treatment, automatic driving, smart city, etc.

[0107] The present application can be applied to the field of natural language processing in the field of artificial intelligence. The following will introduce multiple application scenarios landing in products in the field of natural language processing.

[0108] First, the application scenarios of the present application will be introduced. The present application can be applied in, but is not limited to, application programs with image generation (hereinafter referred to as generation type application programs) or cloud-side server provided cloud services, etc. Next, they will be introduced respectively:

[0109] I. Generation type application program

[0110] The product form of the embodiments of the present application can be a generation type application program. The generation type application program can run on a terminal device or a server on the cloud side.

[0111] In one possible implementation, the generation type application program can implement an image generation task to obtain a processing result. For example, a text-based image generation task. For another example, an image enhancement task based on an image (including but not limited to super-resolution, denoising, defogging, etc.).

[0112] For example, the generation type application program can implement at least an image generation task based on a diffusion method, but is not limited thereto.

[0113] In one possible implementation, a user can open the generation type application program installed on a terminal device and input image data and text data (the text can be triggered by an instruction, and is not necessarily input by the user actively). The generation type application program can process the image and the text through a model trained by the method provided in the embodiments of the present application, or through the method provided in the embodiments of the present application, and present the processing result to the user (the presentation method can be, but is not limited to, display, play, save, upload to the cloud side, etc.).

[0114] In one possible implementation, a user can open the generation type application program installed on a terminal device and input image data and text data. The generation type application program can send the image data and the text data to a server on the cloud side. The server on the cloud side processes the image or the text through a model trained by the method provided in the embodiments of the present application, and returns the processing result to the terminal device. The terminal device can present the processing result to the user (the presentation method can be, but is not limited to, display, play, save, upload to the cloud side, etc.).

[0115] Next, the generation type application program in the embodiments of the present application will be introduced from the functional architecture and the product architecture implementing the function respectively.

[0116] Referring to FIG. 1B, FIG. 1B is a functional architecture diagram of the generation type application program in the embodiments of the present application:

[0117] In a possible implementation, as shown in FIG.1B, the generated application 102 can receive an input parameter 101 (for example, containing an image or text) and generate a processing result 103. The generated application 102 can be executed on, for example, at least one computer system, and includes computer code that, when executed by one or more computers, causes the computers to perform a model trained by a method provided by an embodiment of the present application.

[0118] Referring to FIG.1C, FIG.1C is a schematic diagram of an entity architecture for running a generated application in an embodiment of the present application:

[0119] Referring to FIG.1C, FIG.1C is a schematic diagram of a system architecture. The system can include a terminal 100 and a server 200. The server 200 can include one or more servers (one server is shown in FIG.1C as an example), and the server 200 can provide an image synthesis function for one or more terminals.

[0120] The terminal 100 can be installed with a generated application or open a webpage related to an image synthesis or natural language generation function. The application and the webpage can provide an interface, and the terminal 100 can receive parameters input by a user on the interface of the image synthesis or natural language generation function and send the parameters to the server 200. The server 200 can obtain a processing result based on the received parameters and return the processing result to the terminal 100.

[0121] It should be understood that, in some optional implementations, the terminal 100 can also complete the action of obtaining a processing result based on received parameters by itself without the cooperation of the server, and the embodiments of the present application are not limited in this regard.

[0122] Next, the product form of the terminal 100 in FIG.1C is described.

[0123] The terminal 100 in the embodiments of the present application can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), and the like, and the embodiments of the present application are not limited in this regard.

[0124] FIG.1D shows a schematic diagram of an optional hardware structure of the terminal 100.

[0125] Referring to FIG. 1D, the terminal 100 can include a radio frequency unit 110, a memory 120, an input unit 130, a display unit 140, a camera 150 (optional), an audio circuit 160 (optional), a speaker 161 (optional), a microphone 162 (optional), a processor 170, an external interface 180, a power supply 190, and the like. It will be understood by those skilled in the art that FIG. 1D is merely an example of a terminal or a multi-functional device and does not limit the terminal or the multi-functional device, which can include more or less components than those shown, or can have different arrangements of the components, or different names of the components.

[0126] The input unit 130 can be used to receive inputted digital or character information, and to generate key signal input with respect to user setting and function control of the portable multi-functional device. Specifically, the input unit 130 can include a touch screen 131 (optional) and / or other input device 132. The touch screen 131 can collect a touch operation of a user thereon or therearound (such as an operation of a user using a finger, a knuckle, a stylus, or any suitable object on or near the touch screen), and drive a corresponding connected device according to a pre-set program. The touch screen can detect a touch action of a user on the touch screen, convert the touch action into a touch signal and send the touch signal to the processor 170, and can receive a command from the processor 170 and execute the command; the touch signal at least includes touch point coordinate information. The touch screen 131 can provide an input interface and an output interface between the terminal 100 and the user. In addition, the touch screen can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch screen 131, the input unit 130 can include other input devices. Specifically, the other input device 132 can include one or more of, but is not limited to, a physical keyboard, a function key (such as a volume control button, an on-off button, etc.), a trackball, a mouse, a joystick, and the like.

[0127] The other input device 132 can receive inputted image data or text data.

[0128] The display unit 140 can be used to display information inputted by a user or information provided to a user, various menus of the terminal 100, an interactive interface, file display, and / or playing of any kind of multimedia file. In the embodiments of the present application, the display unit 140 can be used to display an interface of a generation application, a processing result, and the like.

[0129] The memory 120 can be used to store instructions and data. The memory 120 can mainly include a storage instruction area and a storage data area. The storage data area can store various data such as multimedia files, texts, etc. The storage instruction area can store software units such as operating systems, applications, instructions required by at least one function, etc. or their subsets, extended sets. Non-volatile random access memory can also be included. The processor 170 is provided with software and applications that include management of hardware, software and data resources in the computing processing device, support control. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0130] The processor 170 is the control center of the terminal 100, which connects each part of the entire terminal 100 through various interfaces and lines, executes various functions of the terminal 100 and processes data by running or executing instructions stored in the memory 120 and calling data stored in the memory 120, thereby performing overall control of the terminal device. Optionally, the processor 170 can include one or more processing units. Preferably, the processor 170 can integrate an application processor and a modem processor, wherein the application processor mainly processes operating systems, user interfaces and application programs, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 170. In some embodiments, the processor, the memory, can be implemented on a single chip, and in some embodiments, they can also be implemented on separate chips respectively. The processor 170 can also be used to generate corresponding operation control signals to the corresponding components of the computing processing device, read and process data in the software, especially read and process data and programs in the memory 120, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0131] The memory 120 can be used to store data processing method related software code, and the processor 170 can execute the steps of the chip data processing method, or can schedule other units (such as the above-mentioned input unit 130 and display unit 140) to realize corresponding functions.

[0132] The RF unit 110 (optional) can be used to receive and send signals in transceiving information or communication process, for example, receiving the downlink information of the base station, and processing by the processor 170. In addition, the uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF unit 110 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0133] In the embodiments of the present application, the RF unit 110 can send image data or text data to the server 200 and receive the processing result sent by the server 200.

[0134] It should be understood that the RF unit 110 is optional, which can be replaced by other communication interfaces, for example, a network interface.

[0135] The terminal 100 also includes a power supply 190 (such as a battery) for supplying power to each component. Preferably, the power supply can be logically connected to the processor 170 through a power management system, so as to realize the functions of managing charging, discharging and power consumption management through the power management system.

[0136] The terminal 100 also includes an external interface 180, which can be a standard Micro USB interface or a multi-pin connector, and can be used to connect the terminal 100 with other devices for communication, or can be used to connect a charger for charging the terminal 100.

[0137] Although not shown, the terminal 100 can also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, different function sensors, etc., which will not be described here. Some or all of the methods described below can be applied in the terminal 100 as shown in FIG. 1D.

[0138] Next, the product form of the server 200 in FIG. 1C is described;

[0139] FIG. 2 provides a structural schematic diagram of a server 200. As shown in FIG. 2, the server 200 includes a bus 201, a processor 202, a communication interface 203, and a memory 204. The processor 202, the memory 204, and the communication interface 203 communicate through the bus 201.

[0140] The bus 201 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is shown in FIG. 2, but it does not mean that there is only one bus or only one type of bus.

[0141] The processor 202 can be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP), etc.

[0142] The memory 204 can include a volatile memory, such as a random access memory (RAM). The memory 204 can also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a mechanical hard drive (HDD), or a solid state drive (SSD).

[0143] The memory 204 can be used to store software codes related to the data processing method, and the processor 202 can execute the steps of the chip data processing method or schedule other units to realize the corresponding functions.

[0144] It should be understood that the terminal 100 and the server 200 described above can be centralized or distributed devices, and the processors (e.g., the processor 170 and the processor 202) in the terminal 100 and the server 200 can be hardware circuits (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor, a microcontroller, or the like) or a combination of these hardware circuits. For example, the processor can be a hardware system with an instruction execution function, such as a CPU, a DSP, or the like, or a hardware system without an instruction execution function, such as an ASIC, an FPGA, or the like, or a combination of the hardware system without an instruction execution function and the hardware system with an instruction execution function.

[0145] It should be understood that the steps related to the model inference process in the embodiments of the present application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and the server is not limited to the architecture of the processor combined with the memory described above. The system architecture provided by the embodiments of the present application will be described in detail below in conjunction with FIG. 3.

[0146] FIG. 3 is a schematic diagram of a system architecture provided by the embodiments of the present application. As shown in FIG. 3, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data collection system 560.

[0147] The execution device 510 includes a computing module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The target model / rule 501 can be included in the computing module 511, and the preprocessing module 513 and the preprocessing module 514 are optional.

[0148] The execution device 510 can be a terminal device or a server that runs a generated application program, or an image classification or retrieval application program.

[0149] The data collection device 560 is configured to collect training samples. The training samples can be image data or text data, etc. After collecting the training samples, the data collection device 560 stores the training samples in the database 530.

[0150] The training device 520 can train a neural network to be trained (for example, a neural network model in the embodiments of the present application, for example, including an encoder, a generative model (for example, a diffusion model), etc.) based on the training samples maintained in the database 530, to obtain the target model / rule 501.

[0151] It should be understood that the training device 520 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model based on the pre-training.

[0152] It should be noted that in actual application, the training samples maintained in the database 530 may not all come from the collection of the data collection device 560, but may also be received from other devices. In addition, it should be noted that the training device 520 may not train the target model / rule 501 based on the training samples maintained in the database 530, but may also obtain training samples from the cloud or other places for model training, and the above description should not be regarded as a limitation of the embodiments of the present application.

[0153] The target model / rule 501 trained by the training device 520 can be applied to different systems or devices, such as the execution device 510 shown in FIG. 3. The execution device 510 can be a terminal such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR) device, a vehicle-mounted terminal, etc., and can also be a server, etc.

[0154] Specifically, the training device 520 can deliver the trained model to the execution device 510.

[0155] In FIG. 3, the execution device 510 is configured with an input / output (I / O) interface 512 for data interaction with external devices. The user can input data (for example, image data or text data in the embodiments of the present application, etc.) to the I / O interface 512 through the client device 540.

[0156] The pre-processing modules 513 and 514 are used for pre-processing the input data received by the I / O interface 512. It should be understood that there can be no pre-processing modules 513 and 514 or only one pre-processing module. When there is no pre-processing module 513 and 514, the input data can be directly processed by the calculation module 511.

[0157] In the process of pre-processing the input data by the execution device 510, or in the process of performing calculation by the calculation module 511 of the execution device 510, the execution device 510 can call the data, code, etc. in the data storage system 550 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 550.

[0158] Finally, the I / O interface 512 provides the processing result to the client device 540, thereby providing the user.

[0159] In the case shown in FIG. 3, the user can manually give the input data, which can be operated through the interface provided by the I / O interface 512. In another case, the client device 540 can automatically send the input data to the I / O interface 512, and if the authorization of the user is required for the client device 540 to automatically send the input data, the user can set the corresponding permission in the client device 540. The user can view the result output by the execution device 510 in the client device 540, and the specific presentation form can be a specific way such as display, sound, action, etc. The client device 540 can also be used as a data collection end to collect the input data input into the I / O interface 512 and the output result output from the I / O interface 512 as new sample data, and store the new sample data into the database 530. Of course, the input data input into the I / O interface 512 and the output result output from the I / O interface 512 can also be directly stored into the database 530 by the I / O interface 512 without being collected through the client device 540.

[0160] It is worth noting that FIG. 3 is only a schematic diagram of a system architecture provided by the embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the diagram does not constitute any limitation. For example, in FIG. 3, the data storage system 550 is an external memory relative to the execution device 510, and in other cases, the data storage system 550 can also be placed in the execution device 510. It should be understood that the above execution device 510 can be deployed in the client device 540.

[0161] From the inference side of the model:

[0162] In the embodiment of the present application, the calculation module 511 of the execution device 510 can obtain the code stored in the data storage system 550 to implement the steps related to the model inference process in the embodiment of the present application.

[0163] In the embodiments of the present application, the computing module 511 of the execution device 510 can include a hardware circuit (such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller, etc.), or a combination of these hardware circuits. For example, the training device 520 can be a hardware system with an execution instruction function, such as a CPU, a DSP, etc., or a hardware system without an execution instruction function, such as an ASIC, an FPGA, etc., or a combination of the hardware system without an execution instruction function and the hardware system with an execution instruction function.

[0164] Specifically, the computing module 511 of the execution device 510 can be a hardware system with an execution instruction function. The steps related to the model inference process provided in the embodiments of the present application can be software codes stored in a memory. The computing module 511 of the execution device 510 can obtain the software codes from the memory and execute the obtained software codes to implement the steps related to the model inference process provided in the embodiments of the present application.

[0165] It should be understood that the computing module 511 of the execution device 510 can be a combination of a hardware system without an execution instruction function and a hardware system with an execution instruction function. Part of the steps related to the model inference process provided in the embodiments of the present application can also be implemented by the hardware system without an execution instruction function in the computing module 511 of the execution device 510, which is not limited here.

[0166] From the training side of the model:

[0167] In the embodiments of the present application, the training device 520 can obtain the codes stored in the memory (not shown in FIG. 3, which can be integrated in the training device 520 or deployed separately from the training device 520) to implement the steps related to the model training in the embodiments of the present application.

[0168] In this embodiment of the application, the training device 520 may include hardware circuits (such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), general-purpose processors, digital signal processors (DSPs), microprocessors or microcontrollers, etc.) or combinations of these hardware circuits. For example, the training device 520 may be a hardware system with instruction execution capabilities, such as a CPU or DSP, or a hardware system without instruction execution capabilities, such as an ASIC or FPGA, or a combination of the aforementioned hardware systems without instruction execution capabilities and hardware systems with instruction execution capabilities.

[0169] It should be understood that the training device 520 can be a combination of a hardware system without the function of executing instructions and a hardware system with the function of executing instructions. Some steps related to the training of the neutralization model provided in the embodiments of this application can also be implemented by the hardware system in the training device 520 without the function of executing instructions, which is not limited here.

[0170] III. Cloud services providing image compositing functionality:

[0171] In one possible implementation, the server can provide image compositing services to the client side via an application programming interface (API).

[0172] In this process, the terminal device can send relevant parameters (such as image, text and other data) to the server through the API provided by the cloud. The server can obtain the processing results based on the received parameters and return the processing results to the terminal.

[0173] The description of the terminal and server can be found in the above embodiments, and will not be repeated here.

[0174] Figure 4 illustrates the process of using an image compositing function cloud service provided by a cloud platform.

[0175] 1. Activate and purchase content moderation services.

[0176] 2. The user can download a software development kit (SDK) corresponding to the content review service. The cloud platform usually provides multiple development versions of the SDK for the user to select according to the needs of the development environment, such as a JAVA version of the SDK, a python version of the SDK, a PHP version of the SDK, an Android version of the SDK, and the like.

[0177] 3. The user downloads the corresponding version of the SDK to the local according to the needs, imports the SDK project to the local development environment, configures and debugs in the local development environment, and can also develop other functions in the local development environment, so as to form an application that integrates the image synthesis function.

[0178] 4. In the process of using the image synthesis function application, when the image synthesis function is needed, the API call of the image synthesis function can be triggered. When the application triggers the image synthesis function, an API request is initiated to the running instance of the image synthesis function service in the cloud environment, wherein the image or text is carried in the API request. The running instance in the cloud environment processes the image and obtains the processing result.

[0179] 5. The cloud environment returns the processing result to the application, thereby completing one image synthesis function call.

[0180] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the related terms and concepts related to neural networks involved in the embodiments of the present application will be introduced first.

[0181] (1) Neural network

[0182] The neural network can be composed of neural units, and the neural unit can be an operation unit with xs (i.e., input data) and intercept 1 as inputs. The output of the operation unit can be:

[0183] where s = 1, 2, … n, n is a natural number greater than 1, Ws is the weight of xs, b is the bias of the neural unit. f is an activation function of the neural unit, which is used to introduce a nonlinear characteristic into the neural network to convert the input signal in the neural unit into an output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. The neural network is a network formed by connecting multiple single neural units, i.e., the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0184] (2) Deep Neural Network

[0185] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many layers of hidden layers, where "many" is not a special measure. From the position of DNN by different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, and the last layer is the output layer, and the number of layers in between is the hidden layer. The layers are fully connected, that is, any neuron in the i-th layer is connected to any neuron in the i+1-th layer. Although DNN looks very complex, in terms of the work of each layer, it is actually not complex, which is simply a linear relationship expression as follows: where, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is only an input vector after such a simple operation to get the output vector Since the number of layers of DNN is large, the number of coefficients W and offset vectors is also large. These parameters in DNN are defined as follows: taking the coefficient W as an example: suppose in a three-layer DNN, the linear coefficient of the fourth neuron in the second layer to the second neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary: the coefficient of the k-th neuron in the L-1-th layer to the j-th neuron in the L-th layer is defined as It should be noted that the input layer has no W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. In theory, the more parameters a model has, the higher its complexity, and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is to learn the weight matrix, and the ultimate goal is to get all the weight matrices of the trained deep neural network (weight matrix formed by many layers of vectors W).

[0186] (3) Loss function

[0187] In the process of training a deep neural network, because the output of the deep neural network is expected to be as close as possible to the value that is truly intended to be predicted, the weight vectors of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the truly intended target value, after comparing the two (of course, before the first update, there is usually an initialization process, that is, the parameters of each layer in the deep neural network are pre-configured), for example, if the predicted value of the network is too high, adjust the weight vector to make it predict lower, and keep adjusting until the deep neural network can predict the truly intended target value or a value very close to the truly intended target value. Therefore, it is necessary to define in advance "how to compare the difference between the predicted value and the target value", which is the loss function or the objective function, which is an important equation for measuring the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and then the training of the deep neural network becomes a process of trying to minimize this loss.

[0188] (4) Backpropagation algorithm

[0189] The back propagation (BP) algorithm can be used to correct the size of the parameters in the initial model during the training process, so that the error loss of the model becomes smaller and smaller. Specifically, the input signal is forward propagated until the output produces an error loss, and the parameters in the initial model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain optimal model parameters, such as weight matrices.

[0190] (5) Diffusion model

[0191] A kind of generative model, used to generate data such as images, text, etc. The core idea of diffusion model is to diffuse noise to the data, and then gradually remove the noise to recover the original data. Diffusion model includes two stages: forward process (noise diffusion) and reverse process (de-noise recovery).

[0192] As a new generative model, diffusion model has received more and more attention due to its ability to model complex data distribution, and has gradually replaced generative adversarial model and been applied to various downstream tasks. However, diffusion model requires tens or even hundreds of iterations to refine the output of the model during inference, resulting in high latency and computational cost, which poses a great challenge to the deployment and application of the model. To solve this problem, existing methods compress the sampling path of the diffusion model by redesigning the diffusion process, using high-order numerical solvers and knowledge distillation techniques.

[0193] In these methods, the knowledge distillation-based technique is currently the mainstream solution to accelerate the sampling of the diffusion model, as it has the potential to accelerate the sampling efficiency of the model while ensuring that the performance of the model does not decrease, and this technique can be generalized to different diffusion models and has strong generalizability. However, in the existing knowledge distillation techniques for diffusion models, the teacher model needs to process the output of the student model after adding noise, obtain the output of the teacher model, and align the output of the teacher model and the output of the student model. However, the output of the teacher model is highly similar to the real high-definition image, and the existing method will introduce meaningless gradient directions to the student model, making it difficult to effectively transfer the prior knowledge of the teacher model to the student model, resulting in low-quality output data of the distilled student model.

[0194] The data processing method of the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0195] Referring to FIG. 5, FIG. 5 is a flowchart of a data processing method provided by an embodiment of the present application. As shown in FIG. 5, the data processing method provided by an embodiment of the present application can include steps 501 to 506, which will be described in detail below.

[0196] 501, obtaining first data.

[0197] The corresponding embodiment of FIG. 5 can be a training process of a model (e.g., a diffusion model), such as a model pre-training or model fine-tuning process.

[0198] Specifically, the training process can be knowledge distillation. For a given diffusion model (i.e., a teacher model), a student model with the same network structure is first initialized using the teacher model.

[0199] The diffusion model can generate corresponding data based on the input data. For example, the first data can be the original input of the training sample (the second data can be the true value data corresponding to the first data). For example, when the task is image super-resolution, the first data can be a low-resolution image, and the second data can be a high-resolution data. The input data can be text, and the diffusion model can be a text graph model, which can generate an image including the semantic content indicated by the input text.

[0200] 502, obtaining first processing result and second processing result through first model and second model respectively according to the first data, wherein the first model and the second model are diffusion models, the first model is a teacher model, and the second model is a student model;

[0201] The original data is input into the two models. First, the single-step output of the student model and the multi-step output of the teacher model are aligned in the ordinary distillation manner.

[0202] In a possible implementation, the first processing result and the second processing result are at least one of the following: a denoising score (which can also be referred to as a score function of the first data distribution), enhanced data of the first data, or predicted noise.

[0203] In a possible implementation, the specific structure of the diffusion model is not limited by the embodiments of the present application.

[0204] 503, respectively, the first processing result and the second processing result are added with noise to obtain a first added-noise result and a second added-noise result;

[0205] 504, the first added-noise result is denoised by the first model to obtain a first denoised result;

[0206] 505, the second added-noise result is denoised by the first model to obtain a second denoised result;

[0207] In a possible implementation, the outputs of the two models can be re-added with noise, and the consistency score distillation method for high-frequency enhancement proposed in the embodiments of the present application can be used to further improve the performance of the student model.

[0208] In a possible implementation, the first processing result and the second processing result can be added with noise of a target step to obtain a first added-noise result and a second added-noise result. Similarly, the first added-noise result can be denoised by the first model with the target step to obtain a first denoised result, and the second added-noise result can be denoised by the first model with the target step to obtain a second denoised result. That is, the same step (t) is used in the noise adding process and the denoising process.

[0209] In a possible implementation, the target step is obtained by sampling in a preset step range, and steps included in the preset step range are all less than one fourth of a total denoising step.

[0210] As shown in the second row in FIG. 7, the denoising scores of the teacher model and the student model are significantly different mainly at small time steps. Since the diffusion model mainly focuses on high-frequency details of an image at small time steps, this indicates that the image generated by the student model mainly lacks high-frequency details compared with the image generated by the teacher model. Therefore, the present application proposes to calculate the difference between the denoising scores of the outputs of the teacher model and the student model under the interference of slight noise to optimize the student model, thereby enhancing the ability of the student model to accurately capture and reproduce high-frequency details. Specifically, the embodiments of the present application apply slight noise (that is, noise added with a small step) to disturb the outputs of the student model and the teacher model, and model the data distribution of the outputs (that is, predict the score function thereof) by the teacher model.

[0211] 506. Based on the first denoising result and the second denoising result, obtain the first loss, and update the second model based on the first loss.

[0212] Existing score distillation sampling methods inherently introduce meaningless gradient directions into the student model (since the teacher model processes the noisy output of the student model, which struggles to accurately predict the noise added to the student model's output), leading to problems such as blurred output and oversaturation. Existing distribution matching distillation methods require training a new diffusion model to learn the score function of samples generated by the student model and updating the generator based on the difference between the score function predicted by the new model and the score function predicted by the teacher model. However, this approach involves a complex training process, requiring alternating training between the student model and the new diffusion model.

[0213] In this embodiment, in order to simply and effectively eliminate the bias (existing in score distillation sampling) and provide a clear gradient direction for the optimization of the student model, the diffusion model, acting as the teacher, processes the noisy output of the student model while simultaneously processing the original image (consistent with the student model). Then, the teacher model is used to process the noisy output of the teacher model, and the two results obtained from the teacher model are used to construct the loss and update the student model. Thus, without introducing a new model, no meaningless gradient direction is introduced into the student model, thereby improving the quality of the distilled student model.

[0214] The method described in this application is primarily used to accelerate the sampling efficiency of diffusion models. One application framework of this application, as shown in Figure 6, mainly includes a teacher model, a student model, and a discriminator (including a time modulation part and a discriminator head). During training, the single-step output of the student model and the multi-step output of the teacher model are first constrained, transferring the knowledge of the teacher model to the student model (as shown in the upper left part of Figure 6). Subsequently, slight noise is introduced into the samples generated by the student and teacher models. The teacher model predicts the score function (or other forms of output), and the difference between the two score functions is used to guide the optimization of the student network (as shown in the upper right part of Figure 6). Furthermore, to alleviate the performance constraints of the teacher model on the student model, this application designs a time-aware discriminator based on the pre-trained teacher model, enhancing the perceptual quality of generated samples through adversarial training in the latent space (as shown in the lower part of Figure 6).

[0215] Unlike score distillation sampling (a in FIG. 8), the present application does not calculate the gradient between the output of the student model and the output of the teacher model after being predicted by the teacher model, avoiding the bias caused by the diffusion model itself. Unlike distribution matching distillation (b in FIG. 8), the present application does not need to train a new diffusion model to eliminate the inevitable bias between the predicted value of the diffusion model and the true value. The present application takes full advantage of the potential of the teacher model, optimizes the student model by calculating the score difference between the outputs of the teacher model and the student model under noise disturbance, and eliminates the sampling bias of score distillation and reduces the computational overhead. In addition, compared with the above two technologies, the present application can enhance the ability of the student model to generate high-frequency details of the image by sampling noise at a small time step.

[0216] In a possible implementation, second data corresponding to the first data can also be obtained, the second data being true value data; the second data is added with noise to obtain a third noise-added result; first and second features are obtained respectively by the encoding network in the first model according to the third noise-added result and the second noise-added result; first and second discrimination results are obtained respectively by the discriminator according to the first and second features; a second loss is obtained according to the first and second discrimination results, and the second model is updated according to the second loss.

[0217] In a possible implementation, the target step is mapped by a mapping network to obtain a first parameter; the first parameter is fused with the first and second features respectively to obtain first and second fusion results.

[0218] First and second discrimination results are obtained respectively by the discriminator according to the first and second fusion results.

[0219] In a possible implementation, the first feature includes first and second sub-features, the first and second sub-features being features of different scales in the first feature; the first parameter includes first and second sub-parameters, the first and second sub-parameters being different; the first sub-parameter is fused with the first sub-feature, and the second sub-parameter is fused with the second sub-feature to obtain the first fusion result.

[0220] In a possible implementation, the second feature includes a third sub-feature and a fourth sub-feature; the third sub-feature and the fourth sub-feature are features of different scales in the second feature; the first parameter includes a first sub-parameter and a second sub-parameter; the first sub-parameter and the second sub-parameter are different; the first sub-parameter and the third sub-feature are fused, and the second sub-parameter and the fourth sub-feature are fused to obtain a second fusion result.

[0221] In a possible implementation, the target step can be mapped by a mapping network to obtain a first parameter; the first parameter is fused with the first feature and the second feature respectively to obtain a first fusion result and a second fusion result.

[0222] According to the first fusion result and the second fusion result, a first discrimination result and a second discrimination result are obtained respectively by a discriminator.

[0223] In a possible implementation, the first feature includes a first sub-feature and a second sub-feature; the first sub-feature and the second sub-feature are features of different scales in the first feature; the first parameter includes a first sub-parameter and a second sub-parameter; the first sub-parameter and the second sub-parameter are different; the first sub-parameter and the first sub-feature can be fused, and the second sub-parameter and the second sub-feature can be fused to obtain a first fusion result.

[0224] In a possible implementation, the second feature includes a third sub-feature and a fourth sub-feature; the third sub-feature and the fourth sub-feature are features of different scales in the second feature; the first parameter includes a first sub-parameter and a second sub-parameter; the first sub-parameter and the second sub-parameter are different; the first sub-parameter and the third sub-feature are fused, and the second sub-parameter and the fourth sub-feature are fused to obtain a second fusion result.

[0225] In a possible implementation, according to the first discrimination result and the second discrimination result, a second loss is obtained, and the second model, the mapping network, and the discriminator are updated according to the second loss.

[0226] The prior art can distinguish the distribution of the synthesized data and the disturbed distribution of the real data in the latent space, however, it ignores the strong correlation between the output of the diffusion model and the time step, resulting in limited performance of the discriminator. The embodiment of the present application utilizes the pre-trained teacher model to design a time-aware discriminator. As shown in FIG. 9A, in the training stage, the real image and the output of the student network are re-noised in the latent space and input into the pre-trained teacher model to extract multi-scale features. It is worth noting that the information contained in these features is highly related to the time step of the noise, so the present embodiment designs a time modulation module to map the time information into two groups of learnable parameters β k ,γ k (corresponding to weights and biases, respectively), which are used to modulate the corresponding multi-scale features, adaptively adjusting the information intensity derived from the features. Finally, the features of each scale are input into an independent discrimination head for discrimination, and the average result of all discrimination heads is output.

[0227] The discriminator in the embodiment of the present application discriminates between synthesized data and real data in the latent space. At the same time, unlike the method of directly using the features extracted by the diffusion model to discriminate data. Considering the strong correlation between the features (or output) extracted by the diffusion model and the time step, a time modulation module is designed to map the time information into embedding and modulate the corresponding multi-scale features, adaptively adjusting the information intensity derived from the features. Finally, the synthesized data and the real are discriminated according to the modulated features, so as to improve the data processing accuracy of the trained student model.

[0228] For example, the block diagram of the embodiment structure of the embodiment of the present application can be as shown in FIG. 9B. For a given diffusion model (i.e. teacher model), first initialize the student model with the same network structure using the teacher model. On this basis, input the original data into the two models, first align the single-step output of the student model and the multi-step output of the teacher model in the ordinary distillation manner. Subsequently, re-noise the outputs of the two models, and further improve the performance of the student model using the high-frequency enhanced consistency score distillation method proposed in the present application. In addition, the time-aware discriminator proposed in the present application is used to discriminate between synthesized data and target data, and supervision is provided to the student model through the way of generative adversarial training. In the embodiment of the present application, the diffusion model distillation for super-resolution is realized, but the scheme of the present application is not limited to this task.

[0229] There are mainly two kinds of super-resolution networks based on diffusion model: one is a super-resolution network trained from scratch through diffusion loss, and the other is a super-resolution network based on fine-tuning. When the first kind of super-resolution network is used as a teacher model, in the training process, the student model is initialized through the weights of the teacher model, and is trained through the ordinary distillation loss and the two losses proposed in the application. The parameters of all student networks are updated by gradient.

[0230] In addition, when implementing high-frequency enhancement consistent score distillation, the difference between the teacher model and the student model in different degrees of noise can be calculated uniformly, or the difference between the de-noising scores at a specific noise intensity can be emphasized.

[0231] Next, the beneficial effects of the embodiments of the application will be introduced in combination with experiments.

[0232] On multiple given data sets of computer vision bottom-level tasks (general image super-resolution and blind face restoration), the embodiments of the application are compared with the latest general image super-resolution algorithm and blind face restoration algorithm, and have the following beneficial effects.

[0233] Table 1 is a comparison of objective indicators of different image super-resolution methods on the synthetic data set ImageNet-test of the general image super-resolution task, where the number after the method represents the sampling step number of the diffusion model, and * indicates that the result is obtained by reproducing the original method.

[0234] Table 1

[0235] Table 2 is a comparison of objective indicators of different image super-resolution methods on two real data sets of the general image super-resolution task.

[0236] Table 2

[0237] Table 3 is a comparison of objective indicators of different face restoration methods on the synthetic data set CelebA-test of the blind face restoration task.

[0238] Table 3

[0239] Table 4 is a comparison of objective indicators of different face restoration methods on three real data sets of the blind face restoration task.

[0240] Table 4

[0241] In the experiment of the general image super-resolution task, the method has performance equivalent to the most advanced method on the synthetic data set and multiple real data sets, and is significantly better than the existing method in the index close to human subjective perception.

[0242] In the experiment of blind face restoration, the method has performance equivalent to the most advanced method on the synthetic data set and multiple real face data sets.

[0243] In terms of subjective visual effect, the method is superior to the existing method in generating animal hair, human face texture and other fine details.

[0244] Compared with the existing super-resolution method based on the diffusion model, the method can generate a high-resolution image through only a single-step inference, greatly improving the inference efficiency of the model.

[0245] Table 5 is an ablation experiment.

[0246] Table 5

[0247] As shown in Table 5, compared with (a) and (b), the consistency score distillation strategy of high-frequency enhancement in the embodiment of the application guides the optimization of the student model by calculating the difference between the output of the teacher model and the output of the student model under slight noise disturbance, and improves the performance of the student model.

[0248] Compared with (c) and (d) in Table 5, by using the time-aware discriminator to distinguish between synthetic data and real data (t-GAN) in the embodiment of the application, the performance of the model is significantly improved while reducing the cost of generative adversarial training.

[0249] Referring to FIG. 10, FIG. 10 is a structural schematic of a data processing apparatus provided by the embodiment of the application, as shown in FIG. 10, the data processing apparatus provided by the embodiment of the application comprises:

[0250] The acquisition module 1001 is configured to acquire first data.

[0251] The specific description of the acquisition module 1001 can refer to the description of step 501 in the above-mentioned embodiments, which will not be repeated here.

[0252] The processing module 1002 is configured to obtain first processing result and second processing result by the first model and the second model respectively according to the first data, wherein the first model and the second model are diffusion models, the first model is a teacher model, and the second model is a student model.

[0253] The first processing result and the second processing result are respectively added with noise to obtain a first noise-added result and a second noise-added result.

[0254] The first noise-added result is denoised by the first model to obtain a first denoised result.

[0255] The second noise-added result is denoised by the first model to obtain a second denoised result.

[0256] A first loss is obtained according to the first denoised result and the second denoised result, and the second model is updated according to the first loss.

[0257] The specific description of the processing module 1002 can refer to the description of steps 502 to 506 in the above embodiments, and will not be repeated here.

[0258] In a possible implementation, the first processing result and the second processing result are at least one of the following:

[0259] The denoising score, the data enhanced from the first data, or the predicted noise.

[0260] In a possible implementation, the processing module is specifically configured to:

[0261] The first processing result and the second processing result are respectively added with noise of a target step to obtain a first noise-added result and a second noise-added result.

[0262] The first noise-added result is denoised by the first model with the target step to obtain a first denoised result, and the second noise-added result is denoised by the first model with the target step to obtain a second denoised result.

[0263] In a possible implementation, the target step is obtained by sampling in a preset step range, and steps included in the preset step range are all less than one fourth of a total denoising step.

[0264] In a possible implementation, the obtaining module is further configured to:

[0265] Obtain second data, the second data being true value data corresponding to the first data.

[0266] The processing module is further configured to:

[0267] Add noise to the second data to obtain a third noise-added result.

[0268] According to the third noise-added result and the second noise-added result, a first feature and a second feature are respectively obtained by an encoding network in the first model.

[0269] According to the first feature and the second feature, a first discrimination result and a second discrimination result are obtained respectively by a discriminator;

[0270] According to the first discrimination result and the second discrimination result, a second loss is obtained, and the second model is updated according to the second loss.

[0271] In a possible implementation, the processing module is specifically configured to:

[0272] The target step is mapped by a mapping network to obtain a first parameter;

[0273] The first parameter is fused with the first feature and the second feature respectively to obtain a first fusion result and a second fusion result;

[0274] According to the first fusion result and the second fusion result, a first discrimination result and a second discrimination result are obtained respectively by a discriminator.

[0275] In a possible implementation, the first feature includes a first sub-feature and a second sub-feature; the first sub-feature and the second sub-feature are features with different scales in the first feature;

[0276] The first parameter includes a first sub-parameter and a second sub-parameter; the first sub-parameter and the second sub-parameter are different;

[0277] The processing module is specifically configured to:

[0278] The first sub-parameter and the first sub-feature are fused, and the second sub-parameter and the second sub-feature are fused to obtain a first fusion result.

[0279] In a possible implementation, the second feature includes a third sub-feature and a fourth sub-feature; the third sub-feature and the fourth sub-feature are features with different scales in the second feature;

[0280] The first parameter includes a first sub-parameter and a second sub-parameter; the first sub-parameter and the second sub-parameter are different;

[0281] The processing module is specifically configured to:

[0282] The first sub-parameter and the third sub-feature are fused, and the second sub-parameter and the fourth sub-feature are fused to obtain a second fusion result.

[0283] In a possible implementation, the processing module is specifically configured to:

[0284] According to the first discrimination result and the second discrimination result, a second loss is obtained, and the second model, the mapping network and the discriminator are updated according to the second loss.

[0285] In a possible implementation, the first data is an image, and the first processing result and the second processing result are enhanced images.

[0286] Next, a terminal device provided by an embodiment of the present application is introduced. Referring to FIG. 11, FIG. 11 is a structural schematic diagram of a terminal device provided by an embodiment of the present application. The terminal device 1100 can specifically be a virtual reality (VR) device, a mobile phone, a tablet computer, a notebook computer, a smart wearable device, etc., which is not limited herein. Specifically, the terminal device 1100 includes a receiver 1101, a transmitter 1102, a processor 1103 and a memory 1104 (wherein the number of processors 1103 in the terminal device 1100 can be one or more, and one processor is taken as an example in FIG. 11). The processor 1103 can include an application processor 11031 and a communication processor 11032. In some embodiments of the present application, the receiver 1101, the transmitter 1102, the processor 1103 and the memory 1104 can be connected through a bus or other means.

[0287] The memory 1104 can include a read-only memory and a random access memory, and provide the processor 1103 with instructions and data. A part of the memory 1104 can also include a non-volatile random access memory (NVRAM). The memory 1104 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the operation instructions can include various operation instructions for implementing various operations.

[0288] The processor 1103 controls the operation of the execution device. In a specific application, various components of the execution device are coupled together through a bus system, wherein the bus system can include a data bus, a power supply bus, a control bus and a state signal bus, etc. in addition to the data bus. However, in order to clearly illustrate, various buses are referred to as a bus system in the figure.

[0289] The method disclosed in the embodiments of the present application can be applied to the processor 1103 or implemented by the processor 1103. The processor 1103 can be an integrated circuit chip having a signal processing capability. In the implementation process, the steps of the above method can be completed by hardware integrated logic circuits in the processor 1103 or by instructions in the form of software. The processor 1103 described above can be a general processor, a digital signal processor (DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The processor 1103 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the storage 1104, and the processor 1103 reads the information in the storage 1104 and combines the hardware to complete the steps related to model training or model inference process in the above method.

[0290] The receiver 1101 can be used to receive input digital or character information, and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1102 can be used to output digital or character information through the first interface; the transmitter 1102 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1102 can also include a display device such as a display screen.

[0291] The embodiment of the present application further provides a server. Please refer to FIG. 12, which is a structural schematic diagram of the server provided by the embodiment of the present application. The server 1200 can be quite different due to different configurations or performances, and can include one or more central processing units (CPUs) 1212 (for example, one or more processors) and a memory 1232, one or more storage media 1230 (for example, one or more mass storage devices) for storing application programs 1242 or data 1244. The memory 1232 and the storage media 1230 can be temporary storage or persistent storage. The programs stored in the storage media 1230 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the server. Furthermore, the central processing unit 1212 can be configured to communicate with the storage media 1230 and execute the series of instruction operations in the storage media 1230 on the server 1200.

[0292] The server 1200 can further include one or more power supplies 1226, one or more wired or wireless network interfaces 1250, one or more input and output interfaces 1258, or one or more operating systems 1241, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0293] In the embodiment of the present application, the central processing unit 1212 is configured to perform the actions related to model training or model inference in the above embodiments.

[0294] The embodiment of the present application further provides a computer program product, which, when running on a computer, causes the computer to perform the steps performed by the above execution device, or causes the computer to perform the steps performed by the above training device.

[0295] The embodiment of the present application further provides a computer readable storage medium, which stores a program for signal processing, and when running on a computer, causes the computer to perform the steps performed by the above execution device, or causes the computer to perform the steps performed by the above training device.

[0296] The execution device, the training device or the terminal device provided in the embodiments of the present application can specifically be a chip, which comprises a processing unit, for example, a processor, and a communication unit, for example, an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the execution device executes the data processing method described in the above embodiments, or so that the chip in the training device executes the data processing method described in the above embodiments. Alternatively, the storage unit is a storage unit in the chip, such as a register, a cache, etc., and the storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0297] Specifically, please refer to FIG. 13, which is a structural schematic diagram of a chip provided in the embodiments of the present application. The chip can be represented as a neural network processor NPU 1300, which is mounted on a host CPU (Host CPU) as a coprocessor and is assigned tasks by the Host CPU. The core part of the NPU is an operation circuit 1303, which extracts matrix data in a memory and performs multiplication operation under the control of a controller 1304.

[0298] In some implementations, the operation circuit 1303 internally includes a plurality of processing units (PEs). In some implementations, the operation circuit 1303 is a two-dimensional systolic array. The operation circuit 1303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1303 is a general-purpose matrix processor.

[0299] For example, it is assumed that there are an input matrix A, a weight matrix B and an output matrix C. The operation circuit takes corresponding data of the matrix B from the weight memory 1302 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1301 and performs matrix operation with the matrix B, and the partial result or final result of the obtained matrix is saved in an accumulator 1308.

[0300] The unified memory 1306 is used to store input data and output data. The weight data is transferred to the weight memory 1302 through a direct memory access controller (DMAC) 1305. The input data is also transferred to the unified memory 1306 through the DMAC.

[0301] BIU for Bus Interface Unit, i.e., the bus interface unit 1310 is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 1309.

[0302] The bus interface unit 1310 (Bus Interface Unit, referred to as BIU) is used for the instruction fetch buffer 1309 to obtain instructions from the external memory, and is also used for the storage unit access controller 1305 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0303] The DMAC is mainly used to carry the input data in the external memory DDR to the unified memory 1306, or carry the weight data to the weight memory 1302, or carry the input data to the input memory 1301.

[0304] The vector calculation unit 1307 includes a plurality of operation processing units, which further process the output of the operation circuit 1303 as needed, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolution / full connection layer network calculation in neural network, such as batch normalization, pixel-level summation, upsampling of feature plane, etc.

[0305] In some implementations, the vector calculation unit 1307 can store the processed output vector to the unified memory 1306. For example, the vector calculation unit 1307 can apply a linear function; or, a nonlinear function to the output of the operation circuit 1303, such as linear interpolation on the feature plane extracted by the convolution layer, and for example, a vector of accumulated values to generate an activation value. In some implementations, the vector calculation unit 1307 generates normalized values, pixel-level summation values, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1303, for example, for use in subsequent layers in the neural network.

[0306] The controller 1304 is connected to the instruction fetch buffer 1309, which is used to store instructions used by the controller 1304;

[0307] The unified memory 1306, the input memory 1301, the weight memory 1302, and the instruction fetch buffer 1309 are all on-chip memories. The external memory is private to the NPU hardware architecture.

[0308] The processor mentioned in any of the above can be a general central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling execution of the above programs.

[0309] It should be noted that the above-described apparatus embodiments are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the apparatus embodiments provided in the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0310] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software and the necessary general hardware, and of course it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. Generally, functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better embodiment. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., including a plurality of instructions for making a computer device (which can be a personal computer, training device, or network device, etc.) execute the methods described in various embodiments of the present application.

[0311] In the above embodiments, all or part can be realized by software, hardware, firmware or any combination thereof. When realized by software, it can be realized in the form of a computer program product in whole or in part.

[0312] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk (SSD)), etc.

Claims

1. A data processing method, characterized by, The method comprises: obtaining first data; obtaining first processing results and second processing results through first and second models respectively according to the first data, wherein the first and second models are diffusion models, the first model is a teacher model, and the second model is a student model; adding noise to the first and second processing results respectively to obtain first and second noise-added results; denoising the first noise-added result through the first model to obtain first denoised results; denoising the second noise-added result through the first model to obtain second denoised results; obtaining a first loss according to the first and second denoised results, and updating the second model according to the first loss.

2. The method of claim 1, wherein, The first and second processing results are at least one of the following: de-noising scores, data enhanced from the first data, or predicted noise.

3. The method according to claim 1 or 2, characterized in that, The adding noise to the first and second processing results respectively to obtain first and second noise-added results comprises: adding noise to the first and second processing results respectively by a target step to obtain first and second noise-added results. The denoising the first noise-added result through the first model to obtain first denoised results, and denoising the second noise-added result through the first model to obtain second denoised results comprises: denoising the first noise-added result through the first model by the target step to obtain first denoised results, and denoising the second noise-added result through the first model by the target step to obtain second denoised results.

4. The method according to any one of claims 1 to 3, characterized in that, The target step is obtained by sampling in a preset step range, and steps contained in the preset step range are all less than one fourth of a total denoising step.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: obtaining second data, wherein the second data is true value data corresponding to the first data; adding noise to the second data to obtain third noise-added results; obtaining first and second features through an encoding network in the first model according to the third and second noise-added results respectively; obtaining first and second discrimination results through a discriminator according to the first and second features respectively; obtaining a second loss according to the first and second discrimination results, and updating the second model according to the second loss.

6. The method of claim 5, wherein, The obtaining first and second discrimination results through a discriminator according to the first and second features respectively comprises: mapping the target step through a mapping network to obtain first parameters; fusing the first parameters with the first and second features respectively to obtain first and second fusion results; obtaining first and second discrimination results through a discriminator according to the first and second fusion results respectively.

7. The method of claim 6, wherein, The first feature comprises first and second sub-features, and the first and second sub-features are features of different scales in the first feature. The first parameter comprises a first sub-parameter and a second sub-parameter; the first sub-parameter and the second sub-parameter are different; The first parameter is fused with the first feature and the second feature respectively to obtain a first fusion result and a second fusion result, and the first fusion result and the second fusion result are obtained. The first sub-parameter and the first sub-feature are fused, and the second sub-parameter and the second sub-feature are fused to obtain the first fusion result.

8. The method according to claim 6 or 7, characterized in that, The second feature comprises a third sub-feature and a fourth sub-feature; the third sub-feature and the fourth sub-feature are different in scale. The first parameter comprises a first sub-parameter and a second sub-parameter; the first sub-parameter and the second sub-parameter are different; The first parameter is fused with the first feature and the second feature respectively to obtain a first fusion result and a second fusion result, and the first fusion result and the second fusion result are obtained. The first sub-parameter and the third sub-feature are fused, and the second sub-parameter and the fourth sub-feature are fused to obtain the second fusion result.

9. The method according to any one of claims 6 to 8, characterized in that, The second loss is obtained according to the first discrimination result and the second discrimination result, and the second model, the mapping network and the discriminator are updated according to the second loss. The first data is an image, and the first processing result and the second processing result are enhanced images.

10. The method according to any one of claims 1 to 9, characterized in that, The device comprises:

11. A data processing apparatus, characterized by An acquisition module is configured to acquire first data. A processing module is configured to obtain a first processing result and a second processing result by using a first model and a second model respectively according to the first data, wherein the first model and the second model are diffusion models, the first model is a teacher model, and the second model is a student model. The first processing result and the second processing result are respectively added with noise to obtain a first noise-added result and a second noise-added result. The first noise-added result is denoised by using the first model to obtain a first denoised result. The second noise-added result is denoised by using the first model to obtain a second denoised result. A first loss is obtained according to the first denoised result and the second denoised result, and the second model is updated according to the first loss. The first processing result and the second processing result are at least one of the following:

12. The apparatus of claim 11, wherein, Denoising scores, data enhanced from the first data, or predicted noise. The processing module is specifically configured to:

13. The apparatus of claim 11 or 12, wherein, The first processing result and the second processing result are respectively added with noise of a target step to obtain a first noise-added result and a second noise-added result. The first noise-added result is denoised by using the first model to obtain a first denoised result, and the second noise-added result is denoised by using the first model to obtain a second denoised result. The target step is obtained by sampling in a preset step range, and steps contained in the preset step range are all less than one fourth of a total denoising step.

14. The apparatus of any one of claims 11 to 13, wherein, The acquisition module is further configured to:

15. The apparatus of any one of claims 11 to 14, wherein, ​ obtain second data, the second data being ground truth data corresponding to the first data; the processing module is further configured to: add noise to the second data to obtain a third noise-added result; obtain first features and second features respectively through an encoding network in the first model according to the third noise-added result and the second noise-added result; obtain first discrimination results and second discrimination results respectively through a discriminator according to the first features and the second features; obtain a second loss according to the first discrimination results and the second discrimination results, and update the second model according to the second loss.

16. The apparatus of claim 15, wherein, the processing module is specifically configured to: map the target step through a mapping network to obtain first parameters; fuse the first parameters with the first features and the second features respectively to obtain first fusion results and second fusion results; fuse the second parameters with the second features to obtain second fusion results; obtain first discrimination results and second discrimination results respectively through a discriminator according to the first fusion results and the second fusion results.

17. The apparatus of claim 16, wherein, the first features include first sub-features and second sub-features; the first sub-features and the second sub-features are features of different scales in the first features; the first parameters include first sub-parameters and second sub-parameters; the first sub-parameters and the second sub-parameters are different; the processing module is specifically configured to: fuse the first sub-parameters with the first sub-features and fuse the second sub-parameters with the second sub-features to obtain first fusion results.

18. The apparatus of claim 16 or 17, wherein, the second features include third sub-features and fourth sub-features; the third sub-features and the fourth sub-features are features of different scales in the second features; the first parameters include first sub-parameters and second sub-parameters; the first sub-parameters and the second sub-parameters are different; the processing module is specifically configured to: fuse the first sub-parameters with the third sub-features and fuse the second sub-parameters with the fourth sub-features to obtain second fusion results.

19. The apparatus of any one of claims 16 to 18, wherein, the processing module is specifically configured to: obtain a second loss according to the first discrimination results and the second discrimination results, and update the second model, the mapping network and the discriminator according to the second loss.

20. The apparatus of any one of claims 11 to 19, wherein, the first data is an image, and the first processing result and the second processing result are enhanced images.

21. A computer storage medium, comprising, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1-10.

22. A computer program product, characterised in that, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operations of the method of any one of claims 1-10.

23. A system comprising at least one processor, at least one memory; the processor, the memory are connected through a communication bus and complete mutual communication; the at least one memory is used to store codes; The at least one processor is configured to execute the code to perform the method of any one of claims 1-10.

24. A chip comprising a processor, wherein: The processor is configured to support a data processing device to implement the method of any one of claims 1-10.

Images