A data processing method and apparatus

Abstract

A data processing method, the method comprising: acquiring a plurality of first images; wherein the plurality of first images are images obtained by performing image enhancement on a second image multiple times; the semantic content of each first image includes at least one object including a first object; acquiring a first image region and a second image region of the first object in the plurality of first images, the first image region and the second image region belong to different first images, and the image enhancement effect of the first image region is higher than that of the second image region; taking the second image as a conditional input of a target model, taking the first image region and the second image region as model inputs of the target model, obtaining a noise prediction result of the first image region and the second image region, and updating the target model according to a loss function constructed according to the noise prediction result. The embodiments of the present application can improve the enhancement quality of the image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a data processing method and apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] As a novel generative model, diffusion models have attracted increasing attention due to their ability to model complex data distributions, and are gradually replacing generative adversarial models in various downstream tasks.

[0004] Diffusion models have made significant progress in image enhancement tasks such as real-world image super-resolution. These models typically rely on supervised training, learning from paired low-quality and high-quality images, ignoring human feedback. This leads to outputs that are inconsistent with human perceptual preferences, producing harmful content, illusions, and visual artifacts. Direct preference optimization (DPO) is widely used due to its low computational cost and lack of reward-free modeling. While DPO performs well in generation tasks as a reward-free optimization method, its focus on precise reconstruction of every pixel can lead to localized spurious generation, degrading the quality of the generated images. Summary of the Invention

[0005] This application provides a data processing method for improving the quality of generated images.

[0006] In a first aspect, this application provides a data processing method, the method comprising: acquiring a plurality of first images; wherein the plurality of first images are images obtained by performing multiple image enhancements on a second image; the semantic content of each first image includes at least one object, including a first object; acquiring a first image region and a second image region of the first object in the plurality of first images, wherein the first image region and the second image region belong to different first images, and the image enhancement effect of the first image region is higher than that of the second image region; using the second image as a conditional input to a target model, using the first image region and the second image region as model inputs to the target model, obtaining noise prediction results for the first image region and the second image region, and updating the target model based on a loss function constructed from the noise prediction results; wherein the prediction error corresponding to the noise prediction result of the first image region is used as a reward term in the loss function, and the prediction error corresponding to the noise prediction result of the second image region is used as a penalty term in the loss function.

[0007] This application's embodiments overcome the limitations of traditional direct preference optimization under pixel-level reconstruction targets by using instance-level (i.e., the granularity of the semantic objects of the image) alignment, effectively suppressing artifacts and illusions in the generation process, thereby improving the enhancement quality of the image.

[0008] The reward term corresponds to the data fitting part of the loss function, reflecting the difference between the model's prediction and the true label. Essentially, it "positively guides" the model to approximate the true distribution. The penalty term is an additional term in the loss function used to constrain model complexity, aiming to suppress overfitting and improve generalization performance. By using the prediction error corresponding to the noise prediction result of the first image region as the reward term in the loss function, the model can learn the ability to predict the noise distribution of the first image region. Conversely, by using the prediction error corresponding to the noise prediction result of the second image region as the penalty term in the loss function, the model can learn the ability not to predict the noise distribution of the second image region.

[0009] In one possible implementation, the first image region and the second image region are selected based on user preferences. For example, the user may select them from multiple image regions of the first object, or the selection may be made by a model (or other rule-based algorithm) that has the ability to select images based on user preferences.

[0010] In one possible implementation, the first image region is the image region in which the first object has the best image enhancement effect among the plurality of first images, and the second image region is the image region in which the first object has the worst image enhancement effect among the plurality of first images.

[0011] The selection of the first and second image regions can be made based on user preferences. Through a direct instance preference alignment algorithm and a human text description preference alignment mechanism, the generated content is ensured to be highly consistent with human preferences.

[0012] In one possible implementation, the image enhancement is specifically super-resolution.

[0013] In one possible implementation, the prediction error corresponding to the noise prediction result of the first image region is obtained based on the difference between a first difference and a second difference; the first difference is the difference between the noise prediction result and the ground truth value of the first image region; the second difference is the difference between the noise prediction result obtained by using the first image region as the model input and the ground truth value; or...

[0014] The prediction error corresponding to the noise prediction result of the second image region is obtained based on the difference between the third difference and the fourth difference; the third difference is the difference between the noise prediction result and the true value of the second image region; the fourth difference is the difference between the noise prediction result and the true value obtained by using the second image region as the model input of the reference model.

[0015] In one possible implementation, the method further includes: obtaining user feedback information on the image enhancement effect of each object in the image region of the plurality of first images; the step of using the second image as a conditional input to the target model, using the first image region and the second image region as model inputs to the target model, and performing noise prediction through the target model includes: using the second image and a prompt obtained according to the feedback information as conditional inputs to the target model, using the first image region and the second image region as model inputs to the target model, and performing noise prediction through the target model.

[0016] The feedback information can be text data. By analyzing the semantic information in the user feedback, it can be injected into the generation process as a prompt. The model can better understand the user's expectations and preferences, thereby reducing hallucinations and enhancing semantic consistency and detail when generating images.

[0017] In one possible implementation, the different first images are obtained by image enhancement of the second image using the same model under different parameter configurations.

[0018] In one possible implementation, the first image region and the second image region are obtained by image segmentation of the plurality of first images.

[0019] Secondly, this application provides a data processing apparatus, the apparatus comprising:

[0020] An acquisition module is used to acquire multiple first images; wherein the multiple first images are images obtained by performing multiple image enhancements on a second image; the semantic content of each first image includes at least one object, including a first object;

[0021] A semantic segmentation module is used to obtain a first image region and a second image region of the first object in the plurality of first images, wherein the first image region and the second image region belong to different first images, and the image enhancement effect of the first image region is higher than the image enhancement effect of the second image region.

[0022] The model training method uses the second image as the conditional input of the target model, and the first image region and the second image region as the model input of the target model to obtain the noise prediction results of the first image region and the second image region. The target model is then updated based on the loss function constructed from the noise prediction results. The prediction error corresponding to the noise prediction result of the first image region is used as the reward term in the loss function, and the prediction error corresponding to the noise prediction result of the second image region is used as the penalty term in the loss function.

[0023] In one possible implementation, the first image region is the image region in which the first object has the best image enhancement effect among the plurality of first images, and the second image region is the image region in which the first object has the worst image enhancement effect among the plurality of first images.

[0024] In one possible implementation, the image enhancement is specifically super-resolution.

[0025] In one possible implementation, the prediction error corresponding to the noise prediction result of the first image region is obtained based on the difference between a first difference and a second difference; the first difference is the difference between the noise prediction result and the ground truth value of the first image region; the second difference is the difference between the noise prediction result obtained by using the first image region as the model input and the ground truth value; or...

[0026] The prediction error corresponding to the noise prediction result of the second image region is obtained based on the difference between the third difference and the fourth difference; the third difference is the difference between the noise prediction result and the true value of the second image region; the fourth difference is the difference between the noise prediction result and the true value obtained by using the second image region as the model input of the reference model.

[0027] In one possible implementation, the acquisition module is further configured to:

[0028] Obtain user feedback on the image enhancement effect of each object in the image region of the plurality of first images;

[0029] The model is used to train the model for:

[0030] The second image and the prompt obtained based on the feedback information are used as the conditional inputs to the target model, and the first image region and the second image region are used as the model inputs to the target model. Noise prediction is then performed using the target model.

[0031] In one possible implementation, the different first images are obtained by image enhancement of the second image using the same model under different parameter configurations.

[0032] In one possible implementation, the first image region and the second image region are obtained by image segmentation of the plurality of first images.

[0033] Thirdly, embodiments of this application provide a computing device that may include a memory, a processor, and a bus system, wherein the memory is used to store a program, and the processor is used to execute the program in the memory to perform the methods described in the first aspect above and any of its optional methods.

[0034] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the methods described in the first aspect and any of its optional methods.

[0035] Fifthly, embodiments of this application provide a computer program that, when run on a computer, causes the computer to perform the first aspect and any of its optional methods described above.

[0036] Sixthly, this application provides a chip system including a processor for executing data processing devices to realize the functions involved in the foregoing aspects, such as transmitting or processing data involved in the foregoing methods; or, information. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the execution device or training device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of the application system framework of the present invention;

[0038] Figure 2 This is a schematic diagram of a system architecture according to this application;

[0039] Figure 3 A process for providing a cloud service;

[0040] Figure 4 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0041] Figure 5 This application provides an example of an application architecture.

[0042] Figure 6 A schematic diagram of a data processing method provided in an embodiment of this application;

[0043] Figure 7 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application;

[0044] Figure 8 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0045] Figure 9 A schematic diagram of the structure of the training device provided in the embodiments of this application;

[0046] Figure 10 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0047] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0048] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0049] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0050] The terms “substantially,” “about,” and similar terms used herein are used as approximations rather than as terms of degree, and are intended to take into account the inherent biases of measurements or calculations known to those skilled in the art. Furthermore, the use of “may” in describing embodiments of the invention refers to “one or more possible embodiments.” The terms “use,” “using,” and “used” used herein are to be considered synonymous with the terms “utilize,” “utilizing,” and “utilized,” respectively. Additionally, the term “exemplary” is intended to refer to an instance or illustration.

[0051] This application can be applied to the field of natural language processing in the field of artificial intelligence. The following will introduce several application scenarios that have been implemented in products, taking natural language processing as an example.

[0052] First, we will introduce the application scenarios of this application. This application can be applied, but is not limited to, to applications with image generation capabilities (hereinafter referred to as generation applications) or cloud services provided by cloud-side servers, etc., which will be introduced separately below:

[0053] I. Generating class applications

[0054] The product form of this application embodiment can be a generative application. Generative applications can run on terminal devices or cloud servers.

[0055] In one possible implementation, a generative application can perform image generation tasks and obtain the processed results. For example, text-based image generation tasks. Another example is image enhancement tasks based on images (including but not limited to super-resolution, denoising, and dehazing).

[0056] For example, generative applications can implement image generation tasks that include at least diffusion-based methods, but are not limited to this.

[0057] In one possible implementation, a user can open a generative application installed on a terminal device and input image data. The generative application can process the image using a model trained by the method provided in the embodiments of this application, or by the method provided in the embodiments of this application, and present the processing result to the user (the presentation method may include, but is not limited to, displaying, playing, saving, uploading to the cloud, etc.).

[0058] In one possible implementation, a user can open a generative application installed on a terminal device and input image data. The generative application can then send the image data to a cloud-based server. The cloud-based server processes the image using a model trained by the method provided in this application embodiment and sends the processing result back to the terminal device. The terminal device can then present the processing result to the user (the presentation method may include, but is not limited to, displaying, playing, saving, or uploading to the cloud).

[0059] The following sections will introduce the generated application class in this application embodiment from the perspectives of functional architecture and product architecture that implements the functions.

[0060] Reference Figure 1 , Figure 1 This is a schematic diagram of the functional architecture of the generated application class in the embodiments of this application:

[0061] In one possible implementation, such as Figure 1 As shown, the generative application 102 can receive input parameters 101 (e.g., including an image) and generate a processing result 103. The generative application 102 can be executed on at least one computer system (for example) and includes computer code that, when executed by one or more computers, causes the computers to execute a model trained by the methods provided in the embodiments of this application.

[0062] It should be understood that the steps related to the model inference process in the embodiments of this application involve AI-related operations. When performing AI operations, the instruction execution architecture of the terminal device and server is not limited to the processor-memory architecture described above. The following section will further explain... Figure 2 The system architecture provided in the embodiments of this application will be described in detail.

[0063] Figure 2 This is a schematic diagram of the system architecture provided for an embodiment of this application. Figure 2 As shown, the system architecture 500 includes an execution device 510, a training device 520, a database 530, a client device 540, a data storage system 550, and a data acquisition system 560.

[0064] The execution device 510 includes a calculation module 511, an I / O interface 512, a preprocessing module 513, and a preprocessing module 514. The calculation module 511 may include a target model / rule 501, while the preprocessing modules 513 and 514 are optional.

[0065] The execution device 510 can be a terminal device or a server that runs the aforementioned generation-type application.

[0066] The data acquisition device 560 is used to collect training samples. Training samples can be image data, etc. After collecting the training samples, the data acquisition device 560 stores these training samples in the database 530.

[0067] The training device 520 can maintain training samples in the database 530 and obtain the target model / rule 501 from the neural network to be trained (e.g., the neural network model in the embodiments of this application (e.g., including generative models (e.g., diffusion models)).

[0068] It should be understood that the training device 520 can perform a pre-training process on the neural network to be trained based on the training samples maintained in the database 530, or fine-tune the model based on the pre-training.

[0069] It should be noted that in practical applications, the training samples maintained in database 530 may not all come from the data acquisition device 560; they may also be received from other devices. Furthermore, it should be noted that training device 520 may not necessarily train the target model / rule 501 entirely based on the training samples maintained in database 530; it may also obtain training samples from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0070] The target model / rule 501 trained using training device 520 can be applied to different systems or devices, such as... Figure 2 The execution device 510 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server, etc.

[0071] Specifically, the training device 520 can transfer the trained model to the execution device 510.

[0072] exist Figure 2In the execution device 510, an input / output (I / O) interface 512 is configured for data interaction with external devices. Users can input data (such as image data in this embodiment) into the I / O interface 512 through the client device 540.

[0073] Preprocessing modules 513 and 514 are used to preprocess the input data received from the I / O interface 512. It should be understood that preprocessing modules 513 and 514 may be absent, or only one preprocessing module may be used. When preprocessing modules 513 and 514 are absent, the calculation module 511 can be used directly to process the input data.

[0074] During the preprocessing of input data by the execution device 510, or during the calculation module 511 of the execution device 510 performing calculations and other related processes, the execution device 510 can call data, code, etc. in the data storage system 550 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 550.

[0075] Finally, the I / O interface 512 provides the processing result to the client device 540, thereby providing it to the user.

[0076] exist Figure 2 In the illustrated scenario, the user can manually provide input data, which can be done through the interface provided by I / O interface 512. Alternatively, the client device 540 can automatically send input data to I / O interface 512. If user authorization is required for the client device 540 to automatically send input data, the user can set the corresponding permissions in the client device 540. The user can view the output results of the execution device 510 on the client device 540, which can be presented in various forms such as display, sound, or animation. The client device 540 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530. Alternatively, data can be collected directly from the I / O interface 512 without going through the client device 540, using the input data and output results of the input I / O interface 512 as shown in the figure, and storing them as new sample data in database 530.

[0077] It is worth noting that, Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2In this context, the data storage system 550 is an external storage device relative to the execution device 510. However, in other cases, the data storage system 550 may also be placed within the execution device 510. It should be understood that the aforementioned execution device 510 may be deployed within the client device 540.

[0078] The server provides image compositing functionality similar to cloud services.

[0079] In one possible implementation, the server can provide image compositing services to the client side via an application programming interface (API).

[0080] In this process, the terminal device can send relevant parameters (such as images, text, and other data) to the server through the API provided by the cloud. The server can then obtain the processing results based on the received parameters and return the processing results to the terminal.

[0081] The description of the terminal and server can be found in the above embodiments, and will not be repeated here.

[0082] like Figure 3 The process of using an image compositing function cloud service provided by a cloud platform is illustrated.

[0083] 1. Activate and purchase image compositing services.

[0084] 2. Users can download the software development kit (SDK) corresponding to the image compositing service. Cloud platforms usually provide multiple development versions of the SDK for users to choose from according to their development environment needs, such as JAVA version SDK, Python version SDK, PHP version SDK, Android version SDK, etc.

[0085] 3. After downloading the corresponding version of the SDK to their local machine according to their needs, users can import the SDK project into their local development environment, configure and debug it in the local development environment, and develop other functions in the local development environment to form an application that integrates image synthesis capabilities.

[0086] 4. When an application needs to perform image compositing, it can trigger an API call for the image compositing function. When the application triggers the image compositing function, it sends an API request to the running instance of the image compositing service in the cloud environment. The API request carries the image, which is then processed by the running instance in the cloud environment to obtain the processing result.

[0087] 5. The cloud environment returns the processing results to the application, thus completing one image compositing function call.

[0088] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0089] (1) Neural Network

[0090] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0091]

[0092] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0093] (2) Deep Neural Networks

[0094] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as...

[0095] The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows:

[0096] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0097] (3) Loss Function

[0098] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0099] (4) Backpropagation algorithm

[0100] Backpropagation (BP) can be used during training to correct the parameters in the initial model, thereby reducing the model's error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters in the initial model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining optimal model parameters, such as the weight matrix.

[0101] (5) Diffusion model

[0102] A diffusion model is a generative model used to generate data such as images and text. The core idea of ​​a diffusion model is to diffuse noise into the data and then gradually remove the noise to recover the original data. The diffusion model consists of two stages: a forward process (noise diffusion) and a reverse process (noise removal and recovery).

[0103] Diffusion models have made significant progress in image enhancement tasks such as real-world image super-resolution. These models typically rely on supervised training, learning from paired low-quality and high-quality images, ignoring human feedback. This leads to outputs that are inconsistent with human perceptual preferences, producing harmful content, illusions, and visual artifacts. Direct preference optimization (DPO) is widely used due to its low computational cost and lack of reward-free modeling. While DPO performs well in generation tasks as a reward-free optimization method, its focus on precise reconstruction of every pixel can lead to localized spurious generation, degrading the quality of the generated images.

[0104] The data processing method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0105] Reference Figure 4 , Figure 4 This is a flowchart illustrating a data processing method provided in an embodiment of this application, such as... Figure 4 As shown in the embodiment of this application, a data processing method may include steps 401 to 403, which are described in detail below.

[0106] 401. Obtain multiple first images; wherein the multiple first images are images obtained by performing multiple image enhancements on a second image; the semantic content of each first image includes at least one object, including a first object;

[0107] In this application, the image enhancement can be an enhancement that does not change the semantic content of the image, such as super-resolution, dehazing, changing the image style, color and contrast adjustment, smoothing, detail enhancement and sharpening, etc.

[0108] In one possible implementation, multiple image enhancements can be performed on the second image to obtain multiple first images.

[0109] The second image can be a low-quality image, which may be obtained under non-ideal imaging conditions (such as equipment limitations, environmental interference, etc.). The main reasons for this include, but are not limited to: noise interference, such as Gaussian noise introduced due to sensor performance or transmission problems; abnormal brightness: too dark (low illumination) or overexposed, leading to loss of detail; low contrast: concentrated grayscale distribution, insufficient distinction between target and background; low resolution: insufficient pixel density, leading to image blur or loss of detail, such as satellite remote sensing images or low-resolution face images; and blurring and artifacts: structural distortion caused by motion blur, lens shake, or defects in the reconstruction algorithm.

[0110] In one possible implementation, the second image can be enhanced using multiple models with different model structures, or the second image can be enhanced using the same model with different parameter configurations.

[0111] Each first image contains at least one object, including a first object. The first object can be a semantic object, which refers to an entity or region in the image that has a clear semantic meaning and is a visual element that can be recognized and understood by humans or computer vision systems. These objects can be concrete physical entities (such as people, vehicles, buildings, and animals) or abstract scenes or concepts (such as "indoor environment," "natural landscape," or "medical lesion area"). Through semantic segmentation, object detection, and other techniques, computers can classify pixels in an image into different semantic categories.

[0112] 402. Obtain the first image region and the second image region of the first object in the plurality of first images, wherein the first image region and the second image region belong to different first images, and the image enhancement effect of the first image region is higher than the image enhancement effect of the second image region;

[0113] It should be understood that each of the at least one objects has a corresponding image region in each first image. The embodiments of this application are described using the first object as an example.

[0114] In one possible implementation, the first object may have a corresponding image region in each first image, and the image region may be obtained by image segmentation of the plurality of first images. For example, a semantic extraction model can be used to extract instance-level semantic information from each first image.

[0115] In one possible implementation, a first image region and a second image region of the first object in the plurality of first images can be obtained, wherein the first image region and the second image region belong to different first images, and the image enhancement effect of the first image region is higher than that of the second image region. Taking super-resolution as an example, the image enhancement effect can be related to at least one of the following criteria:

[0116] High fidelity: Highly consistent with real high-resolution (HR) images at the pixel level, quantified by objective indicators such as peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM); Detail recovery capability: Able to reconstruct high-frequency information (such as texture and edges), for example, algorithms enhanced by gradient masks or wavelet transforms can clearly restore hair texture, building outlines, etc.; Whether it conforms to human subjective aesthetics; Cross-domain consistency: Consistent with the original scene in color, brightness, and contrast, for example, in medical images, it is necessary to accurately restore the grayscale distribution of lesion areas to avoid misdiagnosis caused by over-enhancement; Robustness and generalization: Still able to output high-quality results stably under complex degradation (such as noise and motion blur).

[0117] In one possible implementation, the first image region is the image region in which the first object has the best image enhancement effect among the plurality of first images, and the second image region is the image region in which the first object has the worst image enhancement effect among the plurality of first images.

[0118] Specifically, the user can select the image region with the best image enhancement effect from the image regions of the first object in multiple first images as the first image region, and the user can select the image region with the worst image enhancement effect from the image regions of the first object in multiple first images as the second image region.

[0119] In one possible implementation, user feedback on the image enhancement effect of each object (e.g., the first object) within the plurality of first images can also be obtained. This feedback information can be text data; by analyzing the semantic information in the user feedback, it can be injected as a cue into the generation process. This allows the model to better understand the user's expectations and preferences, thereby reducing illusions and enhancing semantic consistency and detail representation when generating images.

[0120] For specific details, please refer to Figure 6 The process of User Description Feedback Model is described in the text.

[0121] 403. Using the second image as the conditional input of the target model, and using the first image region and the second image region as the model input of the target model, noise prediction results for the first image region and the second image region are obtained, and the target model is updated based on the loss function constructed from the noise prediction results; wherein, the prediction error corresponding to the noise prediction result of the first image region is used as the reward term in the loss function, and the prediction error corresponding to the noise prediction result of the second image region is used as the penalty term in the loss function.

[0122] Among them, conditional input and model input (such as time steps, noisy images, etc.) are two key parameters that control the generation process. Their differences mainly lie in their mechanism of action and functional objectives:

[0123] Conditional inputs are additional information used to guide the generated content to meet specific requirements, typically injected into the model as external control signals. Examples include text descriptions, category labels, or local image features. Conditions can specify global attributes of the generated result (e.g., "generate a cat"), embedded into the model through text encoding or category labels. Furthermore, conditions can dynamically adjust local regions during generation (e.g., repairing specific objects in an image), achieved through classifier gradients or implicit conditional weights.

[0124] When used as conditional input, the vector of the conditional input can be concatenated with the noisy image along the feature dimension (Vanilla Guidance), or intermediate features of UNet can be injected through a multi-head attention layer (e.g., the interaction between text embeddings and image features).

[0125] The model input is a fundamental parameter essential for the diffusion process, typically including the number of time steps and the noise image. The number of time steps (t) defines the current step in forward diffusion or backward denoising, used to dynamically adjust the noise intensity and the model's prediction target. The noise image (x...)... t ): Intermediate results of progressive denoising during the reverse process. Optionally, time step t is embedded into the model through positional encoding (such as a sine function or a learnable vector) to guide the model to adjust the denoising strategy at different stages.

[0126] In this embodiment, the second image can be used as the conditional input of the target model, and the first image region and the second image region can be used as the model input of the target model to obtain the noise prediction results of the first image region and the second image region. The target model is then updated based on the loss function constructed from the noise prediction results.

[0127] In one possible implementation, the second image and the prompt obtained based on the feedback information can be used as conditional inputs to the target model, and the first image region and the second image region can be used as model inputs to the target model, with noise prediction performed through the target model.

[0128] The loss function in the embodiments of this application will be introduced next:

[0129] In one possible implementation, the prediction error corresponding to the noise prediction result of the first image region is used as a reward term in the loss function, and the prediction error corresponding to the noise prediction result of the second image region is used as a penalty term in the loss function.

[0130] In one possible implementation, the prediction error corresponding to the noise prediction result of the first image region is obtained based on the difference between a first difference and a second difference; the first difference is the difference between the noise prediction result of the first image region and the true value; the second difference is the difference between the noise prediction result obtained by taking the first image region as the model input and the true value.

[0131] In one possible implementation, the prediction error corresponding to the noise prediction result of the second image region is obtained based on the difference between the third difference and the fourth difference; the third difference is the difference between the noise prediction result of the second image region and the true value; the fourth difference is the difference between the noise prediction result obtained by using the second image region as the model input of the reference model and the true value.

[0132] The loss function can take the following form:

[0133]

[0134] The first difference can be The second difference can be The third difference can be The fourth difference can be

[0135] xLQ is a low-quality image, for example, the second image.

[0136] t~U(0,T): Uniform sampling at time step t in the diffusion model.

[0137] sm: Multi-scale features, such as objects that can indicate a region of an image.

[0138] Ldiffref: Noise prediction error loss based on reference image.

[0139] The optimization objective in this embodiment is expressed as a loss function based on noise prediction error, which is within the framework of the diffusion model, thereby ensuring that the optimization at each time step t during the diffusion process is calculated step by step.

[0140] Reference Figure 5 , Figure 5 This document illustrates an application flow of an embodiment of this application, using a super-resolution task as an example. It describes how collecting low-quality (LQ) and high-quality (HQ) image pairs, along with human preference feedback data, provides a foundation for model training. In the preference alignment module, a direct instance preference alignment algorithm and a text description preference alignment mechanism are employed to ensure that the generated images semantically align with human preferences. The super-resolution generation module improves the perceptual quality and fidelity of the images using the DSPO method, while the evaluation and feedback module collects user feedback in real time to reduce illusions and artifacts during the generation process.

[0141] This application's embodiments overcome the limitations of traditional direct preference optimization under pixel-level reconstruction targets by using instance-level (i.e., the granularity of the semantic objects of the image) alignment, effectively suppressing artifacts and illusions in the generation process, thereby improving the enhancement quality of the image.

[0142] The selection of the first and second image regions can be made based on user preferences. Through a direct instance preference alignment algorithm and a human text description preference alignment mechanism, the generated content is ensured to be highly consistent with human preferences.

[0143] The embodiments of this application show significant improvements over the pre-trained model in all evaluation metrics. The LPIPS metric is improved by 3.34% in the single-step SR model and the DISTS metric is improved by 7.42% in the multi-step SR model, demonstrating the superior performance of DSPO in maintaining image realism and visual quality.

[0144] Compared to alignment methods such as SFT, DDPO, and Diffusion-DPO, DSPO performs better in real-world image super-resolution tasks. Improvements in other methods are limited by the conflict between image-level preferences and pixel-level reconstruction goals, while DSPO, through instance-level semantic guidance, accurately captures semantic information, reducing sensitivity to local anomalies and thus improving the overall quality of super-resolution images.

[0145] To analyze the impact of different DSPO optimization objectives on super-resolution, experiments were conducted using perceptual and fidelity metrics to simulate human evaluation. Ablation studies analyzed the effects of semantic instance alignment strategies and user description feedback strategies, where M1 and M2 represent the technical points of the direct instance preference alignment algorithm and the human text description preference alignment mechanism, respectively. Results show that semantic instance alignment significantly improves preference and reduces artifacts and illusions, while combining it with the user description feedback strategy further enhances super-resolution performance, achieving the best overall effect.

[0146] The table below compares the objective indicators of ablation experiments.

[0147]

[0148] Table 1.The ablation study on different optimization objectives. 'All' represents the use of both perception and fidelity metrics. The best and second best results of each metric are highlighted in red and blue, respectively.

[0149]

[0150] Table 2.The ablation study on different strategy.'M1'represents thesemantic instance alignment strategy and'M2'represents the user descriptionfeedback strategy.The red-highlighted values ​​indicate improved performancecompared to the pre-trained method.

[0151] Reference Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application, such as... Figure 7 As shown in the embodiment of this application, a data processing apparatus 700 includes:

[0152] The acquisition module 701 is used to acquire multiple first images; wherein the multiple first images are images obtained by performing multiple image enhancements on a second image; the semantic content of each first image includes at least one object, including a first object;

[0153] For a detailed description of the acquisition module 701, please refer to the description of step 401 in the above embodiments, which will not be repeated here.

[0154] The semantic segmentation module 702 is used to obtain a first image region and a second image region of the first object in the plurality of first images, wherein the first image region and the second image region belong to different first images, and the image enhancement effect of the first image region is higher than the image enhancement effect of the second image region.

[0155] For a detailed description of the semantic segmentation module 702, please refer to the description of step 402 in the above embodiments, which will not be repeated here.

[0156] Model training model 703 is used to take the second image as a conditional input to the target model, perform noise prediction on the first image region and the second image region through the target model, obtain noise prediction results, and update the target model based on the loss function constructed according to the noise prediction results.

[0157] For a detailed description of model training model 703, please refer to the description of step 403 in the above embodiments, which will not be repeated here.

[0158] In one possible implementation, the first image region is the image region in which the first object has the best image enhancement effect among the plurality of first images, and the second image region is the image region in which the first object has the worst image enhancement effect among the plurality of first images.

[0159] In one possible implementation, the image enhancement is specifically super-resolution.

[0160] In one possible implementation, the prediction error corresponding to the noise prediction result of the first image region is used as a reward term in the loss function, and the prediction error corresponding to the noise prediction result of the second image region is used as a penalty term in the loss function.

[0161] In one possible implementation, the acquisition module is further configured to:

[0162] Obtain user feedback on the image enhancement effect of each object in the image region of the plurality of first images;

[0163] The model is used to train the model for:

[0164] The second image and the prompt obtained based on the feedback information are used as the conditional inputs to the target model, and the first image region and the second image region are used as the model inputs to the target model. Noise prediction is then performed using the target model.

[0165] In one possible implementation, the different first images are obtained by image enhancement of the second image using the same model under different parameter configurations.

[0166] In one possible implementation, the first image region and the second image region are obtained by image segmentation of the plurality of first images.

[0167] The following describes a terminal device provided in an embodiment of this application. Please refer to [link to relevant documentation]. Figure 8 , Figure 8 This is a schematic diagram of a terminal device provided in an embodiment of this application. The terminal device 800 can specifically be a virtual reality (VR) device, a mobile phone, a tablet, a laptop computer, a smart wearable device, etc., and is not limited thereto. Specifically, the terminal device 800 includes: a receiver 801, a transmitter 802, a processor 803, and a memory 804 (wherein the terminal device 800 may have one or more processors 803). Figure 8 (Taking a processor as an example), the processor 803 may include an application processor 8031 ​​and a communication processor 8032. In some embodiments of this application, the receiver 801, transmitter 802, processor 803, and memory 804 may be connected via a bus or other means.

[0168] Memory 804 may include read-only memory and random access memory, and provides instructions and data to processor 803. A portion of memory 804 may also include non-volatile random access memory (NVRAM). Memory 804 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0169] The processor 803 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus but also power buses, control buses, and status signal buses. However, for clarity, all buses in the diagram are referred to as the bus system.

[0170] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 803. Processor 803 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above methods can be completed by integrated logic circuits in the hardware of processor 803 or by instructions in software form. Processor 803 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Processor 803 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 804. The processor 803 reads the information from memory 804 and, in conjunction with its hardware, completes the steps involved in the model training or model inference process described above.

[0171] Receiver 801 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 802 can be used to output digital or character information through the first interface; transmitter 802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 802 may also include a display device such as a display screen.

[0172] This application also provides a server; please refer to [link / reference]. Figure 9 , Figure 9This is a schematic diagram of a server structure provided in an embodiment of this application. The server 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 99 (e.g., one or more processors) and memory 932, and one or more storage media 930 (e.g., one or more mass storage devices) for storing application programs 942 or data 944. The memory 932 and storage media 930 can be temporary or persistent storage. The program stored in the storage media 930 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the server. Furthermore, the CPU 99 may be configured to communicate with the storage media 930 and execute the series of instruction operations in the storage media 930 on the server 900.

[0173] Server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input / output interfaces 958; or one or more operating systems 941, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0174] In this embodiment, the central processing unit 99 is used to perform actions related to model training or model inference in the above embodiments.

[0175] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0176] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0177] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the data processing method described in the above embodiments, or to cause the chip within the training device to execute the data processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0178] For details, please refer to Figure 10 , Figure 10 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1000. The NPU 1000 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1003, which is controlled by the controller 1004 to extract matrix data from the memory and perform multiplication operations.

[0179] In some implementations, the arithmetic circuit 1003 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1003 is a two-dimensional pulsating array. The arithmetic circuit 1003 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1003 is a general-purpose matrix processor.

[0180] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1002 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1001 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1008.

[0181] Unified memory 1006 is used to store input and output data. Weight data is directly transferred to weight memory 1002 via Direct Memory Access Controller (DMAC) 1005. Input data is also transferred to unified memory 1006 via DMAC.

[0182] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1009.

[0183] The Bus Interface Unit (BIU) 1010 is used by the instruction fetch memory 1009 to fetch instructions from external memory, and also by the memory access controller 1005 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0184] The DMAC is mainly used to move input data from external memory DDR to unified memory 1006, or to weight data to weight memory 1002, or to input data to input memory 1001.

[0185] The vector computation unit 1007 includes multiple processing units that further process the output of the computation circuit 1003 when needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0186] In some implementations, the vector computation unit 1007 can store the processed output vector in the unified memory 1006. For example, the vector computation unit 1007 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1003, such as performing linear interpolation on the feature planes extracted by the convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1007 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1003, for example, for use in subsequent layers of the neural network.

[0187] The instruction fetch buffer 1009 connected to the controller 1004 is used to store the instructions used by the controller 1004;

[0188] Unified memory 1006, input memory 1001, weight memory 1002, and instruction fetch memory 1009 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0189] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0190] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0191] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0192] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0193] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A data processing method, characterized in that, The method includes: A plurality of first images are obtained; wherein the plurality of first images are images obtained by performing multiple image enhancements on a second image; the semantic content of each first image includes at least one object, including a first object; Obtain the first image region and the second image region of the first object in the plurality of first images, wherein the first image region and the second image region belong to different first images, and the image enhancement effect of the first image region is higher than the image enhancement effect of the second image region; The second image is used as the conditional input of the target model, and the first image region and the second image region are used as the model input of the target model to obtain the noise prediction results of the first image region and the second image region. The target model is updated based on the loss function constructed according to the noise prediction results. The prediction error corresponding to the noise prediction result of the first image region is used as the reward term in the loss function, and the prediction error corresponding to the noise prediction result of the second image region is used as the penalty term in the loss function.

2. The method according to claim 1, characterized in that, The first image region and the second image region are selected according to user preferences.

3. The method according to claim 1 or 2, characterized in that, The first image region is the image region in which the first object has the best image enhancement effect among the plurality of first images, and the second image region is the image region in which the first object has the worst image enhancement effect among the plurality of first images.

4. The method according to any one of claims 1 to 3, characterized in that, The image enhancement specifically refers to super-resolution.

5. The method according to any one of claims 1 to 4, characterized in that, The prediction error corresponding to the noise prediction result of the first image region is obtained based on the difference between the first difference and the second difference; the first difference is the difference between the noise prediction result and the true value of the first image region. The second difference is the difference between the noise prediction result obtained by using the first image region as the model input and the true value; or, The prediction error corresponding to the noise prediction result of the second image region is obtained based on the difference between the third difference and the fourth difference; the third difference is the difference between the noise prediction result of the second image region and the true value. The fourth difference is the difference between the noise prediction result obtained by using the second image region as the model input of the reference model and the true value.

6. The method according to any one of claims 1 to 5, characterized in that, The method further includes: Obtain user feedback on the image enhancement effect of each object in the image region of the plurality of first images; The step of using the second image as a conditional input to the target model, and using the first image region and the second image region as model inputs to the target model, and performing noise prediction through the target model, includes: The second image and the prompt obtained based on the feedback information are used as the conditional inputs to the target model, and the first image region and the second image region are used as the model inputs to the target model. Noise prediction is then performed using the target model.

7. The method according to any one of claims 1 to 6, characterized in that, The target model is a diffusion model.

8. The method according to any one of claims 1 to 7, characterized in that, The first image region and the second image region are obtained by image segmentation of the plurality of first images.

9. A data processing apparatus, characterized in that, The device includes: An acquisition module is used to acquire multiple first images; wherein the multiple first images are images obtained by performing multiple image enhancements on a second image; the semantic content of each first image includes at least one object, including a first object; A semantic segmentation module is used to obtain a first image region and a second image region of the first object in the plurality of first images, wherein the first image region and the second image region belong to different first images, and the image enhancement effect of the first image region is higher than the image enhancement effect of the second image region. The model training method uses the second image as the conditional input of the target model, and the first image region and the second image region as the model input of the target model to obtain the noise prediction results of the first image region and the second image region. The target model is then updated based on the loss function constructed from the noise prediction results. The prediction error corresponding to the noise prediction result of the first image region is used as the reward term in the loss function, and the prediction error corresponding to the noise prediction result of the second image region is used as the penalty term in the loss function.

10. The apparatus according to claim 9, characterized in that, The first image region is the image region in which the first object has the best image enhancement effect among the plurality of first images, and the second image region is the image region in which the first object has the worst image enhancement effect among the plurality of first images.

11. The apparatus according to any one of claims 9 to 10, characterized in that, The prediction error corresponding to the noise prediction result of the first image region is obtained based on the difference between the first difference and the second difference; the first difference is the difference between the noise prediction result and the true value of the first image region. The second difference is the difference between the noise prediction result obtained by using the first image region as the model input and the true value; or, The prediction error corresponding to the noise prediction result of the second image region is obtained based on the difference between the third difference and the fourth difference; the third difference is the difference between the noise prediction result of the second image region and the true value. The fourth difference is the difference between the noise prediction result obtained by using the second image region as the model input of the reference model and the true value.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The acquisition module is also used for: Obtain user feedback on the image enhancement effect of each object in the image region of the plurality of first images; The model is used to train the model for: The second image and the prompt obtained based on the feedback information are used as the conditional inputs to the target model, and the first image region and the second image region are used as the model inputs to the target model. Noise prediction is then performed using the target model.

13. The apparatus according to any one of claims 9 to 12, characterized in that, The different first images are obtained by enhancing the second image using the same model with different parameter configurations.

14. The apparatus according to any one of claims 9 to 13, characterized in that, The first image region and the second image region are obtained by image segmentation of the plurality of first images.

15. A computer storage medium, characterized in that, The computer storage medium stores one or more instructions, which, when executed by one or more computers, cause the one or more computers to perform the operation of the method according to any one of claims 1 to 8.

16. A computer program product, characterized in that, Includes computer-readable instructions that, when executed on a computer device, cause the computer device to perform the method as described in any one of claims 1 to 8.

17. A computing device, comprising at least one processor and at least one memory; the processor and the memory are connected via a communication bus and communicate with each other. The at least one memory is used to store code; The at least one processor is used to execute the code to perform the method as described in any one of claims 1 to 8.

18. A chip, comprising a processor, characterized in that, The processor is used by the data processing apparatus to implement the method as described in any one of claims 1 to 8.

Images