Image processing method, processing method and device of image processing model

Abstract

The application provides an image processing method and a processing method and device of an image processing model, relates to the field of artificial intelligence, and particularly relates to the field of computer vision. The image processing method comprises the following steps: generating a scalar map based on the difference between the distribution of reconstruction errors and the distribution of noise between a first image and a second image, the scalar map comprising one or more regions, and the scalar value corresponding to each region in the one or more regions being used for indicating the quality of one or more image blocks in the second image. The application can intuitively reflect the quality of the image, and is beneficial to guaranteeing the effect of image processing.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of computer vision, and in particular to an image processing method and a processing method and device of an image processing model. BACKGROUND

[0002] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military fields. It is a science about how to use cameras and computers to obtain the data and information of the photographed object. In other words, it is to install eyes (cameras) and brains (algorithms) on computers to replace human eyes to identify, track, and measure targets, so that computers can perceive the environment. Since perception can be regarded as extracting information from sensory signals, computer vision can also be regarded as a science of how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision is to replace the visual organs to obtain input information, and then use computers to replace the brain to process and interpret the input information. The ultimate goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0003] Evaluation of image quality is an indispensable link in the image processing process. Image compression is one of the most basic techniques in the field of image processing. The goal of image compression is to reduce the image rate while preserving the key visual information of the image signal, so as to realize efficient transmission and storage of images. With the development of deep learning, deep learning-based image compression algorithms have been widely applied. The evaluation of image compression quality is an indispensable link in the image compression process. Current evaluation indicators of image compression quality, such as peak signal to noise ratio (PSNR), mean squared error (MSE), and structural similarity index measure (SSIM), mainly measure the closeness of two high-dimensional data from a statistical point of view. Such evaluation indicators have low correlation with human eye perception, in other words, the judgment result of such evaluation indicators on image quality may be different from the judgment result of human eyes on image quality, which affects the judgment on the performance of image compression algorithms, and further affects the effect of image processing. SUMMARY

[0004] The present application provides an image processing method, a processing method and device of an image processing model, which can intuitively reflect the quality of the image, and is beneficial to guarantee the effect of image processing.

[0005] In a first aspect, an image processing method is provided. The method comprises: obtaining a reconstruction error between a first image and a second image corresponding to the first image; generating a scalar map based on a difference between a distribution of the reconstruction error and a distribution of noise, the scalar map comprising one or more regions, each region of the one or more regions corresponding to a scalar value indicating a quality of one or more image blocks in the second image.

[0006] In the scheme of the embodiments, the quality of the reconstructed image is evaluated from the perspective of information theory, i.e., the performance of the image processing model is evaluated based on the difference between the distribution of the reconstruction error and the distribution of the noise. The image processing model used to obtain the second image is trained based on the reconstruction error. The greater the difference between the distribution of the reconstruction error and the distribution of the noise, the more characteristic signals are contained in the reconstruction error. The image processing model mainly adjusts the parameters of the model based on the characteristic signals in the training process, which is beneficial to obtain better processing results, i.e., is beneficial to improve the quality of the reconstructed image. In other words, the greater the difference between the distribution of the reconstruction error and the distribution of the noise, the greater the possibility of the quality of the reconstructed image being high. The result is basically consistent with the judgment of the human eye on the quality of the image, which is beneficial to intuitively reflect the quality of the image, the evaluation result is reliable, and is beneficial to ensure the effect of image processing.

[0007] The second image can be a reconstructed image obtained by processing an initial image. The initial image has a corresponding relationship with the first image. The first image can be understood as a target image corresponding to the initial image. In some implementations, the initial image can be the first image.

[0008] In combination with the first aspect, in some implementations of the first aspect, the second image is obtained by compressing the first image.

[0009] The quality of the second image can reflect the compression quality. The compression process can be implemented by an image compression model. In this case, the quality of the second image can reflect the performance of the image compression model.

[0010] The scheme of the embodiments of the present application measures whether the compressed signal is a meaningful signal, i.e., a signal containing features, by comparing the difference between the distribution of the reconstruction error and the distribution of the noise. The greater the difference between the distribution of the reconstruction error and the distribution of the noise, the more the signal containing features in the compressed signal, which indicates that the image compression model mainly adjusts the parameters of the model based on the signal containing features in the training process, and is conducive to obtaining better compression performance, i.e., improving the quality of the compressed image and reducing the code rate of the image. In other words, the greater the difference between the distribution of the reconstruction error and the distribution of the noise, the greater the possibility of high quality of the compressed image. It is basically consistent with the judgment result of the human eye on the image quality, is conducive to intuitively reflecting the quality of the image, and the evaluation result is reliable.

[0011] In addition, the scheme of the embodiments of the present application can be used to measure the potential for further improving the image compression ratio. If the difference between the distribution of the reconstruction error and the distribution of the noise is large, it indicates that there are fewer noise signals in the reconstruction error, and the potential for further improving the image compression ratio is small. If the difference between the distribution of the reconstruction error and the distribution of the noise is small, it indicates that there are more noise signals in the reconstruction error, and the potential for further improving the image compression ratio is large.

[0012] In combination with the first aspect, in some implementations of the first aspect, the positions of the one or more regions in the scalar map are the same as the positions of the one or more image blocks in the second image.

[0013] In combination with the first aspect, in some implementations of the first aspect, the method further includes obtaining a target scalar value from the scalar map, the target scalar value being used to indicate the quality of the second image.

[0014] In combination with the first aspect, in some implementations of the first aspect, the scalar value corresponding to a first region of the one or more regions is determined based on the difference between the distribution of the reconstruction error and the distribution of the noise between a first image block and a second image block, the first image block belonging to the first image, the second image block belonging to the second image, the difference between the distribution of the reconstruction error and the distribution of the noise between the first image block and the second image block being determined according to a difference value between the energy of at least one first error block in the reconstruction error between the first image block and the second image block and the energy of at least one noise block in the noise; the energy of the at least one first error block is obtained by inputting the at least one first error block into a first energy model or a second energy model for processing, the energy of the at least one noise block is obtained by inputting the at least one noise block into the first energy model or the second energy model for processing, the energy value output by the first energy model being used to indicate the probability that the data input into the first energy model conforms to the distribution of the reconstruction error between the first image block and the second image block, the energy value output by the second energy model being used to indicate the probability that the data input into the second energy model conforms to the distribution of the noise.

[0015] In the embodiments of the present application, the energy-based models (EBMs) are used to explicitly model the probability distribution of high-dimensional data. Specifically, in the embodiments of the present application, the energy-based models are used to explicitly model the probability distribution of reconstruction errors and the probability distribution of noise, and the difference between the distribution of reconstruction errors and the distribution of noise is obtained by the difference between the energy of reconstruction errors and the energy of noise, thereby reducing the amount of calculation.

[0016] With reference to the first aspect, in some implementations of the first aspect, the first energy-based model is trained by taking a plurality of second error blocks in reconstruction errors between the third image block and a fourth image block corresponding to the third image block as training data.

[0017] With reference to the first aspect, in some implementations of the first aspect, the first energy-based model is obtained by adjusting the parameters of the energy-based model in the training process to reduce the value of the loss function in the training process, wherein the value of the loss function in the i th iteration process in the training process is determined according to the difference between the energy sum of the plurality of second error blocks and the energy of the plurality of target generated data subject to the distribution represented by the energy-based model in the i th iteration process, the energy of the plurality of target generated data subject to the distribution represented by the energy-based model in the i th iteration process is calculated by the energy-based model in the i th iteration process, and i is a positive integer.

[0018] With reference to the first aspect, in some implementations of the first aspect, the plurality of target generated data subject to the distribution represented by the energy-based model in the i th iteration process is obtained by adjusting the initial generated data of the energy-based model in the i th iteration process a plurality of times, wherein the generated data obtained by adjusting the initial generated data a k+1 th time is determined according to the generated data obtained by adjusting the initial generated data a k th time, the energy gradient of the generated data obtained by adjusting the initial generated data a k th time, and Gaussian noise, the coefficient of the energy gradient of the generated data obtained by adjusting the initial generated data a k th time is greater than the coefficient of the Gaussian noise, and k is a positive integer.

[0019] In this way, the problem that the energy of the generated sample in the next iteration process cannot be reduced due to the too small gradient term can be avoided, a more stable sampling process can be realized, and the stability of the training process can be improved.

[0020] With reference to the first aspect, in some implementations of the first aspect, the initial generated data of the energy-based model in the i th iteration process is determined according to Gaussian noise and the plurality of target generated data subject to the distribution represented by the energy-based model in the iteration process before the i th iteration.

[0021] The target generated data obtained in the previous iteration process is introduced as the initial generated data, which can further reduce the energy of the obtained sample, and is beneficial to ensure that the distribution of the plurality of target generated data obtained is more consistent with the distribution represented by the energy model in the current iteration process. In addition, the initial generated data is determined according to the target generated data obtained in the previous iteration process and the Gaussian noise, which is beneficial to improve the diversity of the target generated data corresponding to the energy model in the current iteration process, and further improve the training effect of the energy model.

[0022] In combination with the first aspect, in some implementations of the first aspect, the value of the loss function in the i th iteration process is determined according to the difference between the energy of the plurality of second error blocks and the energy of the plurality of target generated data subject to the distribution represented by the energy model in the i th iteration process, the square of the energy of the plurality of second error blocks, and the square of the energy of the plurality of target generated data subject to the distribution represented by the energy model in the i th iteration process.

[0023] In this way, a regularization term is added to the energy of all data in the loss function, for example, the square sum of the energy of the plurality of second error blocks and the square of the energy of the plurality of target generated data is introduced in the loss function, which limits the absolute value of the energy of the positive sample data (i.e., the plurality of second error blocks) and the negative sample data (i.e., the plurality of target generated data), avoids the energy difference between the two being too large to cause the loss function to be unable to decrease, and is beneficial to ensure the training effect of the energy model.

[0024] In combination with the first aspect, in some implementations of the first aspect, the scalar value corresponding to the first region includes any one of the following: a value of a first evaluation index, a value of a second evaluation index, or a value of a third evaluation index; the value of the first evaluation index is determined according to the difference between the energy of at least one first error block and the energy of at least one noise block, the energy of the at least one first error block being obtained by inputting the at least one first error block into the first energy model for processing, and the energy of the at least one noise block being obtained by inputting the at least one noise block into the first energy model for processing; the value of the second evaluation index is determined according to the difference between the energy of at least one first error block and the energy of at least one noise block, the energy of the at least one first error block being obtained by inputting the at least one first error block into the second energy model for processing, and the energy of the at least one noise block being obtained by inputting the at least one noise block into the second energy model for processing; and the value of the third evaluation index is determined according to the value of the first evaluation index and the value of the second evaluation index.

[0025] In combination with the first aspect, in some implementations of the first aspect, the first evaluation index satisfies the following formula:

[0026]

[0027] Where Metric1 represents the first evaluation index, f() represents the energy of the input data calculated using the first energy model, and ε~p error Denotes at least one first error block, ε~p noise Indicates at least one noise block. This represents the average energy of at least one first error block calculated using the first energy model. This represents the average energy of at least one noise block calculated using the first energy model.

[0028] In conjunction with the first aspect, in some implementations of the first aspect, the second evaluation index satisfies the following formula:

[0029]

[0030] Where Metric2 represents the second evaluation index, g() represents the energy of the input data calculated using the second energy model, and ε~p error Denotes at least one first error block, ε~p noise Indicates at least one noise block. This represents the average energy of at least one first error block calculated using the second energy model. This represents the average energy of at least one noise block calculated using the second energy model.

[0031] Secondly, a processing method for an image processing model is provided, including: obtaining the reconstruction error between a first image and a second image corresponding to the first image, wherein the second image is obtained by processing an initial image through an image processing model; and updating the image processing model based on the difference between the distribution of the reconstruction error and the distribution of noise between the first image and the second image.

[0032] Optionally, updating the image processing model based on the difference between the distribution of reconstruction error and the distribution of noise between the first image and the second image includes: replacing the image processing model based on the difference between the distribution of reconstruction error and the distribution of noise between the first image and the second image.

[0033] According to the scheme of the embodiments of this application, the difference between the distribution of reconstruction error and the distribution of noise can distinguish between meaningless noise signals and meaningful feature signals in the reconstruction error, which is beneficial for updating the image processing model to improve the quality of the reconstructed image.

[0034] Furthermore, the solution of this application can reconstruct the difference between the distribution of error and the distribution of noise to measure the performance of the image processing model, and then select an appropriate method to process the image compression model, which is beneficial to improve the quality of the reconstructed image.

[0035] In conjunction with the second aspect, in some implementations of the second aspect, the image processing model is an image compression model, and the initial image is the first image.

[0036] In conjunction with the second aspect, in some implementations of the second aspect, the image processing model is updated based on the difference between the distribution of reconstruction error and the distribution of noise, including: adjusting the reconstruction error between the first image and the second image with the goal of increasing the difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block to obtain the adjusted reconstruction error, wherein the first image block belongs to the first image and the second image block belongs to the second image; and training the image processing model based on the adjusted reconstruction error.

[0037] According to the scheme of the embodiments of this application, by increasing the difference between the distribution of reconstruction error and the distribution of noise, meaningless noise signals and meaningful feature signals can be separated from the reconstruction error, or in other words, noise signals can be removed from the reconstruction error. Based on the adjusted reconstruction error, the training of the image processing model can be guided, which can ensure that the image processing model is trained with the goal of reducing feature signals as much as possible. This is beneficial to retain the feature signals in the reconstructed image, improve the image quality, and thus improve the performance of the model.

[0038] In conjunction with the second aspect, in some implementations of the second aspect, adjusting the reconstruction error between the first image and the second image with the goal of increasing the difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block to obtain the adjusted reconstruction error includes: adjusting the reconstruction error between the first image and the second image with the goal of reducing the energy of the reconstruction error between the first image block and the second image block to obtain the adjusted reconstruction error, wherein the energy between the first image block and the second image block is calculated by a first energy model, and the energy value output by the first energy model is used to indicate the probability that the data input to the first energy model follows the distribution of reconstruction error between the first image block and the second image block.

[0039] In this embodiment, an energy model is used to explicitly model the probability distribution of reconstruction error and the probability distribution of noise. The difference between the distribution of reconstruction error and the distribution of noise is obtained by the difference between the energy of reconstruction error and the energy of noise, which reduces the amount of computation.

[0040] In conjunction with the second aspect, in some implementations of the second aspect, the first error block in the adjusted reconstruction error satisfies the following formula:

[0041]

[0042] Where ε represents the first error block in the reconstruction error between the first image block and the second image block, ε′ represents the first error block in the adjusted reconstruction error, η represents the step size of energy reduction, and f() represents the energy of the input data calculated by the first energy model.

[0043] In conjunction with the second aspect, in some implementations of the second aspect, the first energy model is trained using multiple second error blocks from the reconstruction error between the third image block and the fourth image block corresponding to the third image block as training data.

[0044] In conjunction with the second aspect, in some implementations of the second aspect, the first energy model is obtained by adjusting the parameters of the energy model during the training process with the goal of reducing the value of the loss function during the training process. Here, the value of the loss function in the i-th iteration of the training process is determined based on the difference between the energy of multiple second error blocks and the energy of multiple target generated data that follow the distribution represented by the energy model in the i-th iteration process. The energy of multiple second error blocks and the energy of multiple target generated data that follow the distribution represented by the energy model in the i-th iteration process are calculated by the energy model in the i-th iteration process, where i is a positive integer.

[0045] In conjunction with the second aspect, in some implementations of the second aspect, the multiple target generation data that conform to the distribution represented by the energy model in the i-th iteration process are obtained by multiple iterations of the initial generation data of the energy model in the i-th iteration process. Among them, the generation data obtained in the (k+1)-th iteration of the initial generation data is determined based on the generation data obtained in the k-th iteration, the energy gradient of the generation data obtained in the k-th iteration, and Gaussian noise. The coefficient of the energy gradient of the generation data obtained in the k-th iteration is greater than the coefficient of the Gaussian noise, and k is a positive integer.

[0046] In conjunction with the second aspect, in some implementations of the second aspect, the initial generated data of the energy model in the i-th iteration process is determined based on multiple target generated data that conform to Gaussian noise and the distribution represented by the energy model in the iteration process prior to the i-th iteration.

[0047] In conjunction with the second aspect, in some implementations of the second aspect, the value of the loss function in the i-th iteration process is determined based on the difference between the energy of multiple second error blocks and the energy of multiple target generated data that follow the distribution represented by the energy model in the i-th iteration process, the square of the energy of multiple second error blocks, and the square of the energy of multiple target generated data that follow the distribution represented by the energy model in the i-th iteration process.

[0048] Thirdly, an image processing apparatus is provided, the apparatus including modules or units for performing the methods of the first aspect and any implementation thereof.

[0049] Fourthly, a processing apparatus for an image processing model is provided, the apparatus including modules or units for performing the methods of the second aspect and any implementation thereof.

[0050] It should be understood that the extensions, limitations, interpretations and descriptions of the relevant content in the first aspect above also apply to the same content in the second, third and fourth aspects.

[0051] Fifthly, an image processing apparatus is provided, the apparatus comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the method of the first aspect and any implementation thereof.

[0052] The processor mentioned in the fifth aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network processing processor. The neural network processing processor can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU), etc. The TPU is a dedicated integrated circuit developed by Google for a fully customized artificial intelligence accelerator for machine learning.

[0053] A sixth aspect provides a processing apparatus for an image processing model, the apparatus comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to execute the methods of the second aspect and any implementation thereof.

[0054] The processor mentioned in the sixth aspect above can be a CPU or a combination of a CPU and a neural network processing processor. The neural network processing processor can include GPUs, NPUs, TPUs, etc.

[0055] A seventh aspect provides a computer-readable medium storing program code for execution by a device, the program code including methods for performing any implementation of the first or second aspect.

[0056] Eighthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method in any implementation of the first or second aspect described above.

[0057] In a ninth aspect, a chip is provided, the chip including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the method in any implementation of the first aspect or the second aspect.

[0058] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is configured to execute the instructions stored in the memory. When the instructions are executed, the processor is configured to perform the method in either the first or second implementation. Attached Figure Description

[0059] Figure 1 This is a schematic diagram of the basic framework for image compression provided in an embodiment of this application;

[0060] Figure 2 This is a schematic diagram of a system architecture provided in an embodiment of this application;

[0061] Figure 3 This is a schematic diagram of an image processing apparatus provided in an embodiment of this application;

[0062] Figure 4 This is a schematic flowchart of an image processing method provided in an embodiment of this application;

[0063] Figure 5 This is a schematic diagram illustrating the energy situation of different error blocks provided in the embodiments of this application;

[0064] Figure 6 This is a schematic diagram illustrating the reconstruction errors corresponding to different image compression algorithms provided in the embodiments of this application;

[0065] Figure 7 This is a schematic flowchart illustrating a processing method of an image processing model provided in an embodiment of this application;

[0066] Figure 8 This is a comparison diagram of a first error block before and after adjustment, provided in an embodiment of this application;

[0067] Figure 9 This is a comparison diagram of another first error block before and after adjustment provided in the embodiments of this application;

[0068] Figure 10 This is a comparison diagram of another first error block before and after adjustment provided in the embodiments of this application;

[0069] Figure 11 This is a schematic block diagram of an image processing apparatus provided in an embodiment of this application;

[0070] Figure 12 This is a schematic block diagram of a processing apparatus for an image processing model provided in an embodiment of this application;

[0071] Figure 13 This is a schematic block diagram of another image processing apparatus provided in the embodiments of this application;

[0072] Figure 14 This is a schematic block diagram of a processing apparatus for another image processing model provided in the embodiments of this application. Detailed Implementation

[0073] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0074] To achieve efficient transmission or storage of images / videos, image / video compression is necessary. The solution provided in this application can be applied to scenarios requiring image compression, such as video calls or surveillance.

[0075] The following is a brief introduction to video call and monitoring scenarios.

[0076] Video call:

[0077] When users are making video calls, compressing images can reduce the bandwidth used for data transmission, increase data transmission speed, and improve the user experience.

[0078] The image processing method of this application embodiment can obtain an image compression model suitable for performing image compression tasks, thereby improving compression performance.

[0079] monitor:

[0080] Monitoring scenarios include: smart cities, field monitoring, indoor monitoring, outdoor monitoring, and vehicle monitoring. Among these, smart city scenarios involve collecting a large amount of data, placing significant pressure on hardware storage capacity and the bandwidth required for data transmission.

[0081] By employing the image processing method of the embodiments of this application, an image compression model suitable for performing image compression tasks can be obtained, thereby improving compression performance.

[0082] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0083] (1) Image compression

[0084] With the development of computer vision, deep learning-based image compression algorithms have been widely used.

[0085] Figure 1 This illustrates a basic framework for image compression. It utilizes an encoder g... a Mapping the uncompressed image x from the data space to the latent space, i.e. Figure 1 In the code space, the hidden representation y is obtained, and y is quantized using a quantizer q to obtain the quantization result. Quantification results Encoding is performed, and the resulting code length R is the encoded length corresponding to image x. The decoder g is then used... s Quantification results After processing, a compressed image can be obtained.

[0086] The training objective of image compression algorithms is to minimize the code length R while simultaneously minimizing the reconstruction error D. The reconstruction error is used to indicate the difference between the uncompressed image x and the compressed image x. The difference between them. Specifically, a distance metric can be used to measure the difference between the uncompressed image x and the compressed image. The differences between them. For example, the reconstruction error can be calculated using perceptual distance. Specifically, using the encoder g p Image x before compression and image x after compression Mapping to the perception space yields the processing result z and the processing result z. Based on the processing result z and the processing result Calculate the reconstruction error.

[0087] (2) Neural Network

[0088] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit takes an intercept of 1 as input, and its output can be:

[0089]

[0090] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit.

[0091] f represents the activation function of a neural network, used to introduce nonlinear characteristics and transform the input signal into the output signal. The output signal of this activation function can be used as the input to the next layer. For example, the activation function can be ReLU, tanh, or sigmoid.

[0092] A neural network is a network formed by connecting multiple individual neural units, meaning that the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0093] (3) Deep Neural Networks

[0094] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0095] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through this simple operation. Due to the large number of layers in a DNN, the coefficients W and the offset vector... The number of these parameters is also quite large. The definitions of these parameters in DNNs are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W is located, while the subscript corresponds to the third layer index 2 of the output and the second layer index 4 of the input.

[0096] In summary, the coefficient from the k-th neuron in layer L-1 to the j-th neuron in layer L is defined as...

[0097] It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0098] (4) Loss Function

[0099] In training deep neural networks, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value and update the weight vector of each layer based on the difference. (Of course, there's usually a pre-configuration process before the first update, where parameters are pre-configured for each layer.) For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network can predict the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss. Generally, a smaller loss indicates higher training quality, while a larger loss indicates lower training quality. Similarly, smaller loss fluctuations result in more stable training, while larger loss fluctuations lead to less stable training.

[0100] like Figure 2 As shown, this application embodiment provides a system architecture 100. In Figure 2 In this embodiment, the data acquisition device 170 is used to acquire training data. For example, for the training method of the first energy model in this application embodiment, the training data may include multiple second error blocks.

[0101] After collecting the training data, the data acquisition device 170 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130. This target model / rule 101 is the model used for the vision task.

[0102] The target model / rule 101 in this embodiment can specifically be a neural network model, such as a convolutional neural network or a residual network. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 170; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0103] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 2 The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud service. Figure 2 In the process, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data in this embodiment may include at least one first error block or at least one noise block.

[0104] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0105] Finally, I / O interface 112 returns the processing result, such as the data processing result obtained above, to client device 140, thereby providing it to the user.

[0106] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0107] exist Figure 2In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in various forms such as display, sound, or animation. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0108] It is worth noting that, Figure 2 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 2 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0109] like Figure 3 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 may be the neural network model in this application embodiment.

[0110] This application provides an image processing method and an image compression algorithm, which can improve the image processing effect.

[0111] To better illustrate the image processing method of the embodiments of this application, the following will first refer to the appendix. Figure 3 The image processing apparatus according to embodiments of this application will be described.

[0112] exist Figure 3 In the image processing device 300, there are an algorithm module 310, a preprocessing module 320, an evaluation module 330, and an algorithm processing module 340.

[0113] Algorithm module 310 is used to process the initial image corresponding to the first image to obtain the second image.

[0114] Specifically, the second image can be obtained by processing the initial image using an image processing model.

[0115] The first image is the target image corresponding to the initial image. In other words, the first image can be understood as the target that the image processing model expects to obtain after processing the initial image.

[0116] The first image and the initial image can be the same or different.

[0117] For example, the image processing model can be an image super-resolution model, an image denoising model, or an image compression model, etc.

[0118] The following text mainly uses an image compression model as an example for explanation, and does not constitute a limitation on the solutions of the embodiments of this application.

[0119] Image compression models can also be called image compression algorithms.

[0120] The image compression model can be a pre-trained image compression model, or it can be an image compression model in the training process.

[0121] It should be noted that the algorithm module 310 is an optional module. When the device 300 includes the algorithm module 310, the input to the device 300 can be an image compression model, or in other words, an image compression algorithm. When the device 300 does not include the algorithm module 310, the input to the device 300 can include a first image and a second image. Alternatively, when the device 300 does not include the algorithm module 310, the input to the device 300 can include the reconstruction error between the first image and the second image.

[0122] The preprocessing module 320 is used to obtain an error block of the target size from the reconstruction error between the first image block and the second image block. For example, as... Figure 3 As shown, multiple error blocks, such as error block 1 and error block 2, are obtained.

[0123] The target size is less than or equal to the size of the reconstruction error. In other words, the error block is part or all of the reconstruction error.

[0124] For example, the preprocessing module 320 can trim the reconstruction error to obtain an error block of the target size.

[0125] Furthermore, the preprocessing module 320 can also be used to calculate the reconstruction error between the first image and the second image.

[0126] Optionally, the preprocessing module 320 can also be used to obtain a noise block of a target size from the Gaussian noise. For example, such as Figure 3 As shown, multiple noise blocks, such as noise block 1 and noise block 2, are obtained.

[0127] In other words, the size of the noise block and the size of the error block are the same.

[0128] It should be noted that the preprocessing module 320 is optional. For example, the error block may contain all the reconstruction errors. In this case, the reconstruction errors can be directly input into subsequent modules for processing.

[0129] Evaluation module 330 generates a scalar map based on the differences between the distribution of reconstruction error and the distribution of noise between the first and second images. The scalar map includes one or more regions. A scalar value corresponding to each of these regions is used to indicate the quality of one or more image patches in the second image.

[0130] Specifically, the scalar value corresponding to the first region in the scalar map is determined based on the difference between the distribution of reconstruction error and the distribution of noise between the first image patch and the second image patch. The first image patch belongs to the first image, and the second image patch belongs to the second image patch.

[0131] The difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block is determined based on the difference between the energy of at least one first error block in the reconstruction error between the first image block and the energy of at least one noise block in the noise.

[0132] For example, the evaluation module 330 can calculate using the first energy model. Figure 3 The scalar values ​​corresponding to each region in the scalar map are obtained by analyzing the energy of multiple error blocks and multiple noise blocks in the image. These multiple error blocks are the image blocks in the reconstruction error between the first and second images.

[0133] The energy value output by the first energy model is used to indicate the probability that the data input to the first energy model follows the distribution of reconstruction error.

[0134] For example, the evaluation module 330 can calculate using the second energy model. Figure 3 The scalar values ​​corresponding to each region in the scalar map are obtained by analyzing the energy of multiple error blocks and multiple noise blocks in the image. These multiple error blocks are the image blocks in the reconstruction error between the first and second images.

[0135] The energy value output by the second energy model is used to indicate the probability that the data input to the second energy model follows a noise distribution.

[0136] The specific processing procedure of the evaluation module 330 can be found in step S403 of method 400.

[0137] The algorithm processing module 340 can be used to update the image processing model based on the difference between the distribution of reconstruction error and the distribution of noise between the first image and the second image.

[0138] For example, such as Figure 3 As shown, the algorithm processing module 340 trains the image processing model based on the difference between the distribution of reconstruction error and the distribution of noise between the first image and the second image.

[0139] Specifically, the algorithm processing module 340 can adjust the reconstruction error between the first image and the second image with the goal of increasing the difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block, so as to obtain the adjusted reconstruction error, and train the image processing model based on the adjusted reconstruction error.

[0140] For example, the algorithm processing module 340 can calculate using the first energy model. Figure 4 The energy of multiple error blocks in the model is used to adjust these error blocks with the goal of reducing their energy, thus obtaining the adjusted reconstruction error.

[0141] In other words, the adjusted reconstruction error can be used to guide the training of the image processing model. The specific training method can be the original training method for the image processing model, and this application does not limit this approach.

[0142] The specific processing procedure of the algorithm processing module 340 can be found in step S703 of method 700.

[0143] It should be noted that the device 300 may include either the evaluation module 330 or the algorithm processing module 340, or the device 300 may include both the evaluation module 330 and the algorithm processing module 340.

[0144] The following is combined Figure 4 The image processing methods in the embodiments of this application are described in detail.

[0145] Figure 4 An image processing method 400 provided in an embodiment of this application is shown. Figure 1 The method shown can be executed by a computing device, which can be a cloud service device or a terminal device, such as a computer, server, mobile phone, camera, vehicle, drone or robot, or a system composed of cloud service devices and terminal devices.

[0146] Method 400 includes steps S401 to S403. Steps S401 to S403 will be described in detail below.

[0147] S401, Obtain the first image and the second image corresponding to the first image.

[0148] Specifically, the second image can be obtained by processing the initial image using an image processing model.

[0149] The first image is the target image corresponding to the initial image. In other words, the first image can be understood as the target that the image processing model expects to obtain after processing the initial image.

[0150] It should be noted that there can be one first image, and correspondingly, there can be one second image. There can also be multiple first images, and correspondingly, multiple second images. This application does not limit the number of first and second images, as long as there is a one-to-one correspondence between the first and second images.

[0151] The first image and the initial image can be the same or different.

[0152] For example, the image processing model can be an image super-resolution model. The initial image is super-resolution processed using the image super-resolution model to obtain a second image. The first image is the target image corresponding to the initial image. The resolution of the first image is greater than the resolution of the initial image.

[0153] For example, the image processing model can be an image denoising model. The initial image is denoised using the image denoising model to obtain a second image. The first image is the target image corresponding to the initial image. The noise signal in the first image is less than that in the initial image.

[0154] For example, the image processing model can be an image compression model. The first image is compressed using this model to obtain the second image. The first image is the initial image. In other words, the second image is the image obtained by compressing the first image. The second image can also be called the reconstructed image.

[0155] For ease of description, the following text will only use an image compression model as an example to illustrate method 400, that is, the second image is obtained by compressing the first image, and does not constitute a limitation on the image processing method of the embodiments of this application.

[0156] For example, based on training data Figure 1 The image compression model shown is trained; specifically, the encoder and decoder in the image compression model are trained to obtain a trained image compression model. The image compression model uses the encoder and decoder to encode and decode the first image to obtain the second image. For example, the first image could be... Figure 1 In x, the second image can be Figure 2 In

[0157] The image processing model in this application embodiment can also be called an image processing algorithm, and correspondingly, the image compression model can also be called an image compression algorithm.

[0158] For example, the image compression model can be a Gaussian mixed model (GMM), a context-Gaussian mixture model hyperpriorentropy (CTX_GMM), a context-Gaussian mixture model with residual non-local attention blocks (RNAB_CTX_GMM), or an end-to-end (E2E) image compression model, etc. GMM can also be called a Gaussian mixture model hyperpriorentropy estimation model. An E2E model can also be called a perceptual encoding / decoding model. RNAB_CTX_GMM can also be called an encoding / decoding model with residual non-local attention blocks.

[0159] For example, the image compression model can be a trained model.

[0160] Alternatively, the image compression model can also be the model during the training process.

[0161] For example, the training process of an image compression model is... Figure 3 The training is performed by the training device in the image compression model. Alternatively, the training process of the image compression model can also be performed by other devices. This application does not limit this aspect.

[0162] For example, the second image can be made by Figure 2 The algorithm module 310 in the image is obtained by processing the first image through an image compression model.

[0163] Alternatively, the second image can be made by Figure 3 The execution device 110 in the image processing module obtains the first image by processing it using an image compression model, i.e., target model / rule 101. Alternatively, the second image can also be obtained by processing it using other devices. This application embodiment does not limit the method of obtaining the second image.

[0164] In one implementation, step S401 may include: acquiring an image compression model and a first image, and processing the first image using the image compression model to obtain a second image.

[0165] For example, the image compression model can be input by the user, received from other devices, or trained on a local device. This application embodiment does not limit the method of obtaining the image compression model.

[0166] It should be noted that step S401 is an optional step.

[0167] S402, obtain the reconstruction error between the first image and the second image.

[0168] Reconstruction error is used to indicate the difference between the first image and the second image.

[0169] As mentioned earlier, the second image can be obtained by compressing the first image, and the reconstruction error can also be called the compression error. The number of reconstruction errors is the same as the number of second images.

[0170] For example, a distance metric can be used to measure the difference between the first image and the second image. For instance, perceptual distance can be used to calculate the reconstruction error. Specifically, the encoder maps the first image and the second image to a perceptual space, and the reconstruction error is calculated based on the mapping result.

[0171] It should be understood that the reconstruction error between the first image and the second image can also be calculated in other ways, and the embodiments of this application do not limit this.

[0172] For example, the reconstruction error between the first image and the second image can be calculated using the same method as the reconstruction error calculation during the training process of the image compression model.

[0173] This allows for adjustment of the reconstruction error, and the training of the image compression model can be guided based on the adjusted reconstruction error.

[0174] If method 400 includes step S401, step S402 can be understood as calculating the reconstruction error between the first image and the second image.

[0175] For example, the reconstruction error can be caused by Figure 3 The result is calculated by the preprocessing module 320. Alternatively, it can be obtained from... Figure 3 The algorithm module 310 in the algorithm is used to calculate the result.

[0176] Even without step S401 in method 400, the reconstruction error can be obtained in various ways. For example, step S402 may include obtaining the reconstruction error between the first image and the second image input by the user. Alternatively, step S402 may include receiving the reconstruction error between the first image and the second image sent by another device. This application embodiment does not limit the specific method for obtaining the reconstruction error.

[0177] S403, a scalar map is generated based on the difference between the distribution of reconstruction error and the distribution of noise between the first and second images. The scalar map includes one or more regions. The scalar value corresponding to each of the one or more regions is used to indicate the quality of one or more image patches in the second image.

[0178] For example, step S403 can be performed by the evaluation module 330 in the device 300.

[0179] Each region in the scalar image corresponds one-to-one with each image patch in the second image. The scalar values ​​corresponding to different regions in the scalar image are used to indicate the quality of different image patches in the second image.

[0180] One or more image patches in the first image also correspond one-to-one with one or more image patches in the second image. The two corresponding image patches are located in the same position in the first and second images.

[0181] In other words, there are correspondences between regions in the scalar graph, image patches in the first image, and image patches in the second image.

[0182] Optionally, the location of the one or more regions in the scalar graph is the same as the location of the one or more image blocks in the second image.

[0183] Optionally, method 400 further includes obtaining a target scalar value based on the scalar map, the target scalar value being used to indicate the quality of the second image.

[0184] Specifically, the target scalar value is obtained based on the scalar values ​​corresponding to one or more regions in the scalar graph.

[0185] For example, the average of the scalar values ​​corresponding to each region in the scalar graph can be used as the target scalar value.

[0186] Alternatively, the sum of the scalar values ​​corresponding to each region in the scalar graph can be used as the target scalar value.

[0187] Specifically, the scalar value corresponding to a region in the scalar map is determined based on the difference between the distribution of reconstruction error and the distribution of noise between the image patch corresponding to that region in the first image and the image patch corresponding to that region in the second image.

[0188] For example, if region A in the scalar corresponds to image block A in the first image and image block A' in the second image, then the scalar value corresponding to region A is determined based on the difference between the distribution of reconstruction error and the distribution of noise between image block A and image block A'.

[0189] The reconstruction error between image block A and image block A' is a region of the reconstruction error between the first image and the second image.

[0190] This application embodiment uses only the first region among the one or more regions as an example for illustration, and does not constitute a limitation on the solution of this application embodiment. The scalar values ​​corresponding to other regions among the one or more regions can be obtained in the same way as the first region. The first region can be any region among the one or more regions.

[0191] The scalar value corresponding to the first region is determined based on the difference between the distribution of reconstruction error and the distribution of noise between the first and second image blocks. The first image block belongs to the first image, and the second image block belongs to the second image.

[0192] The first image block and the second image block are corresponding image blocks. The first image block is located in the first image, and the second image block is located in the second image. The second image block is obtained by compressing the first image block.

[0193] The reconstruction error between the first image block and the second image block is a region within the reconstruction error between the first image and the second image.

[0194] The reconstruction error includes one or more error blocks. The dimension of the error block is less than or equal to the dimension of the reconstruction error. In other words, the error block is part or all of the reconstruction error.

[0195] For example, the error block can be obtained by randomly cropping from the reconstruction error. Alternatively, the error block can be obtained by randomly sampling from the reconstruction error. For instance, the dimension of the error block can be n*n, where n is an integer. One or more image blocks of dimension n*n are randomly cropped from the reconstruction error as error blocks.

[0196] The reconstruction error between the first image block and the second image block includes one or more first error blocks.

[0197] It should be noted that the term "first" in "first error block" only refers to the error block in the reconstruction error between the first image block and the second image block. In other words, any error block in the reconstruction error between the first image block and the second image block can be called the first error block.

[0198] The dimension of the first error block is less than or equal to the dimension of the reconstruction error between the first image block and the second image block. In other words, the first error block is part or all of the reconstruction error between the first image block and the second image block.

[0199] Thus, the distribution of the reconstruction error between the first image block and the second image block can also be called the distribution of the first error block. The distribution of the first error block can also be called the probability distribution of the first error block. The probability distribution can be expressed in the form of a probability density function.

[0200] For example, it can be done by Figure 3 The preprocessing module 320 processes the reconstruction error between the first image and the second image to obtain one or more error blocks, namely the first error block.

[0201] For example, the noise in the embodiments of this application can be Gaussian noise.

[0202] The noise consists of one or more noise blocks. The dimension of the noise block is less than or equal to the dimension of the noise. In other words, the noise block is part or all of the noise.

[0203] For example, the noise block can be obtained by randomly cropping from the noise. Or, the noise block can be obtained by randomly sampling from the noise. The dimension of the noise block can be the same as the dimension of the error block.

[0204] The distribution of noise can also be referred to as the distribution of noise blocks.

[0205] For example, it can be done by Figure 5 The preprocessing module 320 processes the Gaussian noise to obtain one or more noise blocks.

[0206] The quality of the second image can reflect the performance of the image processing model; for example, the quality of the second image can reflect the performance of the image compression model.

[0207] In this embodiment, the quality of the reconstructed image is evaluated from an information theory perspective. Specifically, the performance of the image processing model is evaluated based on the difference between the distribution of reconstruction error and the distribution of noise. The image processing model is trained based on the reconstruction error. The greater the difference between the distribution of reconstruction error and the distribution of noise, the more feature signals (e.g., stripes or text) are contained in the reconstruction error. During training, the image processing model primarily adjusts its parameters based on these feature signals, which is beneficial for achieving better processing results and thus improving the quality of the reconstructed image. In other words, the greater the difference between the distribution of reconstruction error and the distribution of noise, the greater the likelihood of a high-quality reconstructed image. This approach is largely consistent with human visual perception of image quality, providing a direct and reliable evaluation of image quality.

[0208] The following explanation uses image compression as an example. In lossy compression, compressing an image introduces compression errors, specifically reconstruction errors as described in this embodiment. Image compression algorithms reduce bitrate because they compress signals containing features from the original image, rather than noise. This embodiment compares the distribution of reconstruction error with the distribution of noise to determine whether the signal compressed by an image compression model is meaningful, i.e., contains features. The greater the difference between the distribution of reconstruction error and the distribution of noise, the more features the image compression model compresses. This indicates that the model's parameters were primarily adjusted based on features during training, leading to better compression performance, improved image quality, and reduced bitrate. In other words, the greater the difference between the distribution of reconstruction error and the distribution of noise, the higher the likelihood of high-quality compressed images. This aligns well with human perception of image quality, providing a direct and reliable evaluation of image quality.

[0209] Furthermore, the solution in this application embodiment can be used to measure the potential for further improving the image compression ratio. That is, if the difference between the distribution of reconstruction error and the distribution of noise is large, it indicates that the reconstruction error contains less noise signal and the potential for further improving the image compression ratio is small. If the difference between the distribution of reconstruction error and the distribution of noise is small, it indicates that the reconstruction error contains more noise signal and the potential for further improving the image compression ratio is large.

[0210] For the sake of brevity, the signal containing the feature is referred to as the feature signal in the embodiments of this application.

[0211] Optionally, the difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block is determined based on the difference between the energy of at least one first error block in the reconstruction error between the first image block and the energy of at least one noise block in the noise. The energy of at least one first error block is obtained by processing at least one first error block into an energy model, and the energy of at least one noise block is obtained by processing at least one noise block into the energy model. The energy model is either a first energy module or a second energy model. The energy value output by the first energy model indicates the probability that the data input to the first energy model conforms to the distribution of reconstruction error, and the energy value output by the second energy model indicates the probability that the data input to the second energy model conforms to the distribution of noise.

[0212] In this embodiment, energy-based models (EBMs) are used to explicitly model the probability distribution of high-dimensional data. Specifically, in this embodiment, energy models are used to explicitly model the probability distribution of reconstruction error and the probability distribution of noise. The difference between the distribution of reconstruction error and the distribution of noise is obtained by the difference between the energy of reconstruction error and the energy of noise, thus reducing the amount of computation.

[0213] The energy of at least one first error block and the energy of at least one noise block can be calculated using either a first energy model or a second energy model. It is acceptable as long as the energy of at least one first error block and the energy of at least one noise block are calculated using the same energy model.

[0214] The energy value output by the first energy model is used to indicate the probability that the data input to the first energy model conforms to the distribution of reconstruction errors between the first image block and the second image block. It can also be understood that the energy value output by the first energy model is used to indicate the degree of consistency between the data input to the first energy model and the distribution of reconstruction errors between the first image block and the second image block.

[0215] Specifically, the higher the energy value output by the first energy model, the lower the consistency between the data input to the first energy model and the distribution of reconstruction errors between the first and second image blocks. Alternatively, it can be understood as a lower probability that the data input to the first energy model follows the distribution of reconstruction errors between the first and second image blocks. Conversely, the lower the energy value output by the first energy model, the higher the consistency between the input data and the distribution of reconstruction errors between the first and second image blocks. Alternatively, it can be understood as a higher probability that the input data follows the distribution of the first error block.

[0216] The at least one first error block may come from the same reconstruction error, or it may come from different reconstruction errors.

[0217] For example, a first image block includes image block a, image block b, and image block c, and a second image block corresponding to the first image block includes image block a', image block b', and image block c'. Accordingly, the reconstruction error between the first and second image blocks includes image block a'", image block b'', and image block c''. Image block a'' is the reconstruction error between image block a and image block a'', image block b'' is the reconstruction error between image block b and image block b'', and image block c'' is the reconstruction error between image block c and image block c''. The at least one first error block can be sampled from at least one of the following: image block a'', image block b'', or image block c''. For example, the at least one first error block can be obtained from the same region or different regions of image block a'', image block b'', and image block c''. The number of first error blocks obtained in each reconstruction error can be the same or different.

[0218] Optionally, the first energy model is trained using multiple second error blocks from the reconstruction error between the third image block and the fourth image block corresponding to the third image block as training data.

[0219] The fourth image patch belongs to the fourth image, and the third image patch belongs to the third image. The fourth image is obtained by processing the initial image corresponding to the third image. Accordingly, the fourth image patch is obtained by processing the image patch in the initial image corresponding to the third image patch.

[0220] The third image can be understood as the target image of the initial image corresponding to the third image. Alternatively, the third image can be understood as the target that the image processing model expects to obtain after processing the initial image corresponding to the third image.

[0221] The specific processing flow is consistent with the processing flow of the initial image corresponding to the first image. For example, the second image is obtained by processing the first image using an image compression model, and the fourth image is obtained by processing the third image using the same image compression model. In other words, the initial image corresponding to the third image is the third image itself. The fourth image block is obtained by compressing the third image block. For a detailed description, please refer to the first and second images mentioned above.

[0222] It should be noted that the "second" in the term "second error block" only refers to the error block in the reconstruction error between the third and fourth image blocks. In other words, any error block in the reconstruction error between the third and fourth image blocks can be called the second error block.

[0223] It should be understood that the image compression model used to process the third image block and the image compression model used to process the first image are the same model. Therefore, the distribution of reconstruction error between the third and fourth image blocks is consistent with the distribution of reconstruction error between the first and second image blocks. For ease of description, the "distribution of reconstruction error between the first and second image blocks" and the "distribution of reconstruction error between the third and fourth image blocks" will be collectively referred to as the "distribution of reconstruction error" or "distribution of error blocks" in the following text. Unless otherwise specified in the following text, the "distribution of reconstruction error" can be understood as the "distribution of reconstruction error between the first and second image blocks".

[0224] There can be one third image block, and correspondingly, there can be one fourth image block. There can also be multiple third image blocks, and correspondingly, multiple fourth image blocks. This application does not limit the number of third and fourth image blocks, as long as there is a one-to-one correspondence between the third and fourth image blocks.

[0225] When there are multiple third and fourth image blocks, there are also multiple reconstruction errors between the third and fourth image blocks. These multiple second error blocks may originate from the same reconstruction error, or they may originate from multiple reconstruction errors.

[0226] For example, the third image block includes image block d, image block e, and image block f, and the fourth image block corresponding to the third image block includes image block d', image block e', and image block f'. Accordingly, the reconstruction error between the third and fourth image blocks includes image block d'", image block e'', and image block f''. Image block d'' is the reconstruction error between image block d and image block d', image block e'' is the reconstruction error between image block e and image block e', and image block f'' is the reconstruction error between image block f and image block f''. These multiple second error blocks can be sampled from at least one of the following: image block d'', image block e'', or image block f''. For example, these multiple second error blocks can be obtained from the same region or different regions of image block d'', image block e'', and image block f''. The number of second error blocks obtained in each reconstruction error can be the same or different.

[0227] The first and third image blocks can be the same or different. Similarly, the second and fourth image blocks can be the same or different.

[0228] In this way, through training, the data distribution represented by the first energy model is made consistent with or close to the distribution of the reconstruction error. In other words, the distribution of the reconstruction error is explicitly expressed through the first energy model. The first energy model can also be understood as an energy model of the reconstruction error.

[0229] The training process of the energy model will be explained below.

[0230] The energy model is defined as follows:

[0231] For data The probability distribution of this data in the energy model is based on the energy function f. θ () is determined, where θ represents the parameters of the energy model. f θ () is R d The points on the graph model the energy, i.e., f. θ (x) represents the energy of data point x. d represents the dimension of data point x, and N represents the number of data points x. In this embodiment, the energy of the input data is calculated through an energy model, which can be understood as calculating the energy of the input data through an energy function in the energy model.

[0232] Data point x is the input data for the energy model. For example, in the embodiments of this application, x can be an error block or a noise block.

[0233] For example, f θ () can be a neural network model.

[0234] Specifically, based on the energy function and the Gibbs distribution, the probability density function p in this energy model can be obtained. θ (x) satisfies the following formula:

[0235]

[0236] Where z represents the normalization constant. z satisfies the following formula:

[0237]

[0238] From the above formula, it can be concluded that for a data point x, in this energy model, the higher the energy of x, the lower the probability that x follows the distribution represented by the energy model.

[0239] The training objective of the energy model is to make the distribution represented by the energy model consistent with the distribution of the training data. In this embodiment, the training objective of the first energy model is to make the distribution represented by the first energy model consistent with the distribution of the reconstruction error.

[0240] Optionally, the first energy model is obtained by adjusting the parameters of the energy model during training with the goal of reducing the value of the loss function during training. The value of the loss function in the i-th iteration of training is determined based on the difference between the energy of multiple second error blocks and the energy of multiple target-generated data following the distribution represented by the energy model in the i-th iteration. The energy of the multiple second error blocks and the energy of the multiple target-generated data following the distribution represented by the energy model in the i-th iteration are calculated using the energy model in the i-th iteration. i is a positive integer.

[0241] The training process can include one iteration or multiple iterations.

[0242] For example, if i is 1, then the energy model for the i-th iteration is the initial energy model. In the first iteration, the parameters of the initial energy model are adjusted with the goal of reducing the loss function value in the first iteration. The loss function value in the first iteration is determined based on the difference between the energy of multiple second error blocks and the energy of multiple target-generated data that follows the distribution represented by the initial energy model. The energy of multiple second error blocks and the energy of multiple target-generated data that follow the distribution represented by the initial energy model are calculated using the initial energy model.

[0243] Multiple target-generated data that conforms to the distribution represented by the energy model refers to sampling multiple data points from the distribution represented by the energy model. For ease of description, multiple target-generated data that conforms to the distribution represented by the energy model can also be called multiple target-generated data corresponding to that energy model. Multiple target-generated data that conforms to the distribution represented by the initial energy model is the multiple target-generated data corresponding to the initial energy model.

[0244] For example, the loss function can be determined by maximum likelihood estimation (MLE). For instance, the loss function L during the training of the first energy model satisfies the following formula:

[0245]

[0246] Where, p data This represents the distribution of the training data, specifically the distribution of the error blocks in the embodiments of this application. This represents the mean energy of the data that follows the distribution of the training data. For example, The value can be the mean energy of all or part of the training data. In the embodiments of this application, it can be achieved through f. θ () Calculate the energy of multiple second error blocks used as training data, and take the average energy of these multiple second error blocks as... The value of p. θ This represents the distribution as depicted by the energy model. This represents the mean energy of the data that follows a distribution represented by an energy model. It can be abbreviated as W + , It can be abbreviated as E_. This energy model is the energy model in the iterative process.

[0247] During the training of the first energy model, the parameters of the initial energy model are adjusted with the goal of reducing the value of the loss function L, resulting in a trained energy model, i.e., the first energy model. Alternatively, it can be understood that during the training of the first energy model, the parameters of the energy model are adjusted with the goal of reducing the energy corresponding to the training data and increasing the energy corresponding to the data that follows the distribution represented by the energy model, resulting in a trained energy model, i.e., the first energy model.

[0248] The second term in the loss function L requires calculating the energy of the data that follows the distribution represented by the energy model in the current iteration process. This involves sampling data from the distribution represented by the energy model in the current iteration process, i.e., x ~ p. θ Data sampled from the distribution represented by the energy model in the current iteration process can also be called generated data or sampled data corresponding to the energy model in the current iteration process.

[0249] Optionally, the multiple target generation data that follow the distribution represented by the energy model in the i-th iteration process are obtained by multiple iterations of the initial generation data of the energy model in the i-th iteration process. The generation data obtained in the (k+1)-th iteration is determined based on the generation data obtained in the k-th iteration, the energy gradient of the generation data obtained in the k-th iteration, and Gaussian noise. The coefficients of the energy gradient of the generation data obtained in the k-th iteration and the coefficients of the Gaussian noise are uncorrelated, and k is a positive integer.

[0250] For example, if i is 1, then the energy model of the i-th iteration is the initial energy model. Multiple target-generated data that conform to the distribution represented by the initial energy model are obtained through multiple iterative adjustments based on the initial generated data of the initial energy model. Specifically, the generated data obtained in the (k+1)-th iteration is determined based on the generated data obtained in the k-th iteration, the energy gradient of the generated data obtained in the k-th iteration, and Gaussian noise. k is a positive integer.

[0251] For example, Langevin Dynamics can be used for Monte Carlo Markov Chain (MCMC) sampling. For instance, the generated data from k+1 iterations satisfies the following formula:

[0252]

[0253] in, This represents the generated data obtained in the (k+1)th iteration. This represents the generated data obtained in the k-th iteration, where s represents the iteration step size, s > 0, and ε noise ~N(0, I) represents standard Gaussian noise. As the number of iterations approaches infinity, i.e., k→∞, Follows distribution p θ .

[0254] The number of iteration steps can be set as needed. For example, the number of iteration steps can be set to 20, that is, the generated data obtained in the 20th iteration is the generated data of multiple targets that follow the distribution represented by the initial energy model.

[0255] Optionally, the coefficients of the energy gradient of the generated data obtained in the k-th iteration are uncorrelated with the coefficients of the Gaussian noise.

[0256] In other words, the coefficients of the energy gradient and the coefficients of the Gaussian noise are independent.

[0257] The generated data obtained in the (k+1)th iteration satisfies the following formula:

[0258]

[0259] Where α represents the coefficient of the energy gradient of the generated data obtained in the k-th iteration, or simply the coefficient of the gradient term, and β represents the coefficient of the Gaussian noise, or simply the coefficient of the noise term. α > β > 0. For example, α is 0.1 and β is 0.001.

[0260] Two positive real numbers (α and β) are used to adjust the gradient term (the second term in the above equation) and the noise term (the third term in the above equation) in each iteration. This avoids the problem that the energy of the generated samples in the next iteration cannot decrease due to an excessively small gradient term, achieving a smoother sampling process and improving the stability of the training process.

[0261] For example, the initial generation data of the energy model in the i-th iteration process can be determined based on Gaussian noise.

[0262] Optionally, the initial generated data of the energy model in the i-th iteration process is determined based on multiple target generated data that conform to Gaussian noise and the distribution represented by the energy model in the iteration process before the i-th iteration.

[0263] In other words, the initial generated data of the energy model in the current iteration process can be determined based on Gaussian noise and multiple target generated data corresponding to the energy model in the previous iteration process.

[0264] It should be noted that the iterative process of the energy model and the iterative process of generating data are two different iterative processes. During the iterative process of generating data, the parameters of the energy model remain unchanged. In each iteration of the energy model, multiple generated data points that conform to the distribution represented by the energy model in the current iteration are obtained through iterative generation of data.

[0265] For example, the initial generated data of the energy model in the i-th iteration process can be sampled from multiple target generated data that are Gaussian noise and follow the distribution represented by the energy model in the iteration process before the i-th iteration.

[0266] For example, in the initial generated data of the current iteration, part of it is the target generated data obtained in the previous iteration, and the other part is Gaussian noise.

[0267] In the first iteration, the energy model is energy model A. The initial generated data in this iteration can be determined based on Gaussian noise. Multiple target generated data conforming to the distribution of energy model A are obtained through multiple iterations of the Gaussian noise. In the second iteration, the energy model is energy model B, which is obtained by adjusting the parameters of energy model A. Part of the initial generated data for energy model B in this iteration is determined based on the multiple target generated data conforming to the distribution of energy model A, and the other part is determined based on Gaussian noise. Multiple target generated data conforming to the distribution of energy model B are obtained through multiple iterations of the initial generated data of energy model B.

[0268] Introducing target generation data obtained from previous iterations as initial generation data can further reduce the energy of the obtained samples, which helps ensure that the distribution of multiple target generation data is more consistent with the distribution represented by the energy model in the current iteration. Furthermore, determining the initial generation data based on the target generation data obtained from previous iterations and Gaussian noise helps improve the diversity of target generation data corresponding to the energy model in the current iteration, thereby improving the training effect of the energy model.

[0269] The ratio of Gaussian noise in the initial generated data of the energy model in the current iteration process to the target generated data obtained in the previous iteration process can be set as needed.

[0270] For example, Gaussian noise accounts for 50% of the initial generated data, and the target generated data obtained in previous iterations accounts for 50% of the initial generated data in the current iteration.

[0271] For example, the target generation data obtained during previous iterations can be stored in a buffer.

[0272] Optionally, the value of the loss function in the i-th iteration process is determined based on the difference between the energy of multiple second error blocks and the energy of multiple target-generated data that follow the distribution represented by the energy model in the i-th iteration process, the square of the energy of multiple second error blocks, and the square of the energy of multiple target-generated data that follow the distribution represented by the energy model in the i-th iteration process.

[0273] For example, the difference between the energy of the multiple second error blocks and the energy of the multiple target generated data can be the difference between the average energy of the multiple second error blocks and the average energy of the multiple target generated data. In this case, the number of second error blocks and the number of target generated data can be the same or different.

[0274] Alternatively, the difference between the energy of multiple second error blocks and the energy of multiple target generated data can be the difference between the sum of the energy of the multiple second error blocks and the sum of the energy of the multiple target generated data. This is acceptable as long as the number of second error blocks is the same as the number of target generated data.

[0275] For example, the loss function satisfies the following formula:

[0276]

[0277] Where λ is the coefficient of the regularization term (i.e., the squared term). λ is a positive number. The value of λ can be set as needed. For example, λ can be 1.

[0278] Optionally, the value of the loss function in the i-th iteration process is determined based on the difference between the energy of multiple second error blocks and the energy of multiple target generated data that follow the distribution represented by the energy model in the i-th iteration process, the absolute value of the energy of multiple second error blocks, and the absolute value of the energy of multiple target generated data that follow the distribution represented by the energy model in the i-th iteration process.

[0279] In this way, a regularization term is added to the energy of all data in the loss function. For example, the squares of the energy of multiple second error blocks and the squares of the energy of multiple target generated data are introduced into the loss function to limit the absolute value of the energy of positive sample data (i.e., multiple second error blocks) and negative sample data (i.e., multiple target generated data). This avoids the loss function from failing to decrease due to the excessive difference between the two energies, which helps to ensure the training effect of the energy model.

[0280] For example, the scalar value corresponding to the first region includes any one of the following: the value of the first evaluation indicator, the value of the second evaluation indicator, or the value of the third evaluation indicator.

[0281] The first evaluation metric uses the distribution of reconstruction error as the standard distribution to measure the difference between the distribution of reconstruction error and the distribution of noise.

[0282] The second evaluation metric uses the distribution of reconstruction error as the standard distribution to measure the difference between the distribution of reconstruction error and the distribution of noise.

[0283] The third evaluation metric is the Jeffreys divergence between the distribution of reconstruction error and the distribution of noise.

[0284] The following describes the three evaluation indicators (first evaluation indicator, second evaluation indicator, and third evaluation indicator) provided in the embodiments of this application.

[0285] For example, the first evaluation index satisfies the following formula:

[0286]

[0287] Where Metric1 represents the first evaluation index, p error p represents the distribution of reconstruction error. noise p represents the distribution of noise. error (ε) represents the probability value of ε calculated using the probability density function of the reconstruction error, where ε represents the sampled value, and ε ~ p. error The sampled values ​​represent the distribution of the reconstruction error, i.e., the at least one first error block in the embodiments of this application, ε~p. noise This represents the sampled value that conforms to the noise distribution, i.e., the at least one noise block in the embodiments of this application. E() represents the calculated average value, and H(p) represents the average value. error ) represents the entropy of the distribution of reconstruction error, CE(p) noise ||p error ) represents the cross-entropy between the distribution of noise and the distribution of reconstruction error.

[0288] The first evaluation metric uses the distribution of reconstruction error as the standard distribution to measure the difference between the distribution of reconstruction error and the distribution of noise. The smaller the value of Metric1, the better the distribution of reconstruction error p... error With the distribution of noise p noise The greater the difference between them, that is, the farther the distribution of reconstruction error is from the distribution of noise, it indicates that there are more characteristic signals than noise signals in the reconstruction error.

[0289] For example, the second evaluation index satisfies the following formula:

[0290]

[0291] Where Metric2 represents the second evaluation index, H(p) noise ) represents the entropy of the noise distribution, CE(p) error ||p noise) represents the cross-entropy between the distribution of reconstruction error and the distribution of noise, p noise (ε) represents the probability value of ε calculated using the probability density function of the noise. The second evaluation metric uses the noise distribution as a standard distribution to measure the difference between the distribution of the reconstruction error and the distribution of the noise. The smaller the value of Metric2, the more accurate the distribution of the reconstruction error p... error With the distribution of noise p noise The greater the difference between them, that is, the farther the distribution of reconstruction error is from the distribution of noise, it indicates that there are more characteristic signals than noise signals in the reconstruction error.

[0292] The value of the third evaluation indicator is determined based on the values ​​of the first and second evaluation indicators.

[0293] For example, the third evaluation index satisfies the following formula:

[0294] Metric3=Metric1+Metric2=-D JD (p error ||p noise );

[0295] Metric3 represents the third evaluation indicator, D JD (p error ||p noise ) represents the Jeffreys divergence between the distribution of reconstruction error and the distribution of noise.

[0296]

[0297] Among them, KL(p error ||p noise ) represents the Kullback-Leibler divergence between the distribution of reconstruction error and the distribution of noise. noise ||p error Metric3 represents the KL divergence between the noise distribution and the reconstruction error distribution. A smaller Metric3 value indicates a larger KL divergence between the reconstruction error distribution and the noise distribution, and thus a more stable reconstruction error distribution. error With the distribution of noise p noise The greater the difference between them, that is, the farther the distribution of reconstruction error is from the distribution of noise, it indicates that there are more characteristic signals than noise signals in the reconstruction error.

[0298] Optionally, step S403 includes: calculating a scalar value corresponding to the first region based on the difference between the energy of at least one first error block and the energy of at least one noise block.

[0299] Optionally, the value of the first evaluation index is determined based on the difference between the energy of at least one first error block and the energy of at least one noise block. The energy of at least one first error block is obtained by processing the at least one first error block into the first energy model, and the energy of at least one noise block is obtained by processing the at least one noise block into the first energy model.

[0300] The higher the energy value output by the first energy model, the lower the consistency between the distribution of input data and reconstruction error; or, it can be understood as a lower probability that the input data belongs to the reconstruction error. Conversely, the lower the energy value output by the first energy model, the higher the consistency between the distribution of input data and reconstruction error; or, it can be understood as a higher probability that the input data belongs to the reconstruction error.

[0301] Specifically, the at least one first error block is input into the first energy model to obtain the energy of the at least one first error block; the at least one noise block is input into the first energy model to obtain the energy of the at least one noise block. The value of the first evaluation index is determined based on the difference between the energy of the at least one first error block and the energy of the at least one noise block.

[0302] For example, based on the formula of the above energy model, the first evaluation index can be obtained by satisfying the following formula:

[0303]

[0304] f() represents the energy of the input data calculated using the first energy model. This represents the average energy of at least one first error block calculated using the first energy model. This represents the average energy of the at least one noise block calculated using the first energy model.

[0305] Optionally, the value of the second evaluation index is determined based on the difference between the energy of at least one first error block and the energy of at least one noise block. The energy of at least one first error block is obtained by processing the at least one first error block into the second energy model, and the energy of at least one noise block is obtained by processing the at least one noise block into the second energy model.

[0306] The higher the energy value output by the second energy model, the lower the consistency between the distribution of input data and noise; or, it can be understood as a lower probability that the input data belongs to noise. Conversely, the lower the energy value output by the second energy model, the higher the consistency between the distribution of input data and noise; or, it can be understood as a higher probability that the input data belongs to noise.

[0307] Specifically, the at least one second error block is input into the second energy model to obtain the energy of the at least one second error block; the at least one noise block is input into the second energy model to obtain the energy of the at least one noise block. The value of the second evaluation index is determined based on the difference between the energy of the at least one second error block and the energy of the at least one noise block.

[0308] For example, the second evaluation index satisfies the following formula:

[0309]

[0310] g() represents the energy of the input data calculated using the second energy model. This represents the average energy of at least one first error block calculated using the second energy model. This represents the average energy of at least one noise block calculated using the second energy model.

[0311] As mentioned above, the noise in the embodiments of this application can be Gaussian noise.

[0312] Gaussian noise refers to a type of noise whose probability density function follows a Gaussian distribution.

[0313] The probability density function of Gaussian noise ε satisfies the following formula:

[0314]

[0315] Where μ represents the mean of the Gaussian distribution and d represents the dimension of the error block.

[0316] Based on the Gibbs distribution, the energy function of the noise satisfies the following formula:

[0317]

[0318] The energy function of noise is the same as the energy function in the second energy model.

[0319] Optionally, the value of the third evaluation indicator is the sum of the values ​​of the first evaluation indicator and the second evaluation indicator.

[0320] For example, the third evaluation index satisfies the following formula:

[0321]

[0322] Figure 5 The energy of different error blocks calculated using the first energy model is shown. For example... Figure 6 As shown, the more obvious the characteristic signal, the smaller the energy corresponding to the error block; the more obvious the noise signal, the higher the energy corresponding to the error block.

[0323] Table 1

[0324]

[0325] Table 2

[0326]

[0327] Tables 1 through 4 show the evaluation metrics for different image compression algorithms. Tables 1 and 2 show the evaluation metrics for different image compression algorithms on the CLIC 2020 dataset. That is, the first image comes from the CLIC 2020 dataset.

[0328] Tables 3 and 4 show the evaluation metrics of different image compression algorithms on the CLIC Professional dataset. That is, the first image is from the CLIC Professional dataset.

[0329] Table 3

[0330]

[0331] Table 4

[0332]

[0333] In Tables 1 to 4, the reconstruction errors of gmm, ctx_gmm, and rnab_ctx_gmm are constructed based on multi-scale structural similarity index (MSSSIM) and mean-square error (MSE), respectively. The parameter values ​​refer to the coefficients of the reconstruction error used by the image compression algorithm during training.

[0334] Figure 6 A schematic diagram illustrating the reconstruction error obtained after image compression using some of the image compression algorithms in Table 2 is shown. Specifically, Figure 6 (a) shows the gmm algorithm with parameter value λ mse The reconstruction error obtained when it is 0.002 Figure 6 (b) shows the gmm algorithm with parameter value λ mse The reconstruction error obtained when the value is 0.05 Figure 6 (c) shows the reconstruction error obtained by the e2e algorithm. Combined with Table 2, it can be seen that... Figure 7 The image compression algorithm in (b) has the highest evaluation index value, the most noise in the reconstruction error, and the fewest feature signals.

[0335] Scalar graphs can be applied in various scenarios. For example, they can be used to select the optimal image compression algorithm to improve image compression quality. Furthermore, scalar graphs can be used to determine whether an image compression algorithm has the potential to improve the image compression ratio, i.e., whether the algorithm can be further trained to increase the compression ratio. Method 400 can be combined with Method 700 described later to further improve image compression quality or compression ratio.

[0336] Figure 7 This application illustrates a processing method for an image processing model provided by an embodiment of the present application. Figure 8 to Figure 10 The method shown can be executed by a computing device, which can be a cloud service device or a terminal device, such as a computer, server, mobile phone, camera, vehicle, drone, or robot, or a system composed of cloud service devices and terminal devices. Some content in method 700 is consistent with that in method 400. To avoid repetition, some descriptions are omitted when describing method 700.

[0337] Method 700 includes steps S701 to S703.

[0338] S701, Obtain the first image and the second image corresponding to the first image. The second image is obtained by processing the initial image through an image processing model.

[0339] For example, step S701 can be performed by the algorithm module 310 in the device 300.

[0340] Specifically, the second image can be obtained by processing the initial image using an image processing model.

[0341] The first image is the target image corresponding to the initial image. In other words, the first image can be understood as the target that the image processing model expects to obtain after processing the initial image.

[0342] It should be noted that there can be one first image, and correspondingly, there can be one second image. There can also be multiple first images, and correspondingly, multiple second images. This application does not limit the number of first and second images, as long as there is a one-to-one correspondence between the first and second images.

[0343] The first image and the initial image can be the same or different.

[0344] For example, the image processing model can be an image compression model. The first image is compressed using this model to obtain the second image. The first image is the initial image. In other words, the second image is the image obtained by compressing the first image. The second image can also be called the reconstructed image.

[0345] Step S701 is an optional step.

[0346] For a detailed description of step S701, please refer to step S401, which will not be repeated here.

[0347] For ease of description, the following text will only use the image processing model as an image compression model to illustrate method 700, that is, the second image is obtained by compressing the first image, and does not constitute a limitation on the solution of the embodiments of this application.

[0348] S702, obtain the reconstruction error between the first image and the second image.

[0349] For example, step S702 can be performed by the preprocessing module 320 in the device 300.

[0350] For a detailed description of step S702, please refer to step S402, which will not be repeated here.

[0351] S703 updates the image processing model based on the difference between the distribution of reconstruction error and the distribution of noise between the first and second images.

[0352] For example, step S703 can be performed by the algorithm processing module 340 in the device 300.

[0353] According to the scheme of the embodiments of this application, the difference between the distribution of reconstruction error and the distribution of noise can distinguish between meaningless noise signals and meaningful feature signals in the reconstruction error, which is beneficial for updating the image processing model to improve the quality of the reconstructed image.

[0354] Furthermore, the solution of this application can measure the performance of the image processing model by reconstructing the difference between the distribution of error and the distribution of noise, and then select an appropriate way to process the image processing model, which is beneficial to improving the quality of the reconstructed image.

[0355] For example, step S703 includes replacing the image processing model based on the difference between the distribution of reconstruction error and the distribution of noise between the first image and the second image.

[0356] For example, method 400 and method 700 can be combined.

[0357] Taking the image processing model as an image compression model as an example, specifically, step S703 may include: generating a scalar map based on the difference between the distribution of reconstruction error and the distribution of noise between the first image and the second image; and replacing the image compression model according to the scalar map. The scalar map includes one or more regions, and the scalar value corresponding to each of the one or more regions is used to indicate the quality of one or more image patches in the second image.

[0358] For example, if the scalar values ​​corresponding to m regions in the scalar image are within the first threshold range, the image compression model is replaced. m is a positive integer.

[0359] The first threshold range can be set as needed.

[0360] For example, if the scalar value is the value of the first evaluation index, and the image compression model is replaced when the scalar values ​​corresponding to m regions are greater than or equal to the first threshold.

[0361] In other words, if the quality of m image blocks in the second image does not meet expectations, the image compression model can be replaced.

[0362] For example, the target scalar value is obtained from the scalar graph, and if the target scalar value is within the range of the second threshold, the image compression model is replaced.

[0363] The first threshold range can be set as needed.

[0364] For example, if the scalar value is the value of the first evaluation indicator, and the image compression model is replaced if the target scalar value is greater than or equal to the second threshold.

[0365] In other words, if the overall quality of the second image does not meet expectations, the image compression model can be replaced.

[0366] It should be understood that the above are merely examples and do not constitute a limitation on the solutions of the embodiments of this application.

[0367] Optionally, step S703 includes: adjusting the reconstruction error between the first image patch and the second image patch with the goal of increasing the difference between the distribution of reconstruction error and the distribution of noise between the first image patch and the second image patch, to obtain the adjusted reconstruction error; and training an image processing model based on the adjusted reconstruction error. The first image patch belongs to the first image, and the second image patch belongs to the second image.

[0368] The descriptions related to the first image block and the second image block can be found in step S403 of method 400, and will not be repeated here.

[0369] In other words, the parameters in the image processing model are updated with the goal of reducing the adjusted reconstruction error.

[0370] The specific training method can adopt the original training method of the image processing model, and this application embodiment does not limit it.

[0371] According to the scheme of the embodiments of this application, by increasing the difference between the distribution of reconstruction error and the distribution of noise, meaningless noise signals and meaningful feature signals can be separated from the reconstruction error, or in other words, noise signals can be removed from the reconstruction error. Based on the adjusted reconstruction error, the training of the image processing model can be guided, which can ensure that the image processing model is trained with the goal of reducing feature signals as much as possible. This is beneficial to retain the feature signals in the reconstructed image, improve the image quality, and thus improve the performance of the model.

[0372] Optionally, the reconstruction error between the first image block and the second image block is adjusted with the goal of reducing the energy of the reconstruction error between the first image block and the second image block to obtain the adjusted reconstruction error. The energy of the reconstruction error between the first image block and the second image block is calculated by the first energy model. The energy value output by the first energy model is used to indicate the probability that the data input to the first energy model follows the distribution of the reconstruction error between the first image block and the second image block.

[0373] In this embodiment, an energy model is used to explicitly model the probability distribution of reconstruction error and the probability distribution of noise. The difference between the distribution of reconstruction error and the distribution of noise is obtained by the difference between the energy of reconstruction error and the energy of noise, which reduces the amount of computation.

[0374] The relevant description of the first energy model can be found in Method 400, and will not be repeated here.

[0375] When the energy of the reconstruction error between the first and second image blocks is calculated using the first energy model, the lower the energy of the reconstruction error between the first and second image blocks, the greater the difference between the distribution of the reconstruction error and the distribution of noise. Reducing the energy of the reconstruction error between the first and second image blocks increases the difference between their distributions.

[0376] The reconstruction error between the first image block and the second image block may include one or more first error blocks. Adjusting the reconstruction error between the first image block and the second image block can be understood as adjusting some or all of the first error blocks in the reconstruction error between the first image block and the second image block. Adjusting the reconstruction error between the first image block and the second image block with the goal of reducing the energy of the reconstruction error between the first image block and the second image block can also be understood as adjusting some or all of the first error blocks with the goal of reducing the energy of some or all of the first error blocks in the reconstruction error between the first image block and the second image block, to obtain the adjusted some or all of the first error blocks.

[0377] In other words, if a portion of the first error block in the reconstruction error between the first image block and the second image block is adjusted, the adjusted reconstruction error includes the adjusted portion of the first error block and the remaining unadjusted portion of the reconstruction error between the first image and the second image. If all the first error blocks are adjusted, the adjusted reconstruction error includes all the adjusted first error blocks and the remaining unadjusted portion of the reconstruction error between the first image and the second image.

[0378] For ease of description, the following explanation uses a first error block as an example to illustrate how to adjust the reconstruction error.

[0379] For example, one error block ε′ in the adjusted reconstruction error satisfies the following formula:

[0380]

[0381] Where ε represents the first error block in the reconstruction error between the first and second images, ε′ represents the first error block in the adjusted reconstruction error, and η represents the step size for energy reduction. The value of η can be set as needed; for example, η can be 0.1. Each first error block in the reconstruction error can be adjusted in the above manner to obtain the adjusted reconstruction error.

[0382] Figure 8 to Figure 10 A comparison chart of the first error block before and after adjustment is shown. Figure 8 In the figure, (a) shows the original error block, i.e. the first error block before adjustment, (b) shows the first error block after adjustment, and (c) shows the difference between the first error block before adjustment and the first error block after adjustment.

[0383] like Figure 8 As shown in Figure (c), the difference between the first error block before and after adjustment contains a significant amount of noise. Figure 8 (a) and Figure 9A comparison with Figure (b) reveals that by reducing the energy of the first error block, the number of characteristic signals in the adjusted first error block increases while the noise signal decreases.

[0384] Furthermore, Figure 8 Figure (a) is in Figure 9 The result is obtained by adding Gaussian noise to Figure (a). By reducing the energy of the first error block, the noise signal in the first error block can be reduced, i.e. Figure 9 Figure (c), compared to Figure 9 Regarding diagram (a), Figure 10 The noise signal in Figure (b) is significantly reduced.

[0385] Furthermore, Figure 10 Figure (a) shows Gaussian noise, which is used as the first error block before adjustment. By reducing the energy of the first error block, the noise signal in the first error block can be removed. Figure 10 Figure (c), compared to Figure 10 Regarding diagram (a), Figure 11 to Figure 14 The noise signal in Figure (b) is significantly reduced.

[0386] Optionally, the reconstruction error between the first image block and the second image block is adjusted with the goal of increasing the energy of the reconstruction error between the first image block and the second image block to obtain the adjusted reconstruction error. The energy of the reconstruction error between the first image block and the second image block is calculated by the second energy model.

[0387] The relevant description of the second energy model can be found in Method 400, and will not be repeated here.

[0388] When the energy of the reconstruction error between the first and second image blocks is calculated using the second energy model, the higher the energy of the reconstruction error between the first and second image blocks, the greater the difference between the distribution of the reconstruction error and the noise distribution. Increasing the energy of the reconstruction error between the first and second image blocks can increase the difference between the distribution of the reconstruction error and the noise distribution.

[0389] For specific handling methods, please refer to the previous text, which will not be repeated here.

[0390] As mentioned above, methods 400 and 700 can be used in combination.

[0391] For example, taking an image processing model as an image compression model, a scalar map is generated based on the difference between the distribution of reconstruction error and the distribution of noise between the first image and the second image. The scalar map includes one or more regions, and the scalar value corresponding to each of the one or more regions is used to indicate the quality of one or more image blocks in the second image. The scalar value corresponding to the first region is determined based on the difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block. When the scalar value corresponding to the first region is within a third threshold range, the reconstruction error between the first image block and the second image block is adjusted with the goal of increasing the difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block, resulting in an adjusted reconstruction error. The image compression model is trained based on the adjusted reconstruction error.

[0392] The third threshold range can be set as needed.

[0393] For example, if the scalar value is the value of the first evaluation index, and the scalar value corresponding to the first region is greater than or equal to the third threshold, the reconstruction error between the first image block and the second image block is adjusted with the goal of increasing the difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block, and the adjusted reconstruction error is obtained; the image compression model is trained based on the adjusted reconstruction error.

[0394] In other words, if the scalar value corresponding to the first region is greater than or equal to the third threshold, it means that the region has the potential for further compression, or that the image compression ratio of the second image patch can be further improved. That is, the image compression model can be further trained to improve the image compression ratio of the second image patch.

[0395] It should be understood that the above are merely examples, and methods 400 and 700 can also be combined in other ways, which are not limited in this application embodiment.

[0396] The following is combined Figure 11 The apparatus of the embodiments of this application will be described below. It should be understood that the apparatus described below is capable of performing the methods of the foregoing embodiments of this application. To avoid unnecessary repetition, repeated descriptions will be appropriately omitted when describing the apparatus of the embodiments of this application below.

[0397] Figure 11 This is a schematic block diagram of the image processing apparatus provided in the embodiments of this application. Figure 12 The image processing apparatus 3000 shown includes an acquisition unit 3010 and a processing unit 3020.

[0398] The acquisition unit 3010 and the processing unit 3020 can be used to execute the image processing method 400 of the embodiments of this application.

[0399] Specifically, the acquisition unit 3010 is used to acquire the reconstruction error between the first image and the second image corresponding to the first image.

[0400] The processing unit 3020 generates a scalar map based on the difference between the distribution of reconstruction error and the distribution of noise between the first image and the second image. The scalar map includes one or more regions, and the scalar value corresponding to each of the one or more regions is used to indicate the quality of one or more image patches in the second image.

[0401] Optionally, the second image is obtained by compressing the first image.

[0402] Optionally, one or more regions may be located in the same position in the scalar graph as one or more image patches may be located in the second image.

[0403] Optionally, the processing unit 3020 is further configured to: obtain a target scalar value based on the scalar map, the target scalar value being used to indicate the quality of the second image.

[0404] Optionally, the scalar value corresponding to the first region in one or more regions is determined based on the difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block, where the first image block belongs to the first image and the second image block belongs to the second image. The difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block is determined based on the difference between the energy of at least one first error block in the reconstruction error between the first image block and the second image block and the energy of at least one noise block in the noise. The energy of at least one first error block is obtained by processing at least one first error block into a first energy model or a second energy model, and the energy of at least one noise block is obtained by processing at least one noise block into a first energy model or a second energy model. The energy value output by the first energy model is used to indicate the probability that the data input to the first energy model conforms to the distribution of reconstruction error between the first image block and the second image block, and the energy value output by the second energy model is used to indicate the probability that the data input to the second energy model conforms to the distribution of noise.

[0405] Optionally, the first energy model is trained using multiple second error blocks from the reconstruction error between the third image block and the fourth image block corresponding to the third image block as training data.

[0406] Optionally, the first energy model is obtained by adjusting the parameters of the energy model during training with the goal of reducing the value of the loss function during training. The value of the loss function in the i-th iteration during training is determined based on the difference between the energy of multiple second error blocks and the energy of multiple target generated data that follow the distribution represented by the energy model in the i-th iteration. The energy of multiple second error blocks and the energy of multiple target generated data that follow the distribution represented by the energy model in the i-th iteration are calculated by the energy model in the i-th iteration, where i is a positive integer.

[0407] Optionally, the multiple target generation data that follow the distribution represented by the energy model in the i-th iteration process are obtained by multiple iterations of the initial generation data of the energy model in the i-th iteration process. The generation data obtained in the (k+1)-th iteration of the initial generation data is determined based on the generation data obtained in the k-th iteration, the energy gradient of the generation data obtained in the k-th iteration, and Gaussian noise. The coefficient of the energy gradient of the generation data obtained in the k-th iteration is greater than the coefficient of the Gaussian noise, and k is a positive integer.

[0408] Optionally, the initial generated data of the energy model in the i-th iteration process is determined based on multiple target generated data that conform to Gaussian noise and the distribution represented by the energy model in the iteration process before the i-th iteration.

[0409] Optionally, the value of the loss function in the i-th iteration process is determined based on the difference between the energy of multiple second error blocks and the energy of multiple target-generated data that follow the distribution represented by the energy model in the i-th iteration process, the square of the energy of multiple second error blocks, and the square of the energy of multiple target-generated data that follow the distribution represented by the energy model in the i-th iteration process.

[0410] Optionally, the scalar value corresponding to the first region includes any one of the following: the value of a first evaluation index, the value of a second evaluation index, or the value of a third evaluation index; the value of the first evaluation index is determined based on the difference between the energy of at least one first error block and the energy of at least one noise block, the energy of at least one first error block being obtained by processing at least one first error block into a first energy model, and the energy of at least one noise block being obtained by processing at least one noise block into a first energy model; the value of the second evaluation index is determined based on the difference between the energy of at least one first error block and the energy of at least one noise block, the energy of at least one first error block being obtained by processing at least one first error block into a second energy model, and the energy of at least one noise block being obtained by processing at least one noise block into a second energy model; the value of the third evaluation index is determined based on the values ​​of the first and second evaluation indices.

[0411] Optionally, the first evaluation index satisfies the following formula:

[0412]

[0413] Where Metric1 represents the first evaluation index, f() represents the energy of the input data calculated using the first energy model, and ε~p error Denotes at least one first error block, ε~p noise Indicates at least one noise block. This represents the average energy of at least one first error block calculated using the first energy model. This represents the average energy of at least one noise block calculated using the first energy model.

[0414] Optionally, the second evaluation index satisfies the following formula:

[0415]

[0416] Where Metric2 represents the second evaluation index, g() represents the energy of the input data calculated using the second energy model, and ε~p error Denotes at least one first error block, ε~p noise Indicates at least one noise block. This represents the average energy of at least one first error block calculated using the second energy model. This represents the average energy of at least one noise block calculated using the second energy model.

[0417] Figure 12 This is a schematic block diagram of the processing apparatus of the image processing model according to an embodiment of this application. Figure 13The image processing device 4000 shown includes an acquisition unit 4010 and a processing unit 4020.

[0418] The acquisition unit 4010 and the processing unit 4020 can be used to execute method 700.

[0419] Specifically, the acquisition unit 4010 is used to acquire the reconstruction error between the first image and the second image corresponding to the first image, wherein the second image is obtained by processing the initial image through an image processing model.

[0420] The processing unit 4020 is used to update the image processing model based on the difference between the distribution of reconstruction error and the distribution of noise between the first image and the second image.

[0421] Optionally, the image processing model is an image compression model, and the initial image is the first image.

[0422] Optionally, the processing unit 4020 is specifically used to: adjust the reconstruction error between the first image and the second image with the goal of increasing the difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block, so as to obtain the adjusted reconstruction error, wherein the first image block belongs to the first image and the second image block belongs to the second image; and train an image processing model based on the adjusted reconstruction error.

[0423] Optionally, the processing unit 4020 is specifically used to: adjust the reconstruction error between the first image and the second image with the goal of reducing the energy of the reconstruction error between the first image block and the second image block, so as to obtain the adjusted reconstruction error. The energy between the first image block and the second image block is calculated by a first energy model. The energy value output by the first energy model is used to indicate the probability that the data input to the first energy model follows the distribution of the reconstruction error between the first image block and the second image block.

[0424] Optionally, the first error block in the adjusted reconstruction error satisfies the following formula:

[0425]

[0426] Where ε represents the first error block in the reconstruction error between the first image block and the second image block, ε′ represents the first error block in the adjusted reconstruction error, η represents the step size of energy reduction, and f() represents the energy of the input data calculated by the first energy model.

[0427] Optionally, the first energy model is trained using multiple second error blocks from the reconstruction error between the third image block and the fourth image block corresponding to the third image block as training data.

[0428] Optionally, the first energy model is obtained by adjusting the parameters of the energy model during training with the goal of reducing the value of the loss function during training. The value of the loss function in the i-th iteration during training is determined based on the difference between the energy of multiple second error blocks and the energy of multiple target generated data that follow the distribution represented by the energy model in the i-th iteration. The energy of multiple second error blocks and the energy of multiple target generated data that follow the distribution represented by the energy model in the i-th iteration are calculated by the energy model in the i-th iteration, where i is a positive integer.

[0429] Optionally, the multiple target generation data that follow the distribution represented by the energy model in the i-th iteration process are obtained by multiple iterations of the initial generation data of the energy model in the i-th iteration process. The generation data obtained in the (k+1)-th iteration of the initial generation data is determined based on the generation data obtained in the k-th iteration, the energy gradient of the generation data obtained in the k-th iteration, and Gaussian noise. The coefficient of the energy gradient of the generation data obtained in the k-th iteration is greater than the coefficient of the Gaussian noise, and k is a positive integer.

[0430] Optionally, the initial generated data of the energy model in the i-th iteration process is determined based on multiple target generated data that conform to Gaussian noise and the distribution represented by the energy model in the iteration process before the i-th iteration.

[0431] Optionally, the value of the loss function in the i-th iteration process is determined based on the difference between the energy of multiple second error blocks and the energy of multiple target-generated data that follow the distribution represented by the energy model in the i-th iteration process, the square of the energy of multiple second error blocks, and the square of the energy of multiple target-generated data that follow the distribution represented by the energy model in the i-th iteration process.

[0432] It should be noted that the aforementioned devices 3000 and 4000 are embodied in the form of functional units. The term "unit" here can be implemented in software and / or hardware, without specific limitations.

[0433] For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0434] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0435] Figure 13 This is a schematic diagram of the hardware structure of the image processing device provided in the embodiments of this application. Figure 11 The image processing device 5000 shown (specifically, the device 5000 can be a computer device) includes a memory 5001, a processor 5002, a communication interface 5003, and a bus 5004. The memory 5001, processor 5002, and communication interface 5003 are interconnected via the bus 5004.

[0436] The memory 5001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 5001 may store a program, and when the program stored in the memory 5001 is executed by the processor 5002, the processor 5002 performs the various steps of the image processing method of this embodiment. Specifically, the processor 5002 may execute method 400 described above.

[0437] The processor 5002 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the image processing method of the embodiments of this application.

[0438] The processor 5002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the image processing method of this application can be completed through the integrated logic circuitry in the processor 5002 or through software instructions.

[0439] The processor 5002 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 5001, and the processor 5002 reads the information in memory 5001 and combines it with its hardware to complete the task. Figure 14 The apparatus shown includes units that are required to perform functions, or to perform the image processing method of the method embodiments of this application.

[0440] The communication interface 5003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 5000 and other devices or communication networks. For example, a first image and a second image can be acquired through the communication interface 5003.

[0441] Bus 5004 may include a pathway for transmitting information between various components of device 5000 (e.g., memory 5001, processor 5002, communication interface 5003).

[0442] Figure 14 This is a schematic diagram of the hardware structure of the image processing device provided in the embodiments of this application. Figure 12 The image processing model shown includes a processing device 6000 (which can specifically be a computer device) comprising a memory 6001, a processor 6002, a communication interface 6003, and a bus 6004. The memory 6001, processor 6002, and communication interface 6003 are interconnected via the bus 6004.

[0443] The memory 6001 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 6001 may store a program. When the program stored in the memory 6001 is executed by the processor 6002, the processor 6002 performs the various steps of the image processing method of the embodiment of this application. Specifically, the processor 6002 may execute method 700 described above.

[0444] The processor 6002 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the image processing model processing method of the method embodiment of this application.

[0445] The processor 6002 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the image processing model's processing method can be completed through the integrated logic circuitry in the processor 6002 or through software instructions.

[0446] The processor 6002 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 6001, and the processor 6002 reads the information in memory 6001 and combines it with its hardware to complete the task. Figure 13 The apparatus shown includes units that are required to perform functions, or to perform the processing method of the image processing model of the method embodiment of this application.

[0447] The communication interface 6003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 6000 and other devices or communication networks. For example, a first image and a second image can be acquired through the communication interface 6003.

[0448] Bus 6004 may include a pathway for transmitting information between various components of device 6000 (e.g., memory 6001, processor 6002, communication interface 6003).

[0449] It should be noted that although only the memory, processor, and communication interface are shown in the above-described devices 5000 and 6000, those skilled in the art should understand that in specific implementations, devices 5000 and 6000 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 5000 and 6000 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that devices 5000 and 6000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 14 and ​ All the devices shown.

[0450] This application also provides a computer-readable storage medium storing program code for execution by a device, the program code including a processing method for executing the image processing method or image processing model of this application embodiment.

[0451] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the image processing method or image processing model described in this application.

[0452] This application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface and executes the image processing method or image processing model of this application.

[0453] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the image processing method or image processing model processing method of the embodiments of this application.

[0454] The aforementioned chip can be either an FPGA or an ASIC.

[0455] It should be understood that the processor in the embodiments of this application can be a central processing unit (CPU), but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0456] It should also be understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0457] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0458] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0459] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0460] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0461] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0462] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0463] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0464] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0465] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0466] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0467] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a reconstruction error between a first image and a second image, the first image being a target image corresponding to an initial image, the second image being obtained by processing the initial image; generating a scalar map, the scalar map comprising one or more regions, each region of the one or more regions corresponding to a scalar value, the scalar value being used to indicate a quality of one or more image blocks in the second image, the scalar value being determined based on a difference between a distribution of the reconstruction error and a distribution of noise.

2. The method of claim 1, wherein, The initial image is the first image, and the second image is obtained by compressing the first image.

3. The method of claim 1, wherein, The one or more regions have the same positions in the scalar map as the one or more image blocks in the second image.

4. The method according to any one of claims 1 to 3, characterized in that, The method further comprises: obtaining a target scalar value from the scalar map, the target scalar value being used to indicate a quality of the second image.

5. The method of claim 1, wherein, A scalar value corresponding to a first region of the one or more regions is determined based on a difference between a distribution of reconstruction error and a distribution of noise between a first image block and a second image block, the first image block belonging to the first image, the second image block belonging to the second image, the difference between the distribution of reconstruction error and the distribution of noise between the first image block and the second image block being determined based on a difference between an energy of at least one first error block in the reconstruction error and an energy of at least one noise block in the noise; The energy of the at least one first error block is obtained by inputting the at least one first error block into a first energy model or a second energy model, the energy of the at least one noise block is obtained by inputting the at least one noise block into the first energy model or the second energy model, the first energy model outputting an energy value indicating a probability that data input into the first energy model conforms to the distribution of reconstruction error between the first image block and the second image block, the second energy model outputting an energy value indicating a probability that data input into the second energy model conforms to the distribution of noise.

6. The method of claim 5, wherein, The first energy model is trained using a plurality of second error blocks in reconstruction error between a third image block and a fourth image block corresponding to the third image block as training data.

7. The method of claim 6, wherein, The first energy model is obtained by adjusting parameters of an energy model in a training process to reduce a value of a loss function in the training process, wherein the value of the loss function in an i-th iteration process in the training process is determined based on a difference between an energy sum of the plurality of second error blocks and an energy of a plurality of target generated data conforming to a distribution represented by the energy model in the i-th iteration process, the energy sum of the plurality of second error blocks and the energy of the plurality of target generated data conforming to the distribution represented by the energy model in the i-th iteration process being calculated by the energy model in the i-th iteration process, i being a positive integer.

8. The method of claim 7, wherein, The plurality of target generated data conforming to the distribution represented by the energy model in the i-th iteration process is obtained through multiple iterations of initial generated data of the energy model in the i-th iteration process, wherein the generated data obtained through the k+1-th iteration of the multiple iterations of the initial generated data is determined according to the generated data obtained through the k-th iteration, an energy gradient of the generated data obtained through the k-th iteration, and Gaussian noise, a coefficient of the energy gradient of the generated data obtained through the k-th iteration is greater than a coefficient of the Gaussian noise, and k is a positive integer.

9. The method of claim 8, wherein, The initial generated data of the energy model in the i-th iteration process is determined according to Gaussian noise and the plurality of target generated data conforming to the distribution represented by the energy model in the iteration process before the i-th iteration.

10. The method of claim 7, wherein, The value of the loss function in the i-th iteration process is determined according to a difference between energy of the plurality of second error blocks and energy of the plurality of target generated data conforming to the distribution represented by the energy model in the i-th iteration process, a square of the energy of the plurality of second error blocks, and a square of the energy of the plurality of target generated data conforming to the distribution represented by the energy model in the i-th iteration process.

11. The method according to any one of claims 5 to 10, characterized in that, The scalar value corresponding to the first region includes any one of the following: a value of a first evaluation index, a value of a second evaluation index, or a value of a third evaluation index; The value of the first evaluation index is determined according to a difference between energy of the at least one first error block and energy of the at least one noise block, the energy of the at least one first error block is obtained by inputting the at least one first error block into a first energy model for processing, and the energy of the at least one noise block is obtained by inputting the at least one noise block into the first energy model for processing; The value of the second evaluation index is determined according to a difference between energy of the at least one first error block and energy of the at least one noise block, the energy of the at least one first error block is obtained by inputting the at least one first error block into a second energy model for processing, and the energy of the at least one noise block is obtained by inputting the at least one noise block into the second energy model for processing; The value of the third evaluation index is determined according to the value of the first evaluation index and the value of the second evaluation index.

12. The method of claim 11, wherein, The first evaluation index satisfies the following formula: ； wherein denotes the first evaluation index, denotes the energy of the input data calculated by the first energy model, denotes the at least one first error block, denotes the at least one noise block, denotes the average of the energy of the at least one first error block calculated by the first energy model, denotes the average of the energy of the at least one noise block calculated by the first energy model.

13. The method of claim 11, wherein, The second evaluation index satisfies the following formula: ； wherein denotes the second evaluation index, denotes the energy of the input data calculated by the second energy model, denotes the at least one first error block, denotes the at least one noise block, denotes the average of the energy of the at least one first error block calculated by the second energy model, denotes the average of the energy of the at least one noise block calculated by the second energy model.

14. A processing method of an image processing model, characterized by, The method comprises: obtaining a reconstruction error between a first image and a second image, the second image being obtained by processing an initial image through an image processing model, and the first image being a target image corresponding to the initial image; updating the image processing model based on a difference between a distribution of the reconstruction error between the first image and the second image and a distribution of noise.

15. The method of claim 14, wherein, The image processing model is an image compression model, and the initial image is the first image.

16. The method of claim 14, wherein, The updating of the image processing model based on the difference between the distribution of the reconstruction error and the distribution of noise comprises: adjusting the reconstruction error between the first image and the second image to obtain an adjusted reconstruction error, the first image block belonging to the first image, the second image block belonging to the second image, the adjusting being targeted at increasing a difference between a distribution of the reconstruction error between the first image block and the second image block and a distribution of the noise; training the image processing model based on the adjusted reconstruction error.

17. The method of claim 16, wherein, The adjusting the reconstruction error between the first image and the second image to obtain an adjusted reconstruction error, the first image block belonging to the first image, the second image block belonging to the second image, the adjusting being targeted at increasing a difference between a distribution of the reconstruction error between the first image block and the second image block and a distribution of the noise, comprises: adjusting the reconstruction error between the first image and the second image to obtain an adjusted reconstruction error, the adjusting being targeted at reducing an energy between the first image block and the second image block, the energy between the first image block and the second image block being calculated by a first energy model, an energy value output by the first energy model being used to indicate a probability that data input into the first energy model is subject to a distribution of the reconstruction error between the first image block and the second image block.

18. The method of claim 17, wherein, A first error block in the adjusted reconstruction error satisfies the following formula: ； wherein denotes a first error block of the reconstruction error between the first and second image blocks, denotes a first error block of the adjusted reconstruction error, denotes a step size of the energy reduction, denotes an energy of the input data calculated by the first energy model.

19. The method of claim 17, wherein, The first energy model is trained by taking a plurality of second error blocks in reconstruction error between a third image block and a fourth image block corresponding to the third image block as training data.

20. The method of claim 19, wherein, The first energy model is obtained by adjusting a parameter of an energy model in a training process to reduce a value of a loss function in the training process, wherein the value of the loss function in an i th iteration process in the training process is determined according to a difference between an energy sum of the plurality of second error blocks and energies of a plurality of target generated data subject to a distribution represented by the energy model in the i th iteration process, the energy sum of the plurality of second error blocks and the energies of the plurality of target generated data subject to the distribution represented by the energy model in the i th iteration process being calculated by the energy model in the i th iteration process, i being a positive integer.

21. The method of claim 20, wherein, The plurality of target generated data subject to the distribution represented by the energy model in the i th iteration process is obtained by adjusting initial generated data of the energy model in the i th iteration process a plurality of times, wherein generated data obtained by a k+1 th iteration of the initial generated data is determined according to generated data obtained by a k th iteration, an energy gradient of the generated data obtained by the k th iteration, and Gaussian noise, a coefficient of the energy gradient of the generated data obtained by the k th iteration being greater than a coefficient of the Gaussian noise, k being a positive integer.

22. The method of claim 21, wherein, The initial generated data of the energy model in the i th iteration process is determined according to Gaussian noise and a plurality of target generated data subject to a distribution represented by an energy model in an iteration process before the i th iteration.

23. The method of any one of claims 20-22, wherein, A value of the loss function in the ith iteration process is determined according to a difference between energies of the plurality of target generated data subject to the distribution represented by the energy model in the ith iteration process, a square of the energies of the plurality of second error blocks, and a square of the energies of the plurality of target generated data subject to the distribution represented by the energy model in the ith iteration process.

24. An image processing apparatus characterized by comprising: Comprise: An acquisition unit, configured to acquire a reconstruction error between a first image and a second image, the first image being a target image corresponding to an initial image, and the second image being obtained by processing the initial image; A processing unit, configured to generate a scalar map, the scalar map comprising one or more regions, each region of the one or more regions corresponding to a scalar value used to indicate a quality of one or more image blocks in the second image, the scalar value being determined based on a difference between a distribution of the reconstruction error and a distribution of noise.

25. The apparatus of claim 24, wherein, The initial image is the first image, and the second image is obtained by compressing the first image.

26. The apparatus of claim 25, wherein, The one or more regions have the same positions in the scalar map as positions of the one or more image blocks in the second image.

27. The apparatus of claim 25 or 26, wherein, The processing unit is further configured to: Obtain a target scalar value from the scalar map, the target scalar value being used to indicate a quality of the second image.

28. The apparatus of claim 24, wherein, A scalar value corresponding to a first region of the one or more regions is determined based on a difference between a distribution of reconstruction error between a first image block and a second image block and the distribution of noise, the first image block belonging to the first image, the second image block belonging to the second image, and the difference between the distribution of reconstruction error between the first image block and the second image block and the distribution of noise being determined according to a difference between an energy of at least one first error block in the reconstruction error between the first image block and the second image block and an energy of at least one noise block in the noise; The energy of the at least one first error block is obtained by inputting the at least one first error block into a first energy model or a second energy model, the energy of the at least one noise block is obtained by inputting the at least one noise block into the first energy model or the second energy model, an energy value output by the first energy model is used to indicate a probability that data input into the first energy model is subject to the distribution of reconstruction error between the first image block and the second image block, and an energy value output by the second energy model is used to indicate a probability that data input into the second energy model is subject to the distribution of noise.

29. The apparatus of claim 28, wherein, The first energy model is trained using a plurality of second error blocks in reconstruction error between a third image block and a fourth image block corresponding to the third image block as training data.

30. The apparatus of claim 29, wherein, The first energy model is obtained by adjusting parameters of the energy model in the training process to reduce a value of a loss function in the training process, wherein the value of the loss function in an i th iteration process in the training process is determined according to a difference between an energy sum of the plurality of second error blocks and energies of the plurality of target generated data subject to a distribution represented by the energy model in the i th iteration process, the energy sum of the plurality of second error blocks and the energies of the plurality of target generated data subject to the distribution represented by the energy model in the i th iteration process are calculated by the energy model in the i th iteration process, and i is a positive integer.

31. The apparatus of claim 30, wherein, The plurality of target generated data subject to the distribution represented by the energy model in the i th iteration process is obtained by iteratively adjusting initial generated data of the energy model in the i th iteration process, wherein generated data obtained in a k+1 th iteration of the plurality of iterations of the initial generated data is determined according to generated data obtained in a k th iteration, an energy gradient of the generated data obtained in the k th iteration, and Gaussian noise, a coefficient of the energy gradient of the generated data obtained in the k th iteration is greater than a coefficient of the Gaussian noise, and k is a positive integer.

32. The apparatus of claim 31, wherein, The initial generated data of the energy model in the i th iteration process is determined according to Gaussian noise and the plurality of target generated data subject to a distribution represented by an energy model in an iteration process before the i th iteration.

33. The apparatus of claim 30, wherein, The value of the loss function in the i th iteration process is determined according to a difference between the energy sum of the plurality of second error blocks and the energies of the plurality of target generated data subject to the distribution represented by the energy model in the i th iteration process, a square of the energy sum of the plurality of second error blocks, and a square of the energies of the plurality of target generated data subject to the distribution represented by the energy model in the i th iteration process.

34. The apparatus of claim 28, wherein, The scalar value corresponding to the first region includes any one of a value of a first evaluation index, a value of a second evaluation index, or a value of a third evaluation index; The value of the first evaluation index is determined according to a difference between an energy of the at least one first error block and an energy of the at least one noise block, the energy of the at least one first error block is obtained by inputting the at least one first error block into a first energy model for processing, and the energy of the at least one noise block is obtained by inputting the at least one noise block into the first energy model for processing; The value of the second evaluation index is determined according to a difference between an energy of the at least one first error block and an energy of the at least one noise block, the energy of the at least one first error block is obtained by inputting the at least one first error block into a second energy model for processing, and the energy of the at least one noise block is obtained by inputting the at least one noise block into the second energy model for processing; The value of the third evaluation index is determined according to the value of the first evaluation index and the value of the second evaluation index.

35. The apparatus of claim 34, wherein, The first evaluation index satisfies the following formula: ； wherein denotes the first evaluation index, denotes the energy of the input data calculated by the first energy model, denotes the at least one first error block, denotes the at least one noise block, denotes the average of the energy of the at least one first error block calculated by the first energy model, denotes the average of the energy of the at least one noise block calculated by the first energy model.

36. The apparatus of claim 34 or 35, wherein, The second evaluation index satisfies the following formula: ； wherein denotes the second evaluation index, denotes the energy of the input data calculated by the second energy model, denotes the at least one first error block, denotes the at least one noise block, denotes the average of the energy of the at least one first error block calculated by the second energy model, denotes the average of the energy of the at least one noise block calculated by the second energy model.

37. A processing device of an image processing model, characterized in that, The method comprises: An acquisition unit is configured to acquire a reconstruction error between a first image and a second image, the second image being obtained by processing an initial image through an image processing model, and the first image being a target image corresponding to the initial image; A processing unit is configured to update the image processing model based on a difference between a distribution of the reconstruction error between the first image and the second image and a distribution of noise.

38. The device of claim 37, wherein, The image processing model is an image compression model, and the initial image is the first image.

39. The device of claim 37, wherein, The processing unit is specifically configured to: adjust the reconstruction error between the first image and the second image to obtain an adjusted reconstruction error, in order to increase a difference between a distribution of a reconstruction error between a first image block and a second image block and a distribution of the noise, the first image block belonging to the first image, and the second image block belonging to the second image; train the image processing model based on the adjusted reconstruction error.

40. The device of claim 39, wherein, The processing unit is specifically configured to: adjust the reconstruction error between the first image and the second image to obtain an adjusted reconstruction error, in order to reduce an energy between the first image block and the second image block, the energy between the first image block and the second image block being calculated through a first energy model, and an energy value output by the first energy model being used to indicate a probability that data input into the first energy model is subject to a distribution of the reconstruction error between the first image block and the second image block.

41. The device of claim 40, wherein, A first error block in the adjusted reconstruction error satisfies the following formula: ； wherein denotes a first error block of the reconstruction error between the first and second image blocks, denotes a first error block of the adjusted reconstruction error, denotes a step size of the energy reduction, denotes an energy of the input data calculated by the first energy model.

42. The device of claim 40, wherein, The first energy model is trained based on a plurality of second error blocks in a reconstruction error between a third image block and a fourth image block corresponding to the third image block.

43. The device of claim 42, wherein, The first energy model is obtained by adjusting a parameter of an energy model in a training process in order to reduce a value of a loss function in the training process, a value of the loss function in an i-th iteration process in the training process being determined according to a difference between an energy sum of the plurality of second error blocks and energies of a plurality of target generated data subject to a distribution represented by an energy model in the i-th iteration process, the energies of the plurality of target generated data subject to the distribution represented by the energy model in the i-th iteration process being calculated through the energy model in the i-th iteration process, i being a positive integer.

44. The device of claim 43, wherein, The plurality of target generated data subject to the distribution represented by the energy model in the i-th iteration process is obtained by adjusting initial generated data of the energy model in the i-th iteration process a plurality of times, generated data obtained in a k+1-th iteration of the initial generated data being determined according to generated data obtained in a k-th iteration, an energy gradient of the generated data obtained in the k-th iteration, and Gaussian noise, a coefficient of the energy gradient of the generated data obtained in the k-th iteration being greater than a coefficient of the Gaussian noise, k being a positive integer.

45. The device of claim 44, wherein, The initial generated data of the energy model in the i-th iteration process is determined according to a plurality of target generated data subject to a distribution represented by the energy model in the iteration process before the i-th iteration and Gaussian noise.

46. The apparatus of any one of claims 43-45, wherein, The value of the loss function in the i-th iteration process is determined according to a difference between energies of the plurality of second error blocks and energies of a plurality of target generated data subject to a distribution represented by the energy model in the i-th iteration process, squares of the energies of the plurality of second error blocks, and squares of the energies of the plurality of target generated data subject to the distribution represented by the energy model in the i-th iteration process.

47. An image processing apparatus characterized by comprising: A processor and a memory, the memory being configured to store program instructions, the processor being configured to invoke the program instructions to perform the method of any one of claims 1-13.

48. A processing device of an image processing model, the processing device comprising: A processor and a memory, the memory being configured to store program instructions, the processor being configured to invoke the program instructions to perform the method of any one of claims 14-23.

49. A computer-readable storage medium, characterized in that, The computer-readable storage medium is configured to store program code for execution by a device, the program code comprising instructions for performing the method of any one of claims 1-13 or 14-23.

50. A computer program product comprising instructions, wherein: The computer program product, when running on a computer, causes the computer to perform the method of any one of claims 1-13 or 14-23.

51. A chip, comprising: The chip comprises a processor and a data interface, the processor reading instructions stored on a memory through the data interface to perform the method of any one of claims 1-13 or 14-23.

Images