Training method and device of image inpainting model, image inpainting method and device

Abstract

Embodiments of the present application provide a kind of training method and device of image restoration model, image restoration method and device, belong to artificial intelligence technical field.The method comprises: obtaining the sample image pair including first sample image and second sample image, respectively to first, second sample image is carried out image standardization processing and obtains first, second standard image;First standard image is input to the preset original image processing model;By upsampling submodel, first standard image is carried out upsampling processing, and preliminary image is obtained;The image resolution of preliminary image is higher than the image resolution of second sample image;By diffusion denoising model, preliminary image is carried out diffusion denoising processing, and intermediate image is obtained;According to second sample image, preliminary image, intermediate image, calculate target loss data;According to target loss data, the parameter adjustment of original image processing model is carried out, and the target image restoration model obtained can improve the precision, efficiency and accuracy of image restoration.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a training method and device of an image restoration model and an image restoration method and device. BACKGROUND

[0002] High-resolution images have high pixel density and can provide more information for image content, which has important practical significance in real life. For example, many old photo images are distorted and have relatively low clarity, and old photos are the carrier of many people's memories. If the quality of old photos can be enhanced, it has high practical significance and emotional value. Current algorithms for image super-resolution include interpolation-based methods, and current algorithms have achieved certain results on some data sets, but still have some defects: among them, interpolation-based methods (such as cubic interpolation, etc.), although fast and simple model, but there are defects in the repair precision. SUMMARY

[0003] The main purpose of the embodiments of the present application is to provide a training method and device of an image restoration model and an image restoration method and device. The trained model can improve the precision, efficiency and accuracy of image restoration.

[0004] To achieve the above purpose, the first aspect of the embodiments of the present application provides a training method of an image restoration model, which comprises:

[0005] Obtain the original image data of the target to be detected;

[0006] Obtain a sample image pair; wherein the sample image pair comprises a first sample image and a second sample image, and the image resolution of the first sample image is lower than that of the second sample image;

[0007] Perform image standardization processing on the first sample image to obtain a first standard image, and perform image standardization processing on the second sample image to obtain a second standard image;

[0008] Input the first standard image into a preset original image processing model; wherein the original image processing model comprises an upsampling sub-model and a diffusion denoising sub-model;

[0009] Perform upsampling processing on the first standard image through the upsampling sub-model to obtain a preliminary image; wherein the image resolution of the preliminary image is higher than that of the second sample image;

[0010] Perform diffusion denoising processing on the preliminary image through the diffusion denoising sub-model to obtain an intermediate image;

[0011] According to the second sample image, the preliminary image, and the intermediate image, target loss data is calculated;

[0012] According to the target loss data, parameter adjustment is performed on the original image processing model, and a target image restoration model is obtained.

[0013] In some embodiments, the diffusion denoising of the preliminary image by the diffusion denoising sub-model obtains an intermediate image, including:

[0014] The preliminary image is diffused by the diffusion denoising sub-model to obtain a noise image containing noise;

[0015] The noise image is denoised by the diffusion denoising sub-model to obtain the intermediate image.

[0016] In some embodiments, the target loss data includes super-resolution loss data, and the calculation of the target loss data according to the second sample image, the preliminary image, and the intermediate image includes:

[0017] According to the second sample image, the preliminary image, and the intermediate image, target loss data is calculated, including:

[0018] According to the second sample image and the preliminary image, first loss data is calculated;

[0019] According to the second sample image and the intermediate image, second loss data is calculated;

[0020] According to the first loss data and the second loss data, weighted calculation is performed to obtain the super-resolution loss data.

[0021] In some embodiments, the target loss data further includes spatial similarity loss data, and the calculation of the target loss data according to the second sample image, the preliminary image, and the intermediate image further includes:

[0022] According to the second sample image, the preliminary image, and the intermediate image, target loss data is calculated, including:

[0023] Image spatial feature extraction is performed on the noise image to obtain a noise spatial feature vector;

[0024] The spatial distance between the noise spatial feature vector and a preset reference noise feature vector is calculated to obtain image spatial similarity data;

[0025] According to the spatial similarity data, the spatial similarity loss data is obtained.

[0026] In some embodiments, the image standardization processing of the first sample image obtains a first standard image, and the image standardization processing of the second sample image obtains a second standard image, including:

[0027] A first pixel point of the first sample image is obtained, and a second pixel point of the second sample image is obtained.

[0028] The first pixel point is standardized to obtain the first standard image, and the second pixel point is standardized to obtain the second standard image; wherein the pixel point of the first standard image is in the interval [-1, 1], and the pixel point of the second standard image is in the interval [-1, 1].

[0029] To achieve the above object, a second aspect of the embodiments of the present application proposes an image repairing method, which includes:

[0030] Obtaining a target image to be repaired;

[0031] Inputting the target image into a target image repairing model; wherein the target image repairing model is trained according to the training method of the first aspect;

[0032] Repairing the target image by the target image repairing model to obtain a current image; wherein the image resolution of the current image is higher than that of the target image.

[0033] To achieve the above object, a third aspect of the embodiments of the present application proposes a training device of an image repairing model, which includes:

[0034] A sample image acquisition module is configured to acquire a sample image pair; wherein the sample image pair includes a first sample image and a second sample image, and the image resolution of the first sample image is lower than that of the second sample image;

[0035] An image standardization module is configured to perform image standardization processing on the first sample image to obtain a first standard image, and perform image standardization processing on the second sample image to obtain a second standard image;

[0036] An image input module is configured to input the first standard image into a preset original image processing model; wherein the original image processing model includes an upsampling sub-model and a diffusion denoising sub-model;

[0037] An upsampling module is configured to perform upsampling processing on the first standard image by the upsampling sub-model to obtain a preliminary image; wherein the image resolution of the preliminary image is higher than that of the second sample image;

[0038] a diffusion denoising module configured to perform diffusion denoising processing on the preliminary image by the diffusion denoising sub-model to obtain an intermediate image;

[0039] a loss calculation module configured to calculate target loss data according to the second sample image, the preliminary image, and the intermediate image;

[0040] a parameter adjustment module configured to perform parameter adjustment on the original image processing model according to the target loss data to obtain a target image inpainting model.

[0041] To achieve the above object, a fourth aspect of the embodiment of the present application provides an image inpainting device, which comprises:

[0042] a target image acquisition module configured to acquire a target image to be inpainted;

[0043] a target image input module configured to input the target image to a target image inpainting model, wherein the target image inpainting model is trained according to the training device of the third aspect described above;

[0044] an image inpainting module configured to perform inpainting processing on the target image by the target image inpainting model to obtain a current image, wherein the image resolution of the current image is higher than that of the target image.

[0045] To achieve the above object, a fifth aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of the first aspect described above or the method of the second aspect described above when executing the computer program.

[0046] To achieve the above object, a sixth aspect of the embodiment of the present application provides a storage medium, which is a computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the method of the first aspect described above or the method of the second aspect described above.

[0047] The training method of the image restoration model, the training device of the image restoration model, the image restoration method, the image restoration device, the electronic equipment and the storage medium provided by the present application obtain a sample image pair constructed by a first sample image and a second sample image, the image resolution of the first sample image in the sample image pair is lower than the image resolution of the second sample image, and the first sample image is subjected to image standardization processing to obtain a first standard image, the second sample image is subjected to image standardization processing to obtain a second standard image, the first standard image is input into a preset original image processing model, the first standard image is subjected to up-sampling processing by an up-sampling sub-model of the original image processing model, a preliminary image is obtained, the image resolution of the preliminary image is higher than the image resolution of the second sample image, the preliminary image is subjected to diffusion denoising processing by a diffusion denoising sub-model of the original image processing model to obtain an intermediate image, target loss data is calculated according to the second sample image, the preliminary image and the intermediate image, and the original image processing model is subjected to parameter adjustment according to the target loss data, so that the obtained target image restoration model can improve the precision, efficiency and accuracy of image restoration. BRIEF DESCRIPTION OF DRAWINGS

[0048] Figure 1 is a flowchart of the training method of the image restoration model provided by the embodiments of the present application;

[0049] Figure 2 is a flowchart of step 102 in Figure 1

[0050] Figure 3 is a flowchart of step 105 in Figure 1

[0051] Figure 4 is a flowchart of step 106 in Figure 1

[0052] Figure 5 is a flowchart of step 106 in another embodiment Figure 1

[0053] Figure 6 is a flowchart of the image restoration method provided by the embodiments of the present application;

[0054] Figure 7 is a structural schematic diagram of the training device of the image restoration model provided by the embodiments of the present application;

[0055] Figure 8 is a structural schematic diagram of the image restoration device provided by the embodiments of the present application;

[0056] Figure 9 is a hardware structural schematic diagram of the electronic equipment provided by the embodiments of the present application. DETAILED DESCRIPTION​​​​

[0057] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not intended to limit the present application.

[0058] It should be noted that although the functional modules are divided in the device schematic diagram, and the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in a manner different from the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0060] First, the terms involved in the present application are analyzed:

[0061] Artificial intelligence (AI): is a new technical science to study, develop the theory, method, technology and application system for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, artificial intelligence attempts to understand the essence of intelligence, and produces a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems, etc. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computer or digital computer controlled machine to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0062] Natural language processing (NLP): NLP uses computers to process, understand and use human languages (such as Chinese, English, etc.). NLP is a branch of artificial intelligence and is an interdisciplinary subject of computer science and linguistics, also commonly known as computational linguistics. Natural language processing includes syntax analysis, semantic analysis, discourse understanding, etc. Natural language processing is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining, etc. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research related to language processing, and linguistic research related to language computing.

[0063] High-resolution image: In a high-resolution image, the pixel density is high, which can provide more information for the image content. These information has important practical significance in real life. For example, many old photos are distorted and have low clarity, and old photos are the carrier of many people's memories. If the quality of the old photos can be enhanced, it has high practical significance and emotional value. The image super-resolution method (Super-Resolution Image Reconstruction) can infer the possible picture according to the residual information of the low-resolution image, realize image reconstruction, and reconstruct the low-resolution image into a high-resolution image, which can greatly improve the clarity. At present, the mainstream algorithms of image super-resolution mainly include three kinds: interpolation-based method, reconstruction-based method and learning-based method. The above mainstream algorithms of image super-resolution have achieved certain effect on part of the data set, but still have some defects: among them, the interpolation-based method (such as cubic interpolation, etc.), although it is fast and the model is simple, but it has the defect of repair precision; and the reconstruction-based method (such as gradient profile prior method, etc.), in some scenes, it can generate relatively real details, but with the increase of the scale factor, the performance of many reconstruction-based methods decreases sharply, and the requirement for computer computing power is very high; in addition, the learning-based method generally adopts an adversarial training method, and the training stability is very challenging and difficult to converge.

[0064] Therefore, the embodiments of the present application provide a training method of an image restoration model, an image restoration method, a training device of an image restoration model, an image restoration device, an electronic device and a storage medium, which aims to improve the precision, efficiency and accuracy of image restoration.

[0065] The training method of the image restoration model, the image restoration method, the image restoration method, the image restoration device, the electronic device and the storage medium provided by the embodiments of the present application are specifically explained by the following embodiments. First, the image restoration method in the embodiments of the present application is described.

[0066] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. The artificial intelligence (AI) is a theory, method, technology and application system for simulating, extending and expanding human intelligence by using a digital computer or a machine controlled by a digital computer, perceiving an environment, acquiring knowledge and using the knowledge to obtain optimal results.

[0067] The artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. The artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric identification technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0068] The image repairing method provided by the embodiments of the present application relates to the field of artificial intelligence technology. The training method of the image repairing model or the image repairing method provided by the embodiments of the present application can be applied in a terminal, can also be applied in a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, etc.; the server end can be configured as an independent physical server, can also be configured as a server cluster or a distributed system composed of multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN, and big data and artificial intelligence platform; and the software can be an application for implementing the training method of the image repairing model or the image repairing method, etc., but is not limited to the above forms.

[0069] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0070] It should be noted that in various specific embodiments of the present application, when relevant processing needs to be performed on data related to the identity or characteristics of the user, such as user information, user behavior data, user image data, user history data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of such data will comply with relevant laws, regulations, and standards. In addition, when the embodiments of the present application need to obtain sensitive personal information of the user, the user's separate permission or separate consent will be obtained through a pop-up window or a jump to a confirmation page, and after obtaining the user's separate permission or separate consent, the necessary user-related data for the normal operation of the embodiments of the present application will be obtained.

[0071] Figure 1 is an optional flowchart of a training method of an image restoration model provided by the embodiments of the present application, Figure 1 The method in the method can include but is not limited to steps 101-107.

[0072] Step 101, obtaining a sample image pair; wherein the sample image pair includes a first sample image and a second sample image, and the image resolution of the first sample image is lower than that of the second sample image;

[0073] Step 102, performing image standardization processing on the first sample image to obtain a first standard image, and performing image standardization processing on the second sample image to obtain a second standard image;

[0074] Step 103, inputting the first standard image into a preset original image processing model; wherein the original image processing model includes an upsampling sub-model and a diffusion denoising sub-model;

[0075] Step 104, performing upsampling processing on the first standard image through the upsampling sub-model to obtain a preliminary image; wherein the image resolution of the preliminary image is higher than that of the second sample image;

[0076] Step 105, performing diffusion denoising processing on the preliminary image through the diffusion denoising sub-model to obtain an intermediate image;

[0077] Step 106, calculating target loss data according to the second sample image, the preliminary image, and the intermediate image;

[0078] Step 107, adjusting the parameters of the original image processing model according to the target loss data to obtain a target image restoration model.

[0079] The steps 101 to 107 shown in the embodiments of the present application obtain a sample image pair constructed by a first sample image and a second sample image, the image resolution of the first sample image in the sample image pair is lower than that of the second sample image, perform image standardization processing on the first sample image to obtain a first standard image, perform image standardization processing on the second sample image to obtain a second standard image, input the first standard image into a preset original image processing model, perform up-sampling processing on the first standard image by using an up-sampling sub-model of the original image processing model to obtain a preliminary image, the image resolution of the preliminary image is higher than that of the second sample image, perform diffusion denoising processing on the preliminary image by using a diffusion denoising sub-model of the original image processing model to obtain an intermediate image, calculate target loss data according to the second sample image, the preliminary image and the intermediate image, and adjust parameters of the original image processing model according to the target loss data to obtain a target image restoration model, which can improve the precision, efficiency and accuracy of image restoration.

[0080] In step 101 of some embodiments, the first sample image is a low-resolution image, and the second sample image is a high-resolution image. The image resolution of the first sample image is lower than that of the second sample image. Each sample image pair used for model training is composed of a first sample image and a second sample image, that is, the training data in the present application is composed of a pair of low-resolution images and high-resolution images.

[0081] In addition, in the application scenario of the present application, the sample image pair is an image pair including a person, and each sample image pair includes a low-resolution first sample image and a high-resolution second sample image. The target image restoration model of the present application can be used for quality restoration of old photos with image distortion and blurred clarity.

[0082] In another embodiment of the present application, the sample image pair can represent a sample image data set, and includes two paired image sets: a low-resolution image set and a high-resolution image set ; wherein the low-resolution image set includes a plurality of first sample images, and the high-resolution image set includes a plurality of second sample images.

[0083] Referring to Figure 2 In step 102 of some embodiments, the image standardization processing on the first sample image to obtain a first standard image and the image standardization processing on the second sample image to obtain a second standard image can include, but are not limited to, steps 201 to 202:

[0084] Step 201, obtaining a first pixel point by acquiring a pixel point of the first sample image, and obtaining a second pixel point by acquiring a pixel point of the second sample image;

[0085] Step 202, obtaining a first standard image by performing pixel point standardization on the first pixel point, and obtaining a second standard image by performing pixel point standardization on the second pixel point; wherein the pixel point of the first standard image is in the interval [-1, 1], and the pixel point of the second standard image is in the interval [-1, 1].

[0086] In the embodiments of the present application, no excessive preprocessing is required for the sample image pair, only image standardization processing is required for the first sample image and the second sample image in the sample image pair, and compared with the traditional image super-resolution method, the time cost and labor cost consumed by the preprocessing can be omitted, and the efficiency of the image super-resolution method is improved. The pixel points of the first standard image and the second standard image obtained through image standardization processing are in the interval [-1, 1].

[0087] In step 201 of a specific application scenario, the principle of pixel point standardization is that the first pixel point value is divided by 255, multiplied by 2, and then subtracted by 1, so that the pixel point of the first standard image is in the interval [-1, 1]; similarly, the second pixel point value is divided by 255, multiplied by 2, and then subtracted by 1, so that the pixel point of the second standard image is in the interval [-1, 1].

[0088] In step 103 of some embodiments, the original image processing model includes an upsampling sub-model and a diffusion denoising sub-model. The upsampling sub-model can be used to perform upsampling processing on the input image to expand the image dimension, and can convert the input low-resolution image into a high-resolution preliminary image, and the preliminary image is a super-resolution image, and the image resolution of the preliminary image is higher than that of the first sample image. The diffusion denoising sub-model can be used to perform noise adding processing on the high-resolution preliminary image obtained through the upsampling processing of the upsampling sub-model, and perform denoising processing after the noise adding processing. The specific steps of the upsampling processing based on the upsampling sub-model can refer to step 104. The specific steps of the noise adding processing and the denoising processing based on the diffusion denoising sub-model can refer to step 105.

[0089] In step 104 of some embodiments, the first standard image can be upsampling processed based on the upsampling sub-model, so as to expand the image dimension of the first standard image, and the image resolution of the obtained preliminary image is higher than that of the first standard image. In the embodiments of the present application, the preliminary image obtained through the upsampling processing is a super-resolution image, and the image resolution of the preliminary image is higher than that of the second sample image. In order to facilitate the description of the technical solutions of the present application, the preliminary image is denoted as , the first sample image in the sample image pair is denoted as , the second sample image in the sample image pair is denoted as Step 102 is image standardization processing on the first sample image, and the image resolution of the first sample image is not changed, so the first standard image obtained by image standardization processing on the first sample image is also denoted as .

[0090] In addition, the preliminary image obtained by upsampling processing on the first standard image based on the upsampling sub-model can also be denoted as , wherein f represents upsampling processing.

[0091] Please refer to Figure 3 In step 105 of some embodiments, the preliminary image is subjected to diffusion denoising processing by the diffusion denoising sub-model to obtain an intermediate image, which can include but is not limited to steps 301 to 302:

[0092] Step 301, the preliminary image is subjected to diffusion processing by the diffusion denoising sub-model to obtain a noise image containing noise;

[0093] Step 302, the noise image is subjected to denoising processing by the diffusion denoising sub-model to obtain an intermediate image.

[0094] In step 301 of some embodiments, the preliminary image is subjected to diffusion processing based on the diffusion denoising sub-model, that is, the preliminary image is subjected to noise adding processing, noise is added to the preliminary image, and the obtained noise image contains noise.

[0095] In an application scenario, diffusion processing is denoted as q, and denoising processing is denoted as p. The preliminary image is subjected to the first step of diffusion processing and is denoted as: to obtain , and is subjected to the second step of diffusion processing and is denoted as: to obtain , and so on. In this application, the preliminary image is subjected to T steps of diffusion processing, and the T steps of diffusion processing are diffusion processes with Markov property to obtain , obeys a Gaussian distribution with mean 0 and variance unit matrix . Wherein, is a unit matrix, is a constant, and is usually a geometric sequence with a value between 0 and 1, . The process of denoising processing is: sampling from the distribution of , and is subjected to the first step of denoising processing and is denoted as: to obtain , and is subjected to the second step of denoising processing and is denoted as: obtained , and so on, a total of T steps of denoising processing, the T steps of denoising processing is a denoising process with Markov property , and finally obtaining the intermediate image corresponding to the original image, the intermediate image is the super-resolution image.

[0096] In step 302 of some embodiments, the noise image is denoised based on the diffusion denoising model, and the obtained intermediate image can be represented as .

[0097] In the embodiments of the present application, the up-sampling processing is performed through step 104, and the diffusion processing and denoising processing are performed through step 301, so as to realize the mapping from the low-resolution image space to the super-resolution image space.

[0098] Please refer to Figure 4 In step 106 of some embodiments, the target loss data includes super-resolution loss data, and the target loss data is calculated according to the second sample image, the preliminary image and the intermediate image, which can include but is not limited to steps 401 to 403:

[0099] The target loss data is calculated according to the second sample image, the preliminary image and the intermediate image, including:

[0100] Step 401, calculating first loss data according to the second sample image and the preliminary image;

[0101] Step 402, calculating second loss data according to the second sample image and the intermediate image;

[0102] Step 403, performing weighted calculation according to the first loss data and the second loss data to obtain super-resolution loss data.

[0103] In step 401 of some embodiments, the first loss data is represented as , the first loss data = , wherein the second sample image is , and the preliminary image is ).

[0104] In step 402 of some embodiments, the second loss data is represented as , the first loss data = , wherein the second sample image is , and the intermediate image can be represented as .

[0105] In step 403 of some embodiments, the first loss data and the second loss data are added to obtain the super-resolution loss data + The super-resolution loss data can be calculated according to the following formula (1):

[0106] (1)

[0107] The second sample image is The preliminary image is The intermediate image can be represented as .

[0108] The super-resolution loss data obtained through steps 401-403 can constrain the target image restoration model, ensuring that the target image restoration model captures and describes the feature information of high-resolution person image data, improves the up-sampling capability and the generation capability of the Denoising Diffusion Probabilistic Model (DDPM), and thus can achieve the task of image super-resolution processing with high precision.

[0109] Referring to Figure 5 In step 106 of some other embodiments, the target loss data further includes spatial similarity loss data, and the target loss data is calculated according to the second sample image, the preliminary image, and the intermediate image, which can further include but is not limited to steps 501-503:

[0110] Step 501: performing image spatial feature extraction on the noise image to obtain a noise spatial feature vector;

[0111] Step 502: calculating the spatial distance between the noise spatial feature vector and a preset reference noise feature vector to obtain image spatial similarity data;

[0112] Step 503: obtaining spatial similarity loss data according to the spatial similarity data.

[0113] In some embodiments, the spatial similarity loss data is represented as The spatial similarity loss data is used to represent the gap between the noise distribution in the image and the standard normal distribution.

[0114] In step 501 of some embodiments, the noise image is obtained by performing diffusion processing on the preliminary image in step 301. Step 501 is used to perform image spatial feature extraction on the noise image, so as to obtain the noise spatial feature vector of the noise image in space.

[0115] In step 502 of some embodiments, the preset reference noise feature vector is a noise vector of the standard normal distribution. The image spatial similarity data represents the spatial distance between a noise spatial feature vector and a reference noise feature vector.

[0116] In step 503 of some embodiments, the spatial similarity loss data is represented as , the spatial similarity loss data = , specifically, the calculation of the image spatial similarity data is performed from t = 1 to t = T, so that the spatial similarity loss data of , can be represented as shown in the following formula (2):

[0117] (2)

[0118] wherein is a result of sampling from a standard normal distribution, that is, a preset reference noise feature vector, is a noise adding part in the diffusion denoising model, that is, a noise adding process corresponding to the noise adding part, that is, a noise spatial feature vector, please refer to the following formula (3) shown as :

[0119] = α1α2α 3... α t = (1-β1) (1-β2) (1-β3)... (1-β t ) (3)

[0120] wherein β t is a constant, usually taking a geometric sequence between 0 and 1, . The above spatial similarity loss data represents the distance between the reference noise feature vector of the standard normal distribution and the noise spatial feature vector, and the loss of the spatial similarity loss data is getting smaller and smaller, which means that the final distribution of the diffusion process is slowly approaching or even overlapping with the standard normal distribution.

[0121] Based on the super-resolution loss data obtained in steps 401 to 403, and based on the spatial similarity loss data obtained in steps 501 to 503 , a weighted calculation is performed to obtain target loss data, specifically, the target loss data is shown in the following formula (4):

[0122] (4)

[0123] wherein can refer to the aforementioned formula (1) shown; is a weight coefficient, which can be designed according to actual conditions, and .

[0124] In the embodiments of the present application, the upsampling processing is performed through step 104, the diffusion processing is performed through step 301, the denoising processing is performed through step 301 again, the super-resolution loss constraint is performed based on steps 401 to 403, and the spatial similarity loss constraint is performed based on steps 501 to 503, so as to realize the mapping from the low-resolution image space to the super-resolution image space.

[0125] In addition, in an application scenario, the super-resolution loss data and the spatial similarity loss data are simultaneously minimized The model parameters can be updated by using a stochastic gradient descent (SGD) method. Meanwhile, the early stopping method is used to control the model training progress. When the validation error of the model continuously increases in more than k iterations, the training is stopped, and the final target image restoration model is obtained, wherein the value of k can be set according to actual conditions. The early stopping method is a widely used method, which is better than the regularization method in many cases.

[0126] In the embodiments of the present application, compared with the traditional Denoising Diffusion Probabilistic Model (DDPM) method, not only the sampling generation process is needed, but also the training process is needed. In order to realize the function of accurate image super-resolution, the loss function of the target image restoration model in the embodiments of the present application includes two parts of super-resolution loss data and spatial similarity loss data, which can better constrain the training direction. The super-resolution loss data helps the target image restoration model to capture the mapping relationship from the low-resolution image space to the super-resolution image space, and lays the foundation for the target image restoration model to realize the super-resolution capability. The spatial similarity loss data helps the target image restoration model to mine the information of the super-resolution image space, so as to capture and describe the internal feature information of the super-resolution image space, thereby assisting the completion of the super-resolution task.

[0127] ​The training method of the image restoration model of the embodiment of the present application comprises the following steps: obtaining a sample image pair constructed by a first sample image and a second sample image, the image resolution of the first sample image in the sample image pair being lower than the image resolution of the second sample image, performing image standardization processing on the first sample image to obtain a first standard image, performing image standardization processing on the second sample image to obtain a second standard image, inputting the first standard image into a preset original image processing model, performing up-sampling processing on the first standard image by using an up-sampling sub-model of the original image processing model to obtain a preliminary image, the image resolution of the preliminary image being higher than the image resolution of the second sample image, performing diffusion denoising processing on the preliminary image by using a diffusion denoising sub-model of the original image processing model to obtain an intermediate image, calculating target loss data according to the second sample image, the preliminary image and the intermediate image, and adjusting parameters of the original image processing model according to the target loss data to obtain a target image restoration model. The target image restoration model can improve the precision, efficiency and accuracy of image restoration. In addition, the target image restoration model can stably and efficiently learn the potential relationship between low-resolution images (low-pixel images) and low-resolution images (high-pixel images), and can accurately realize the image super-resolution method.

[0128] The training method of the embodiment of the present application is trained according to the auxiliary model framework of DDPM, avoids the adversarial model framework, and is stable in the training process. Meanwhile, the super-resolution precision of the traditional up-sampling model is improved. The target image restoration model obtained by the training embodiment of the present application performs more ideally.

[0129] In addition, the loss function of the target image restoration model of the embodiment of the present application comprises two parts of super-resolution loss data and spatial similarity loss data, which can better constrain the training direction. The super-resolution loss data helps the target image restoration model to capture the mapping relationship from the low-resolution image space to the super-resolution image space, lays the foundation for the ability of the target image restoration model to realize super-resolution, and the spatial similarity loss data helps the target image restoration model to mine the image space information of the super-resolution, thereby assisting the completion of the super-resolution task.

[0130] The target image restoration model of the embodiment of the present application can learn the distribution representation and potential relationship between the low-resolution image space and the high-resolution image space, and can be used for subsequent efficient and accurate completion of the old photo quality restoration task.

[0131] Figure 6 The method in the embodiment of the present application provides an optional flowchart of the image restoration method, Figure 6 The method in the embodiment of the present application can comprise but is not limited to comprising steps 601 to 603.

[0132] Step 601, obtaining a target image to be repaired;

[0133] Step 602: Input the target image into the target image restoration model; wherein, the target image restoration model is trained using the image restoration model training method described above;

[0134] Step 603: The target image is repaired using the target image repair model to obtain the current image; wherein the image resolution of the current image is higher than that of the target image.

[0135] In this embodiment of the application, a low-resolution target image is used as an example for explanation. The target image is an old photo to be restored. In the application scenario of restoring the image quality of old photos, it is only necessary to input the old photo to be restored into the pre-trained target image restoration model for restoration processing, so as to obtain the current image after the resolution is improved (high-resolution human image). The principle is: the old photo is upsampled from the noise space, and after the denoising process, a large amount of high-resolution human image data can be obtained.

[0136] Meanwhile, since the target image restoration model in this application utilizes the advancements in DDPM-assisted super-resolution imaging, this application embodiment can also be used to restore images with high resolution. The principle is illustrated by the following example: For the original high-resolution image X, noise is first added, then denoised, thereby generating a new high-resolution image Y based on the original image X and the target image restoration model. The graphic features of the new high-resolution image Y are different from those of the original image X. For target images with sufficiently high resolution, upsampling processing via an upsampling sub-model is unnecessary.

[0137] While upsampling low-resolution images can improve resolution, it degrades image quality. Therefore, in this embodiment, the target image restoration model performs restoration on the target image by first upsampling the low-resolution image, then performing diffusion processing, and finally denoising to obtain the final current image. This current image improves resolution and enhances image quality.

[0138] The image inpainting method provided in this application is a supervised learning image super-resolution method. It fully mines the feature information (including pixel features) of low-resolution and high-resolution images based on DDPM, and further explores the potential relationship between the corresponding spatial dimensions of low-resolution and high-resolution images, training to stably complete the image super-resolution task. For low-resolution images... The super-resolution image is obtained after upsampling. To further limit the final quality of image super-resolution, DDPM is used for constraint on the generated super-resolution image. ,go through Step diffusion treatment After obtaining the Gaussian noise with a mean of 0 and a variance of a unit matrix , and then Step de-noising processing After that, the picture of the image super-resolution is obtained Unlike the traditional DDPM, this step is not only a sampling generation process, but also needs to participate in the training process. In order to realize the function of accurate image super-resolution, the target loss data of the embodiment of the present application includes two parts of super-resolution loss data and spatial similarity loss data, wherein the super-resolution loss data can be used to ensure that the target image restoration model describes the potential relationship between different spaces, improve the description ability of the target image restoration model to the mapping relationship from the low-resolution image space to the super-resolution picture image space, and the spatial similarity loss data can be used to ensure that the target image restoration model captures and describes the internal feature information of the super-resolution image space, and assists the target image restoration model to realize the image super-resolution method.

[0139] Please refer to Figure 7 The embodiment of the present application also provides a training device of an image restoration model, which can realize the training method of the above-mentioned image restoration model. The device comprises:

[0140] A sample image acquisition module is configured to acquire a sample image pair; wherein the sample image pair comprises a first sample image and a second sample image, and the image resolution of the first sample image is lower than that of the second sample image;

[0141] An image standardization module is configured to perform image standardization processing on the first sample image to obtain a first standard image, and perform image standardization processing on the second sample image to obtain a second standard image;

[0142] An image input module is configured to input the first standard image to a preset original image processing model; wherein the original image processing model comprises an upsampling sub-model and a diffusion de-noising sub-model;

[0143] An upsampling module is configured to perform upsampling processing on the first standard image by using the upsampling sub-model to obtain a preliminary image; wherein the image resolution of the preliminary image is higher than that of the second sample image;

[0144] A diffusion de-noising module is configured to perform diffusion de-noising processing on the preliminary image by using the diffusion de-noising sub-model to obtain an intermediate image;

[0145] A loss calculation module is configured to calculate target loss data according to the second sample image, the preliminary image and the intermediate image;

[0146] A parameter adjustment module is configured to adjust the parameters of the original image processing model according to the target loss data to obtain a target image restoration model.

[0147] The specific implementation of the training device of the image repairing model is basically the same as the above-mentioned specific embodiments of the training method of the image repairing model, and will not be repeated here.

[0148] Please refer to Figure 8 The embodiment of the present application also provides an image repairing device, which can implement the above-mentioned image repairing method. The device comprises:

[0149] An target image acquisition module is configured to acquire a target image to be repaired;

[0150] An target image input module is configured to input the target image to the target image repairing model; wherein the target image repairing model is obtained by the training of the above-mentioned training device of the image repairing model.

[0151] An target image repairing module is configured to repair the target image by the target image repairing model to obtain a current image; wherein the image resolution of the current image is higher than that of the target image.

[0152] In another embodiment, the target image repairing model is obtained by the above-mentioned training method of the image repairing model.

[0153] The specific implementation of the image repairing device is basically the same as the above-mentioned specific embodiments of the image repairing method, and will not be repeated here.

[0154] The embodiment of the present application also provides an electronic device, which comprises a memory and a processor. The memory stores a computer program, and the processor implements the above-mentioned training method of the image repairing model or the image repairing method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0155] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which comprises:

[0156] The processor 901 can be implemented in the form of a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute a related program to implement the technical solutions provided by the embodiments of the present application.

[0157] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 902 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 902 and are invoked and executed by the processor 901 to implement the technical solutions provided by the embodiments of the present application.

[0158] The input / output interface 903 is configured to realize information input and output.

[0159] The communication interface 904 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0160] The bus 905 is configured to transmit information between various components (for example, the processor 901, the memory 902, the input / output interface 903, and the communication interface 904) of the device.

[0161] The processor 901, the memory 902, the input / output interface 903, and the communication interface 904 are connected to each other through the bus 905 to realize the communication connection between the device.

[0162] The embodiments of the present application also provide a storage medium, which is a computer readable storage medium. The storage medium stores a computer program. The computer program is executed by a processor to implement the training method of the image repairing model or the image repairing method.

[0163] The memory is a non-transitory computer readable storage medium, which can be used to store non-transitory software programs and non-transitory computer executable programs. In addition, the memory can include a high-speed random access memory and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor. These remote memories can be connected to the processor through a network. Examples of the network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0164] The training method of the image restoration model, the image restoration method, the training device of the image restoration model, the image restoration device, the electronic equipment and the storage medium provided by the embodiments of the present application obtain a sample image pair constructed by a first sample image and a second sample image, the image resolution of the first sample image in the sample image pair is lower than the image resolution of the second sample image, perform image standardization processing on the first sample image to obtain a first standard image, perform image standardization processing on the second sample image to obtain a second standard image, input the first standard image to a preset original image processing model, perform up-sampling processing on the first standard image by using an up-sampling sub-model of the original image processing model to obtain a preliminary image, the image resolution of the preliminary image is higher than the image resolution of the second sample image, perform diffusion denoising processing on the preliminary image by using a diffusion denoising sub-model of the original image processing model to obtain an intermediate image, calculate target loss data according to the second sample image, the preliminary image and the intermediate image, and adjust parameters of the original image processing model according to the target loss data to obtain a target image restoration model. The target image restoration model can improve the precision, efficiency and accuracy of image restoration. In addition, the target image restoration model can stably and efficiently learn the potential relationship between low-resolution images (low-pixel images) and high-resolution images (high-pixel images), and implement the image super-resolution method with high precision.

[0165] The training method of the embodiments of the present application is trained according to the auxiliary model framework of DDPM, avoids the adversarial model framework, and is stable in the training process. At the same time, the super-resolution precision of the traditional up-sampling model is avoided. The target image restoration model obtained by the training embodiments of the present application performs more ideally.

[0166] In addition, the loss function of the target image restoration model of the embodiments of the present application includes two parts of super-resolution loss data and spatial similarity loss data, which can better constrain the training direction. The super-resolution loss data helps the target image restoration model to capture the mapping relationship from the low-resolution image space to the super-resolution image space, and lays the foundation for the ability of the target image restoration model to realize super-resolution. The spatial similarity loss data helps the target image restoration model to mine the image space information of the super-resolution, thereby assisting the completion of the super-resolution task.

[0167] The target image restoration model of the embodiments of the present application can learn the distribution representation and potential relationship between the low-resolution image space and the high-resolution image space, and can be used for an image restoration method to efficiently and accurately complete the old photo quality restoration task.

[0168] The embodiments described in the specification are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0169] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0170] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0171] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0172] The terms "first", "second", "third", "fourth" and the like (if any) in the specification and above-described drawings of the present application are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0173] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0174] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0175] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or they can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0176] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0177] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0178] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. A training method for an image restoration model, characterized in that, The method includes: Obtain sample image pairs; wherein, the sample image pair includes a first sample image and a second sample image, and the image resolution of the first sample image is lower than that of the second sample image; The first sample image is subjected to image standardization processing to obtain a first standard image, and the second sample image is subjected to image standardization processing to obtain a second standard image; The first standard image is input into a preset original image processing model; wherein, the original image processing model includes an upsampling sub-model and a diffusion denoising sub-model; The first standard image is upsampled using the upsampling sub-model to obtain a preliminary image; wherein the image resolution of the preliminary image is higher than that of the second sample image. The intermediate image is obtained by performing diffusion denoising on the preliminary image using the diffusion denoising sub-model. Target loss data is calculated based on the second sample image, the preliminary image, and the intermediate image; The parameters of the original image processing model are adjusted based on the target loss data to obtain the target image restoration model.

2. The method according to claim 1, characterized in that, The step of performing diffusion denoising processing on the preliminary image using the diffusion denoising sub-model to obtain an intermediate image includes: The initial image is diffused using the diffusion denoising sub-model to obtain a noisy image containing noise. The intermediate image is obtained by denoising the noisy image using the diffusion denoising sub-model.

3. The method according to claim 2, characterized in that, The target loss data includes over-resolution loss data, and the calculation of the target loss data based on the second sample image, the preliminary image, and the intermediate image includes: Calculate target loss data based on the second sample image, the preliminary image, and the intermediate image, including: Calculate the first loss data based on the second sample image and the preliminary image; Calculate the second loss data based on the second sample image and the intermediate image; The over-resolution loss data is obtained by weighting the first loss data and the second loss data.

4. The method according to claim 3, characterized in that, The target loss data also includes spatial similarity loss data, and the calculation of target loss data based on the second sample image, the preliminary image, and the intermediate image further includes: Calculate target loss data based on the second sample image, the preliminary image, and the intermediate image, including: Image spatial features are extracted from the noisy image to obtain a noise spatial feature vector; Calculate the spatial distance between the noise spatial feature vector and the preset reference noise feature vector to obtain image spatial similarity data; The spatial similarity loss data is obtained based on the spatial similarity data.

5. The method according to any one of claims 1 to 4, characterized in that, The step of performing image standardization processing on the first sample image to obtain a first standard image, and performing image standardization processing on the second sample image to obtain a second standard image, includes: The first pixel is obtained by acquiring the pixels of the first sample image, and the second pixel is obtained by acquiring the pixels of the second sample image; The first standard image is obtained by standardizing the first pixel, and the second standard image is obtained by standardizing the second pixel; wherein the pixels of the first standard image are in the interval [-1, 1], and the pixels of the second standard image are in the interval [-1, 1].

6. An image restoration method, characterized in that, The method includes: Obtain the target image to be repaired; The target image is input into the target image restoration model; wherein the target image restoration model is trained using the training method according to any one of claims 1 to 5; The target image is repaired using the target image restoration model to obtain the current image; wherein the image resolution of the current image is higher than that of the target image.

7. A training device for an image restoration model, characterized in that, The device includes: A sample image acquisition module is used to acquire sample image pairs; wherein, the sample image pair includes a first sample image and a second sample image, and the image resolution of the first sample image is lower than that of the second sample image; The image standardization module is used to perform image standardization processing on the first sample image to obtain a first standard image, and to perform image standardization processing on the second sample image to obtain a second standard image; An image input module is used to input the first standard image into a preset original image processing model; wherein, the original image processing model includes an upsampling sub-model and a diffusion denoising sub-model; An upsampling module is used to upsample the first standard image using the upsampling sub-model to obtain a preliminary image; wherein the image resolution of the preliminary image is higher than that of the second sample image; The diffusion denoising module is used to perform diffusion denoising processing on the preliminary image through the diffusion denoising sub-model to obtain an intermediate image; The loss calculation module is used to calculate target loss data based on the second sample image, the preliminary image, and the intermediate image; The parameter adjustment module is used to adjust the parameters of the original image processing model based on the target loss data to obtain the target image restoration model.

8. An image restoration device, characterized in that, The device includes: The target image acquisition module is used to acquire the target image to be repaired. A target image input module is used to input the target image into a target image restoration model; wherein the target image restoration model is trained by the training device according to claim 7; An image restoration module is used to restore the target image using the target image restoration model to obtain a current image; wherein the image resolution of the current image is higher than that of the target image.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement: The method according to any one of claims 1 to 5; or The method of claim 6.

10. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to achieve the following: The method according to any one of claims 1 to 5; or The method of claim 6.

Images