Image restoration method, system, storage medium and terminal device

Abstract

The embodiment of the present invention discloses an image restoration method, system, storage medium and terminal device, which are applied to the field of information processing technology based on artificial intelligence. When restoring a target facial image with some pixels missing or blocked, the image restoration system can obtain a three-dimensional image of the target facial image according to the three-dimensional coefficient of the target facial image, and call a pre-trained image restoration model. In the image restoration model, the features of the target facial image are adjusted by the three-dimensional image to obtain adjusted features, and the restored image of the target facial image is obtained by obtaining the adjusted features, which includes the missing or blocked pixels in the target facial image. In this way, when restoring a target facial image with some or blocked pixels missing, the corresponding three-dimensional image is combined, wherein the data of the three-dimensional image can describe the facial image in more detail, so that the effect of the restored image obtained by restoration is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of information processing technology based on artificial intelligence, and in particular to an image restoration method, system, storage medium and terminal device. Background Art

[0002] During the process of shooting or network transmission, part of the image may be damaged, resulting in missing data of some pixels in the shot or transmitted image. In this way, these pixels will not be displayed after the image is displayed. Therefore, the image needs to be restored to achieve the integrity of the image data.

[0003] In the existing image restoration process, the image can be restored based on the machine learning model of artificial intelligence to obtain the restored image. However, after using the existing image restoration method to restore some images (such as face images, etc.), the effect of the restored image is not very good due to the complexity of the objects in these images. Summary of the invention

[0004] The embodiments of the present invention provide an image restoration method, system, storage medium and terminal device, which improve the restoration effect of facial images.

[0005] An embodiment of the present invention provides an image restoration method, comprising:

[0006] Acquire a target face image, wherein some pixels in the target face image are missing or blocked, and acquire a three-dimensional coefficient of a face contained in the target face image; the three-dimensional coefficient describes the face in complete pixels;

[0007] Acquire a three-dimensional image of the target face image according to the three-dimensional coefficients of the target face image;

[0008] Call the pre-trained image restoration model;

[0009] In the image restoration model, the features of the target facial image are adjusted by the three-dimensional image to obtain adjusted features, a restored image of the target facial image is obtained based on the adjusted features, and the restored image is output. The restored image includes pixels that are missing or blocked in the target facial image.

[0010] Another aspect of the present invention provides an image restoration system, comprising:

[0011] A coefficient acquisition unit, used to acquire a target face image, wherein some pixels in the target face image are missing or blocked, and to acquire three-dimensional coefficients of a face contained in the target face image; the three-dimensional coefficients describe the face in complete pixels;

[0012] A three-dimensional image unit, used for acquiring a three-dimensional image of the target face image according to the three-dimensional coefficient of the target face image;

[0013] A model calling unit, used to call a pre-trained image restoration model;

[0014] A restoration unit is used to adjust the features of the target facial image through the three-dimensional image in the image restoration model to obtain adjusted features, obtain a restored image of the target facial image according to the adjusted features, and output the restored image, wherein the restored image includes pixels that are missing or blocked in the target facial image.

[0015] Another aspect of the embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of computer programs, wherein the computer programs are suitable for being loaded by a processor and executing the image restoration method as described in one aspect of the embodiment of the present invention.

[0016] Another aspect of the embodiment of the present invention further provides a terminal device, including a processor and a memory;

[0017] The memory is used to store multiple computer programs, and the computer programs are used to be loaded by the processor and execute the image restoration method as described in one aspect of an embodiment of the present invention; the processor is used to implement each computer program in the multiple computer programs.

[0018] It can be seen that in the method of this embodiment, when restoring a target face image with some pixels missing or blocked, the image restoration system can obtain a three-dimensional image of the target face image according to the three-dimensional coefficient of the target face image, and call a pre-trained image restoration model, adjust the features of the target face image in the image restoration model through the three-dimensional image, and obtain a restored image of the target face image according to the adjusted features, which includes the pixels missing or blocked in the target face image. In this way, when restoring a target face image with some pixels missing or blocked, the corresponding three-dimensional image is combined, and the data of the three-dimensional image can describe the face image in more detail, so that the effect of the restored image obtained by restoration is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 is a schematic diagram of an image restoration method provided by an embodiment of the present invention;

[0021] Figure 2 is a flow chart of an image restoration method provided by an embodiment of the present invention;

[0022] Figure 3a is a structural schematic diagram of an image restoration model called in one embodiment of the present invention;

[0023] Figure 3b is a schematic diagram of the structure of another image restoration model called in one embodiment of the present invention;

[0024] Figure 4 is a flow chart of a method for training an image restoration model in one embodiment of the present invention;

[0025] Figure 5 is a schematic diagram of an initial image restoration model determined when training an image restoration model in an application embodiment of the present invention;

[0026] Figure 6a is a schematic diagram of a fusion module in an application embodiment of the present invention;

[0027] Figure 6b is a schematic diagram of a second decoding module in an application embodiment of the present invention;

[0028] Figure 6c is a schematic diagram of drawing a line on a second face sample image in an application embodiment of the present invention;

[0029] Figure 7 is a schematic diagram of an image restoration method provided by an application embodiment of the present invention;

[0030] Figure 8 is a schematic diagram of a distributed system to which an image restoration method in another application embodiment of the present invention is applied;

[0031] Fig. 9 is a schematic diagram of a block structure in another application embodiment of the present invention;

[0032] Fig.10 is a schematic diagram of the logical structure of an image restoration system provided by an embodiment of the present invention;

[0033] Fig.11 It is a schematic diagram of the logical structure of a terminal device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] The embodiment of the present invention provides an image restoration method, which is mainly used to restore any face image (i.e., target face image) with missing or blocked pixel data. Figure 1 As shown, the image restoration system of this embodiment can restore the target face image according to the following method:

[0037] A target facial image is obtained, in which some pixels are missing or blocked, and a three-dimensional coefficient of a face contained in the target facial image is obtained; the three-dimensional coefficient describes the face with complete pixels; a three-dimensional image of the target facial image is obtained according to the three-dimensional coefficient of the target facial image; a pre-trained image restoration model is called; in the image restoration model, features of the target facial image are adjusted by the three-dimensional image to obtain adjusted features, a restored image of the target facial image is obtained according to the adjusted features, and the restored image is output, wherein the restored image contains pixels that are missing or blocked in the target facial image.

[0038] In practical applications, the above-mentioned image restoration system can be mainly applied to the following application terminals: mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, aircraft, etc.

[0039] The pre-trained image restoration model here is a machine learning model based on artificial intelligence, where artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0040] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0041] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0042] In this way, when restoring a target face image with some pixels missing or blocked, the corresponding three-dimensional image is combined, wherein the data of the three-dimensional image can describe the face image in more detail, so that the effect of the restored image is greatly improved.

[0043] The embodiment of the present invention provides an image restoration method, which is mainly a method performed by an image restoration system. The flow chart is as follows: Figure 2 As shown, including:

[0044] Step 101, obtaining a target face image, in which some pixels are missing or blocked, and obtaining a three-dimensional coefficient of a face contained in the target face image; the three-dimensional coefficient describes a face with complete pixels.

[0045] It can be understood that, in one case, the image restoration system can provide an interactive interface with the user, so that the user can specify a facial image with some missing or blocked pixels as the target facial image through the interactive interface, and initiate the image restoration process of this embodiment. Or, in another case, when certain specific events occur, the image restoration system can be triggered to use a facial image with some missing or blocked pixels as the target facial image, and initiate the image restoration process of this embodiment.

[0046] In this embodiment, the three-dimensional coefficient of the acquired facial image refers to the information required to realize three-dimensional reconstruction of the facial image contained in the two-dimensional target facial image to obtain the corresponding three-dimensional image, such as the shape and posture of the face contained in the target facial image.

[0047] Specifically, the image restoration system can use a preset coefficient prediction model to obtain the three-dimensional coefficients of any face image. Here, the coefficient prediction model is a machine learning model based on artificial intelligence, which can be trained by a certain training method, and the operation logic of the trained coefficient prediction model is stored in the system in advance. When the process of this embodiment is initiated, the preset coefficient prediction module in the system can be directly called to obtain the three-dimensional coefficients of the target face image.

[0048] The coefficient prediction model may adopt any network structure, such as a residual neural network (Residual Neural Network, ResNet) 50, etc. The three-dimensional coefficients predicted by the coefficient prediction model may describe a complete pixel face.

[0049] Step 102: Acquire a three-dimensional image of the target face image according to the three-dimensional coefficients of the target face image.

[0050] Specifically, the image restoration system can first adopt a three-dimensional reconstruction method to reconstruct the three-dimensional (3D) mesh information of the target face image according to the three-dimensional coefficients of the target face image. The three-dimensional mesh information may include the shapes (S) and textures (T) of multiple faces that make up the three-dimensional face. The three-dimensional face here is the three-dimensional face corresponding to the face contained in the target face image; and then the three-dimensional mesh information is projected onto a two-dimensional plane through rendering to obtain a three-dimensional image of the target face image.

[0051] Among them, the three-dimensional network information refers to the information of all faces composed of three-dimensional objects (such as triangular faces, etc.). In this embodiment, the image restoration system can call a three-dimensional deformable face model (3DMM), and the 3DMM can reconstruct the three-dimensional network information based on the three-dimensional coefficients obtained in the above steps.

[0052] It should be noted that when reconstructing the three-dimensional grid information based on the three-dimensional coefficients, the three-dimensional grid information of a small area of ​​the face can be reconstructed, such as the three-dimensional network information of the area other than the hair and ears; or the three-dimensional grid information of more areas of the face can be reconstructed, so that the restored three-dimensional image has a better effect.

[0053] In addition, it should be noted that even if a large part of the pixels in the target facial image are missing or blocked, the reasonable expression and facial shape in the facial image can still be reconstructed through three-dimensional reconstruction. This is because the three-dimensional deformable face model contains prior information such as the facial shape. It is only necessary to ensure that the missing part of the input image can be determined as a facial image based on the non-missing image in the target facial image, and the three-dimensional mesh information can be obtained through three-dimensional reconstruction.

[0054] Step 103, calling a pre-trained image restoration model.

[0055] Step 104, in the image restoration model, the features of the facial image are adjusted through the three-dimensional image to obtain adjusted features, a restored image of the target facial image is obtained based on the adjusted features, and the restored image is output. The restored image includes pixels that are missing or blocked in the target facial image.

[0056] Specifically, the image restoration system can call the image restoration model with the following structure, such as Figure 3a and Figure 3b As shown, the image restoration model may include: a mapping module, multiple encoding modules, multiple fusion modules, multiple decoding modules and an output module, wherein: the output of an encoding module is connected to an input of a decoding module through a fusion module, and the output of the mapping module is respectively connected to another input of multiple decoding modules; multiple encoding modules are connected in series, multiple decoding modules are also connected in series, and the output of the last decoding module in the multiple decoding modules is connected to the output module. The number of encoding modules, fusion modules and decoding modules is equal. The features of the target face image are extracted through multiple encoding modules, and the features of the target face image are adjusted through the three-dimensional image through multiple fusion modules, and the multiple fusion modules are not connected.

[0057] The relationship between the encoding module and the mapping module can be as follows Figure 3a As shown in Figure 2, there is no direct connection between the mapping module and the encoding module. Their inputs are the same, namely the target face image. The relationship between the encoding module and the mapping module can also be as follows: Figure 3b As shown, the output of the last encoding module among the multiple encoding modules is connected to the mapping module.

[0058] In this way, when obtaining the restored image, the mapping module determines the latent variable according to the data of the target face image, and the latent variable is used to adjust the convolution weight of the convolution calculation involved in the decoding module; the spatial feature of a resolution of the target face image is extracted through the encoding module, that is, one encoding module corresponds to one resolution, and the resolutions corresponding to different encoding modules are different; the fusion module adjusts the spatial feature of a resolution extracted by an encoding module according to the three-dimensional image of the target face image to obtain an adjusted spatial feature of a resolution; the second decoding module among the multiple decoding modules performs convolution calculation according to the latent variable, the adjusted spatial feature of a resolution obtained by a fusion module, and the feature information of a resolution obtained by the previous decoding module of the second decoding module to obtain feature information of another resolution; further, the output module obtains the restored image of the target face image according to the feature information obtained by the last second decoding module.

[0059] Specifically, when the second decoding module obtains feature information of another resolution, the second decoding module can first adjust the convolution weights involved in the convolution calculation according to the latent variables to obtain the adjusted convolution weights, and then perform convolution calculation on the feature information of a resolution obtained by the previous decoding module according to the adjusted convolution weights to obtain the convolution features; finally, based on the adjusted spatial features and convolution features of a resolution obtained by a fusion module, the feature information of another resolution is obtained.

[0060] In this process, the first decoding module among the multiple decoding modules will perform convolution calculation according to the latent variables and the adjusted spatial features of a resolution obtained by a fusion module to obtain feature information of a resolution. Specifically, the first decoding module can first adjust the convolution weights involved in the convolution calculation according to the latent variables to obtain the adjusted convolution weights, and then perform convolution calculation on the preset initial feature information according to the adjusted convolution weights to obtain the convolution features; finally, according to the adjusted spatial features of a resolution obtained by a fusion module and the convolution features, feature information of another resolution is obtained. And in this process, the first encoding module among the multiple encoding modules directly extracts spatial features according to the data of the target face image to obtain spatial features of one resolution; and the other encoding modules among the multiple encoding modules (except the first encoding module) obtain spatial features of another resolution according to the spatial features of a resolution obtained by the previous encoding module.

[0061] It should also be noted that in the above process of obtaining the restored image, the mapping module determines the latent variables. Figure 3a In the case shown in the figure, the mapping module directly determines the latent variables based on the data of the target face image; Figure 3bIn the case shown, the data of the target face image is first extracted by multiple encoding modules to obtain spatial features of a certain resolution, and the mapping module determines the latent variable according to the spatial features of a certain resolution output by the last encoding module among the multiple encoding modules.

[0062] It can be seen that in the method of this embodiment, when restoring a target face image with some pixels missing or blocked, the image restoration system can obtain a three-dimensional image of the target face image according to the three-dimensional coefficient of the target face image, and call a pre-trained image restoration model, adjust the features of the target face image through the three-dimensional image in the image restoration model, and obtain a restored image of the target face image according to the adjusted features, which includes the pixels missing or blocked in the target face image. In this way, when restoring a target face image with some pixels missing or blocked, the corresponding three-dimensional image is combined, and the data of the three-dimensional image can describe the face image in more detail, so that the effect of the restored image obtained by restoration is greatly improved.

[0063] In a specific embodiment, the image restoration model called in step 103 above can be pre-trained according to the following steps, as shown in the flowchart: Figure 4 As shown, including:

[0064] Step 201, determining an initial image restoration model, wherein the initial image restoration model includes: a restoration submodule and a discrimination submodule.

[0065] It can be understood that when the image restoration system determines the initial image restoration model, it will determine the initial values ​​of the parameters in the multi-layer structure and each layer of the initial image restoration model. The parameters of the initial image restoration model refer to the fixed parameters used in the calculation process of each layer of the initial image restoration model, which do not need to be assigned at any time, such as parameter scale, number of network layers, weight value and other parameters.

[0066] Specifically, Figure 5 As shown, the image restoration initial model may include: a complex submodule 10 and a discriminant submodule 11. The complex submodule 10 is used to obtain a restored image of the first face sample image according to the first face sample image and its corresponding three-dimensional face image, and the discriminant submodule 11 is used to discriminate whether the restored image obtained by the complex submodule 10 is true. In this embodiment, the discriminant submodule 11 also needs to discriminate whether the second face sample image is true.

[0067] Specifically, the complex atom module 10 may include: a mapping module 110, multiple encoding modules 120, multiple decoding modules, an output module 140 and multiple fusion modules 150, wherein:

[0068] The encoding module 120 is used to extract spatial features of a resolution of the first face sample image, wherein the first encoding module 120 directly extracts the spatial features of a resolution of the first face sample image, and other encoding modules 120 can obtain spatial features of another resolution based on the spatial features of a resolution obtained by the previous encoding module 120; the mapping module 110 is used to determine latent variables based on the spatial features of a certain resolution extracted by the last encoding module 120, and the latent variables are used to adjust the convolution weights of the convolution calculations involved in each decoding module; each fusion module 150 is used to adjust the spatial features of a resolution extracted by a coding module 120 according to the three-dimensional face image corresponding to the first face sample image, so as to obtain adjusted spatial features of a resolution.

[0069] The second decoding module 131 among the multiple decoding modules is used to perform convolution calculation based on latent variables, adjusted spatial features of a resolution obtained by a fusion module 150, and feature information of a resolution obtained by a previous decoding module of the second decoding module 131 to obtain feature information of another resolution, and the first decoding module 130 among the multiple decoding modules will perform convolution calculation based on latent variables and adjusted spatial features of a resolution obtained by a fusion module 150 to obtain feature information of another resolution; the output module 140 obtains a restored image of the first face sample image based on the feature information obtained by the last second decoding module 131.

[0070] It should be noted that the complex atomic module 10 and the discriminant submodule 11 determined here constitute an adversarial network (Generative Adversarial Network, GAN), and the adversarial network can be used to implement unsupervised machine learning. The adversarial network is mainly composed of a generator and a discriminator. In this embodiment, the generator network is specifically the complex atomic module 10, and the discriminator network is specifically the discriminator submodule 11. Practice has proved that the image restoration model obtained by training the adversarial network has improved the quality of the restored image of the complete pixels for any face image with missing or occluded pixels.

[0071] Step 202, determining training samples, the training samples include multiple sample groups, each sample group includes a first face sample image with missing or partially blocked pixels and its corresponding second face sample image with complete pixels and a three-dimensional face image.

[0072] Specifically, when acquiring the first face sample image, the second face sample image may be processed for occluded pixels to obtain the first face sample image, wherein the processing for occluded pixels may include drawing regular or irregular lines on the second face sample image.

[0073] Step 203, in the reconstruction submodule 10, the features of the first face sample image are adjusted by the three-dimensional face image to obtain adjusted features, and a restored image of the first face sample image is obtained according to the adjusted features. The discrimination submodule 11 discriminates whether the restored image obtained by the reconstruction submodule is true.

[0074] Specifically, the encoding module of the complex atom module 10 in the initial model of image restoration extracts the spatial features of the first face sample image, adjusts the spatial features through the fusion module, and obtains the adjusted spatial features. At the same time, the latent variables are obtained through the mapping module to adjust the convolution weights in the decoding module; the decoding module then performs convolution calculation on the adjusted spatial features obtained by the fusion module based on the convolution weights adjusted by the latent variables, so as to obtain the convolution features, and finally the output module outputs the restored image of the first face sample image according to the convolution features.

[0075] The discrimination submodule 11 further discriminates the restored image obtained by the reconstruction submodule 10. In this embodiment, the discrimination submodule 11 needs to discriminate whether the restored image obtained by the reconstruction submodule 10 is authentic, and also needs to discriminate whether the second face sample image is authentic.

[0076] Step 204, according to the result obtained by the discriminant submodule 11 and the second face sample image in the training sample, the image restoration initial model is adjusted, and the restoration submodule 10 in the adjusted image restoration initial model is a pre-trained image restoration model, which is used to restore images with some pixels missing or blocked.

[0077] Specifically, the image restoration system will first calculate the loss function related to the initial image restoration model based on the result obtained by the discriminant submodule 11 in the above step 203 and the corresponding second face sample image. The loss function is used to indicate the error between the restored image of the first face sample image obtained by the restoration submodule 10 and the actual complete pixel image of the first face sample image (i.e., the second face sample image), such as the cross entropy loss function, etc. The error can be obtained through the result obtained by the discriminant submodule 11; then the parameter values ​​of the parameters in the initial image restoration model are adjusted according to the loss function.

[0078] In the embodiment of the present invention, when calculating the loss function, the image restoration system may calculate, but is not limited to, the following loss sub-functions: feature loss sub-function, adversarial loss sub-function, and reconstruction loss sub-function, and use the function calculation values ​​of the feature loss sub-function, adversarial loss sub-function, and reconstruction loss sub-function as the loss function related to the initial image restoration model, such as using the sum of these sub-functions as the loss function. Wherein:

[0079] Adversarial loss subfunction L GAN, which is used to represent the result D(G(input)) of the above-mentioned discrimination submodule 11 judging whether the restored image of the first face sample image obtained by the complex submodule 10 is true, and the expectation of the result D(GT) of the discrimination submodule 11 judging whether the actual complete pixel image of the first face sample image (i.e., the second face sample image) is true, which can be specifically represented by the following formula 1, where D represents the discrimination submodule 11, G represents the complex submodule 10, GT represents the second face sample image with complete pixels in the training sample, and input represents the first face sample image with missing or blocked pixels in the training sample:

[0080] L GAN =m G inm D axE[log(D(GT))]+E[log(1-D(G(input)))] (1)

[0081] Reconstruction loss subfunction L rec , which is used to represent the difference between the restored image G(input) of the first face sample image obtained by the above-mentioned reconstruction submodule 10 and the actual complete pixel image of the first face sample image (i.e., the second face sample image GT), can be specifically represented by the following formula 2. The reconstruction loss subfunction can include the difference between the restored image G(input) and the second face sample image GT, and the difference between the feature information LPIPS(G(input)) of the restored image and the feature information LPIPS(GT) of the second face sample image:

[0082] L rec =|G(input)-GT l -|LPIPS(G(input))-LPIPS(GT) l (2)

[0083] Feature loss subfunction L rec , which is used to represent the difference between the feature information of the restored image G(input) of the first face sample image obtained by the restoration submodule and the feature information of the actual complete pixel image of the first face sample image (i.e., the second face sample image GT), and can be specifically represented by the following formula 3. The feature loss subfunction can include the feature information F of the restored image G(input): ArcFace (G(input)) and the feature information F of the second face sample image GT ArcFace (GT) distance between:

[0084] L ID =1-F cos (F ArcFace (G(input)),F ArcFace(GT)) (3)

[0085] Therefore, the loss function L related to the initial image restoration model calculated by the image restoration system can be expressed by the following formula 4:

[0086] L=L GAN +L rec +L ID (4)

[0087] The training process of the image restoration model is to minimize the value of the above loss function. The training process is to continuously optimize the parameter values ​​of the parameters in the initial image restoration model determined in step 201 through a series of mathematical optimization methods such as back propagation derivation and gradient descent, and minimize the calculated value of the above loss function. Specifically, when the function value of the calculated loss function is large, such as greater than a preset value, it is necessary to change the parameter value, such as reducing the weight value of a certain neuron connection, so that the function value of the loss function calculated according to the adjusted parameter value is reduced.

[0088] It should be noted that, in the actual implementation process of adjusting the initial model of image restoration according to the calculated loss function, in one case, the parameter values ​​of the parameters in the complex submodule 10 and the discriminant submodule 11 can be adjusted simultaneously according to the loss function obtained by the above formula 4. In another case, the parameter values ​​of the parameters in the complex submodule 10 can be fixed first, and the parameter values ​​of the parameters in the discriminant submodule 11 can be adjusted according to the adversarial loss subfunction obtained by the above formula 1; and then the parameter values ​​of the parameters in the discriminant submodule 11 after the adjustment are fixed, and the parameter values ​​of the parameters in the complex submodule 10 can be adjusted according to the loss function obtained by the above formulas 1 to 4.

[0089] Among them, the process of adjusting the parameter values ​​of the parameters in the discriminant submodule 11 is to minimize the adversarial loss subfunction obtained by the above formula 1, that is, it is necessary to make the discriminant submodule 11 judge that the restored image of the first face sample image obtained by the complex submodule 10 is false, and judge that the second face sample image is true.

[0090] In addition, it should be noted that the above steps 203 to 204 are the results obtained by the reconstruction submodule 10 and the discrimination submodule 11 in the image restoration initial model, which are an adjustment of the parameter values ​​in the image restoration initial model. In actual applications, it is necessary to continuously loop through the above steps 203 to 204 until the adjustment of the parameter values ​​meets certain stop conditions.

[0091] Therefore, after executing steps 201 to 204 of the above embodiment, the image restoration system needs to determine whether the current adjustment of the parameter value meets the preset stop condition. If it does, the process ends, and the restoration atom module in the image restoration initial model obtained by the adjustment in step 204 is used as a pre-trained image restoration model and preset into the image restoration system; if it does not meet the condition, the image restoration initial model after the parameter value is adjusted returns to execute the above steps 203 to 204. The preset stop condition includes but is not limited to any one of the following conditions: the difference between the current adjusted parameter value and the last adjusted parameter value is less than a threshold value, that is, the adjusted parameter value reaches convergence; and the number of parameter value adjustments is equal to the preset number of times, etc.

[0092] It can be seen that in the process of pre-training the image restoration model in this embodiment, the adversarial network formed by the restoration sub-module 10 and the discriminant sub-module 11 does not need to be manually labeled for any sample image in the determined training samples, thereby realizing unsupervised training of the image restoration model and saving the time spent on manual labeling.

[0093] The following is a specific application example to illustrate the image restoration method of the present invention. The method of this embodiment may include the following two parts:

[0094] (I) Pre-trained image restoration model

[0095] Specifically, the method used by the image restoration system when training the image restoration model is the same as that described above. Figure 4 The training method shown is similar, except that in this embodiment:

[0096] (1) In this embodiment, when determining the structure of the initial model for image restoration, it can be specifically determined that the restoration submodule includes 7 decoding modules, 7 fusion modules and 7 encoding modules, and the resolutions corresponding to the encoding modules and decoding modules can be increased from 4*4 to 512*512, wherein the structure of any fusion module is as follows: Figure 6a As shown, where:

[0097] Assume that the three-dimensional face image corresponding to the first face sample image is I 3d , the spatial feature of a resolution extracted by the encoding module connected to the fusion module is The fusion module will transform the 3D face image I 3d Zoom to the space feature The size of is consistent, and the spatial domain adaptive regularization module (Spatially-Adaptive Normalization, SPADE) can be used to obtain the adjustment coefficients α and β to adjust the spatial features Specifically, it can be expressed as follows: The spatial feature of a resolution After a convolution layer, the adjustment coefficients are superimposed through two overlay layers to obtain a spatial feature with an adjusted resolution. Specifically, it can be expressed as the following formula 6:

[0098] α, β = SPADE (I 3d ) (5)

[0099]

[0100] (2) The structure of the second decoding module included in the decoding module in the initial image restoration model determined in this embodiment is as follows: Figure 6b As shown, where:

[0101] Assume that a fusion module connected to the second decoding module obtains a spatial feature with an adjusted resolution of The feature information of a resolution obtained by the previous decoding module of the second decoding module is in, and The corresponding resolutions are the same, so:

[0102] When the latent variable w of the mapping module is input to the second decoding module, it is processed by A, Mod and Demod and then input to a convolution layer (Conv) to adjust the convolution weight of the convolution layer to obtain the adjusted convolution weight; and the feature information of a resolution After upsampling and a convolution layer, convolution calculation is performed in the convolution layer according to the adjusted convolution weight to obtain the convolution feature. When the mapping module obtains the latent variable w, it mainly calculates the first face sample image I with missing or blocked pixels. lq It is obtained, which is expressed as follows:

[0103] w=MLP(I lq ) (7)

[0104] The adjusted spatial features of one resolution are Input to a concatenation layer C, which adjusts the spatial features Based on feature information The convolutional features obtained can be connected in series to obtain feature information of another resolution. Specifically, it can be expressed by the following formula 8:

[0105]

[0106] (3) In this embodiment, when determining the training sample, a second face sample image with complete and unblocked pixels can be first determined, and then irregular lines can be drawn on the second face sample image. These irregular lines can be as follows: Figure 6c As shown, the first face sample image of the partially blocked pixels can be obtained.

[0107] (4) In this embodiment, the learning rate ratio of each encoding module, decoding module, output module, and discrimination submodule in the complex atomic module can be 100:10:1.

[0108] It should be noted that after the image restoration model is trained, the operating logic of the image restoration model can be preset in the system.

[0109] (ii) Figure 7 As shown in the figure, the image restoration system can restore any target face image with missing or blocked pixels:

[0110] In step 301, a user may specify a facial image with some pixels missing or blocked as a target facial image through a user interface provided by the image restoration system, and trigger restoration of the target facial image.

[0111] Step 302: The image restoration system can predict the target face image I based on the pre-trained 3D coefficient prediction network, such as ResNet50. lq The three-dimensional coefficient (coeff) of the face image contained in , which can describe the face of the complete pixel, can be specifically expressed by the following formula 9:

[0112] coeff=F res50 (I lq ) (9)

[0113] Step 303, the image restoration system can call the three-dimensional deformable face model (3DMM) to reconstruct the three-dimensional mesh information (3D mesh) according to the three-dimensional coefficients obtained above. The three-dimensional mesh information may include the shapes (S) and textures (T) of multiple faces constituting the three-dimensional face (and the face image in the target face image). Then, the three-dimensional mesh information can be projected onto a two-dimensional plane through a renderer to obtain the three-dimensional face image I of the target face image. 3d , the data of the three-dimensional image may include information such as face shape, light and shadow, etc. Specifically, when obtaining the three-dimensional image, it can be expressed as follows:

[0114] S, T = F 3dmm (coeff) (10)

[0115] I 3d =F render(S,T) (11)

[0116] Step 304: the image restoration system calls a preset image restoration model and uses the target face image and its 3D image data as two inputs of the image restoration model, namely, the encoding module and the fusion module in the image restoration model.

[0117] Step 305: In the image restoration model, the features of the target facial image are adjusted by the three-dimensional image to obtain adjusted features, and a restored image of the target facial image is obtained according to the adjusted features, and the restored image of the target facial image is output.

[0118] Specifically, the encoding module in the image restoration model extracts the spatial features of the target face image, the mapping module obtains latent variables based on the spatial features extracted by the encoding module to adjust the convolution weights in the decoding module, and the fusion module adjusts the spatial features extracted by the encoding module according to the three-dimensional image of the target face image; then the decoding module performs convolution calculation on the adjusted spatial features obtained by the fusion module based on the convolution weights adjusted by the latent variables to obtain the convolution features, and finally the output module outputs the restored image of the target face image based on the convolution features.

[0119] It can be seen that the pre-trained image restoration model in this embodiment can be combined with the two-dimensional target face image and its three-dimensional image, and the restored image obtained has complete pixels, and the effect is improved.

[0120] The following is another specific application example to illustrate the image restoration method of the present invention. The image restoration system in the embodiment of the present invention is mainly a distributed system 100, which may include a client 300 and multiple nodes 200 (any form of computing device in the access network, such as a server, a user terminal), and the client 300 and the node 200 are connected through network communication.

[0121] Take the distributed system as the blockchain system as an example, see Figure 8 This is an optional structural diagram of the distributed system 100 provided by the embodiment of the present invention applied to the blockchain system, which is formed by multiple nodes 200 (any form of computing devices in the access network, such as servers and user terminals) and clients 300, and a peer-to-peer (P2P) network is formed between the nodes. The P2P protocol is an application layer protocol running on the Transmission Control Protocol (TCP) protocol. In a distributed system, any machine such as a server or a terminal can join and become a node, and the node includes a hardware layer, an intermediate layer, an operating system layer, and an application layer.

[0122] See also Figure 8The functions of each node in the blockchain system shown include:

[0123] 1) Routing: a basic function of a node, used to support communication between nodes.

[0124] In addition to the routing function, the node can also have the following functions:

[0125] 2) Applications, which are deployed in the blockchain to implement specific businesses according to actual business needs, record data related to the implementation of functions to form record data, carry digital signatures in the record data to indicate the source of the task data, and send the record data to other nodes in the blockchain system for other nodes to add the record data to the temporary block when they successfully verify the source and integrity of the record data.

[0126] For example, the business implemented by the application includes the code of the image restoration function, which mainly includes:

[0127] A target face image is obtained, in which some pixels are missing or blocked, and a three-dimensional coefficient of the face contained in the target face image is obtained; the three-dimensional coefficient describes the face of the complete pixels; a three-dimensional image of the target face image is obtained according to the three-dimensional coefficient of the target face image; a pre-trained image restoration model is called; in the image restoration model, the features of the target face image are adjusted by the three-dimensional image to obtain adjusted features, a restored image of the target face image is obtained according to the adjusted features, and the restored image is output, wherein the restored image contains pixels that are missing or blocked in the target face image.

[0128] 3) Blockchain, including a series of blocks that are connected to each other in the order of their generation. Once a new block is added to the blockchain, it will not be removed. The block records the record data submitted by the nodes in the blockchain system.

[0129] See also Fig. 9 An optional schematic diagram of the block structure provided by the embodiment of the present invention, each block includes the hash value of the transaction record stored in this block (the hash value of this block) and the hash value of the previous block, and each block is connected by the hash value to form a blockchain. In addition, the block can also include information such as the timestamp when the block was generated. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains relevant information for verifying the validity of its information (anti-counterfeiting) and generating the next block.

[0130] The embodiment of the present invention also provides an image restoration system, the structural diagram of which is shown in FIG. Fig.10As shown, it may specifically include:

[0131] The coefficient acquisition unit 20 is used to acquire a target face image, in which some pixels are missing or blocked, and to acquire three-dimensional coefficients of the target face image, wherein the three-dimensional system describes a face with complete pixels.

[0132] The three-dimensional image unit 21 is used to obtain a three-dimensional image of the target face image according to the three-dimensional coefficients of the target face image obtained by the coefficient obtaining unit 20.

[0133] The three-dimensional image unit 21 is specifically used to reconstruct the three-dimensional network information of the target facial image based on the three-dimensional coefficients of the target facial image; the three-dimensional network information includes the shapes and textures of multiple faces that constitute the three-dimensional face, and the three-dimensional face is a three-dimensional face corresponding to the face contained in the target facial image; the three-dimensional network information is projected onto a two-dimensional plane by rendering to obtain a three-dimensional image of the target facial image.

[0134] The model calling unit 22 is used to call a pre-trained image restoration model.

[0135] The model calling unit 22 is specifically used to call the image restoration model of the following structure: the image restoration model includes: a mapping module, multiple encoding modules, multiple fusion modules, multiple decoding modules and an output module, wherein: the output of an encoding module is connected to an input of a decoding module through a fusion module, and the output of the mapping module is respectively connected to another input of the multiple decoding modules; the multiple encoding modules are connected in series, the multiple decoding modules are connected in series, and the output of the last decoding module among the multiple decoding modules is connected to the output module.

[0136] The restoration unit 23 is used to adjust the features of the target facial image in the image restoration model called by the model calling unit 22 through the three-dimensional image obtained by the three-dimensional image unit 21 to obtain adjusted features, obtain a restored image of the target facial image according to the adjusted features, and output the restored image, wherein the restored image includes pixels that are missing or blocked in the target facial image.

[0137] The restoration unit 23 is specifically used for the mapping module to determine latent variables according to the data of the target face image, and the latent variables are used to adjust the convolution weights of the convolution calculation involved in the decoding module; the spatial features of a resolution of the target face image are extracted by the encoding module; the fusion module adjusts the spatial features of a resolution extracted by the encoding module according to the three-dimensional image to obtain an adjusted spatial feature of a resolution; the second decoding module among the multiple decoding modules performs convolution calculation according to the latent variables, the adjusted spatial features of a resolution obtained by a fusion module, and the feature information of the resolution obtained by the previous decoding module of the second decoding module to obtain feature information of another resolution; the output module obtains the restored image of the target face image according to the feature information obtained by the last of the second decoding modules.

[0138] Among them, when the second decoding module in the restoration unit 23 performs convolution calculation according to the latent variable, the adjusted spatial features of one resolution obtained by a fusion module, and the feature information of the one resolution obtained by the previous decoding module of the second decoding module to obtain feature information of another resolution, the second decoding module is specifically used to adjust the convolution weights involved in the convolution calculation according to the latent variable to obtain the adjusted convolution weights, and perform convolution calculation on the feature information of the one resolution according to the adjusted convolution weights to obtain the convolution features; and obtain the feature information of another resolution according to the adjusted spatial features of the one resolution and the convolution features. And in the process of the image restoration model obtaining the restored image of the target face image according to the target face image and the three-dimensional image, the first decoding module among the multiple decoding modules performs convolution calculation according to the latent variable and the adjusted spatial features of one resolution obtained by a fusion module to obtain feature information of another resolution.

[0139] Furthermore, the image restoration system in this embodiment may also include:

[0140] The training unit 24 is used to determine an initial image restoration model; the initial image restoration model includes: a complex submodule and a discriminant submodule; determine training samples, the training samples include multiple sample groups, each sample group includes a first face sample image with missing or blocked pixels and its corresponding second face sample image with complete pixels and a three-dimensional face image; in the complex submodule, an adjusted feature is obtained according to the feature of the first face sample image through three-dimensional face image adjustment, and the discriminant submodule discriminates whether the restored image obtained by the complex submodule is true based on the restored image of the first face sample image with the adjusted feature; according to the result obtained by the discriminant submodule and the second face sample image in the training sample, the initial image restoration model is adjusted, and the complex submodule in the adjusted initial image restoration model is the pre-trained image restoration model.

[0141] The training unit 24 adjusts the initial image restoration model according to the results obtained by the discriminant submodule and the second face sample image in the training sample. It is specifically used to calculate the loss function related to the initial image restoration model according to the results obtained by the discriminant submodule and the second face sample image in the training sample, and adjust the parameter values ​​of the parameters in the initial image restoration model according to the loss function. Among them, when calculating the loss function related to the initial model of image restoration according to the result obtained by the discriminant submodule, the training unit 24 is specifically used to calculate the feature loss subfunction, the adversarial loss subfunction and the reconstruction loss subfunction; wherein the feature loss subfunction is used to represent the difference between the feature information of the restored image of the first face sample image obtained by the complex submodule and the feature information of the second face sample image; the adversarial loss subfunction is used to represent the result of the discriminant submodule judging whether the restored image of the first face sample image obtained by the complex submodule is true, and the expectation of the result of the discriminant submodule judging whether the second face sample image is true; the reconstruction loss subfunction is used to represent the difference between the restored image of the first face sample image obtained by the complex submodule and the second face sample image; the function calculation value of the feature loss subfunction, the adversarial loss subfunction and the reconstruction loss subfunction is used as the loss function.

[0142] Furthermore, the training unit 24 is also used to stop adjusting the parameter value when the number of adjustments to the parameter value is equal to a preset number, or when the difference between the currently adjusted parameter value and the last adjusted parameter value is less than a threshold.

[0143] It can be seen that in the image restoration system of this embodiment, the three-dimensional image unit 21 can obtain the three-dimensional image of the target face image according to the three-dimensional coefficient of the target face image with missing or blocked pixels, and the model calling unit 22 calls the pre-trained image restoration model, and adjusts the features of the target face image through the three-dimensional image in the restoration unit 23, and obtains the restored image of the target face image according to the adjusted features, which includes the missing or blocked pixels and has complete pixels. In this way, when restoring the target face image with missing or blocked pixels, the corresponding three-dimensional image is combined, and the data of the three-dimensional image can describe the face image in more detail, so that the effect of the restored image obtained by restoration is greatly improved.

[0144] The embodiment of the present invention further provides a terminal device, the structural diagram of which is shown in FIG. Fig.11 As shown, the terminal device may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 30 (for example, one or more processors) and memory 31, and one or more storage media 32 (for example, one or more mass storage devices) storing application programs 321 or data 322. Among them, the memory 31 and the storage medium 32 can be short-term storage or permanent storage. The program stored in the storage medium 32 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the terminal device. Furthermore, the central processing unit 30 can be configured to communicate with the storage medium 32 to execute a series of instruction operations in the storage medium 32 on the terminal device.

[0145] Specifically, the application 321 stored in the storage medium 32 includes an image restoration application, and the program may include the coefficient acquisition unit 20, the three-dimensional image unit 21, the model calling unit 22, the restoration unit 23 and the training unit 24 in the above-mentioned image restoration system, which will not be described in detail here. Furthermore, the central processing unit 30 can be configured to communicate with the storage medium 32, and execute a series of operations corresponding to the image restoration application stored in the storage medium 32 on the terminal device.

[0146] The terminal device may also include one or more power supplies 33, one or more wired or wireless network interfaces 34, one or more input and output interfaces 35, and / or one or more operating systems 323, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, FreeBSD TM, etc.

[0147] The steps performed by the image restoration system in the above method embodiment can be based on the Fig.11The structure of the terminal device shown.

[0148] Furthermore, another aspect of the embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a plurality of computer programs, wherein the computer programs are suitable for being loaded by a processor and executing the image restoration method performed by the above-mentioned image restoration system.

[0149] Another aspect of the embodiment of the present invention further provides a terminal device, including a processor and a memory;

[0150] The memory is used to store a plurality of computer programs, and the computer programs are used to be loaded by the processor and execute the image restoration method executed by the above-mentioned image restoration system; the processor is used to implement each computer program in the plurality of computer programs.

[0151] In addition, according to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the image restoration method provided in the above various optional implementations.

[0152] A person skilled in the art may understand that all or part of the steps in the various methods of the above-mentioned embodiments may be completed by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the storage medium may include: a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc.

[0153] The above is a detailed introduction to an image restoration method, system, storage medium and terminal device provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. An image restoration method, characterized in that: include: Acquire a target face image, wherein some pixels in the target face image are missing or blocked, and acquire a three-dimensional coefficient of a face contained in the target face image; the three-dimensional coefficient describes the face in complete pixels; Acquire a three-dimensional image of the target face image according to the three-dimensional coefficients of the target face image; Call the pre-trained image restoration model; In the image restoration model, the features of the target facial image are adjusted by the three-dimensional image to obtain adjusted features, a restored image of the target facial image is obtained according to the adjusted features, and the restored image is output, wherein the restored image includes pixels that are missing or blocked in the target facial image; The method of adjusting the features of the target face image by using the three-dimensional image in the image restoration model to obtain adjusted features, obtaining a restored image of the target face image according to the adjusted features, and outputting the restored image specifically includes: The mapping module in the image restoration model determines latent variables according to the data of the target face image, and the latent variables are used to adjust the convolution weights of the convolution calculation involved in the decoding module in the image restoration model; Extracting a spatial feature of a resolution of the target face image through the encoding module in the image restoration model; The fusion module in the image restoration model adjusts a spatial feature of a resolution extracted by the encoding module according to the three-dimensional image to obtain an adjusted spatial feature of a resolution; A second decoding module among the multiple decoding modules in the image restoration model performs convolution calculation according to the latent variable, the adjusted spatial features of one resolution obtained by a fusion module, and the feature information of the one resolution obtained by a previous decoding module of the second decoding module to obtain feature information of another resolution; The output module in the image restoration model obtains the restored image of the target face image according to the feature information obtained by the last of the second decoding modules.

2. The method according to claim 1, characterized in that The step of acquiring a three-dimensional image of the target face image according to the three-dimensional coefficient of the target face image specifically includes: Reconstructing three-dimensional network information of the target face image according to the three-dimensional coefficient of the target face image; the three-dimensional network information includes shapes and textures of multiple faces constituting a three-dimensional face, and the three-dimensional face is a three-dimensional face corresponding to the face included in the target face image; The three-dimensional network information is projected onto a two-dimensional plane by rendering to obtain a three-dimensional image of the target face image.

3. The method according to claim 1, characterized in that The calling of the pre-trained image restoration model specifically includes: Call the image restoration model with the following structure: The image restoration model includes: a mapping module, multiple encoding modules, multiple fusion modules, multiple decoding modules and an output module, wherein: The output of an encoding module is connected to an input of a decoding module through a fusion module, and the output of the mapping module is respectively connected to another input of the multiple decoding modules; the multiple encoding modules are connected in series, the multiple decoding modules are connected in series, and the output of the last decoding module among the multiple decoding modules is connected to the output module.

4. The method according to claim 1, characterized in that The second decoding module among the multiple decoding modules performs convolution calculation according to the latent variable, the adjusted spatial features of one resolution obtained by a fusion module, and the feature information of the one resolution obtained by a previous decoding module of the second decoding module to obtain feature information of another resolution, specifically including: The second decoding module adjusts the convolution weights involved in the convolution calculation according to the latent variable to obtain adjusted convolution weights, and performs convolution calculation on the feature information of the one resolution according to the adjusted convolution weights to obtain convolution features; Feature information of another resolution is obtained according to the adjusted spatial features of the one resolution and the convolved features.

5. The method according to claim 1, characterized in that In the process of obtaining the restored image of the target face image according to the target face image and the three-dimensional image by the image restoration model, the first decoding module among the multiple decoding modules performs convolution calculation based on the adjusted spatial features of one resolution obtained by the latent variables and a fusion module to obtain feature information of another resolution.

6. The method according to any one of claims 1 to 5, characterized in that: Determine an initial image restoration model; the initial image restoration model includes: a restoration submodule and a discrimination submodule; Determine a training sample, wherein the training sample includes a plurality of sample groups, each sample group includes a first face sample image with missing or blocked pixels and a corresponding second face sample image with complete pixels and a three-dimensional face image; In the complex submodule, the features of the first face sample image are adjusted by the three-dimensional face image to obtain adjusted features, a restored image of the first face sample image is obtained according to the adjusted features, and the discrimination submodule discriminates whether the restored image obtained by the complex submodule is true; The image restoration initial model is adjusted according to the result obtained by the discriminant submodule and the second face sample image in the training sample, and the restoration submodule in the adjusted image restoration initial model is the pre-trained image restoration model.

7. The method according to claim 6, characterized in that The adjusting the image restoration initial model according to the result obtained by the discriminant submodule and the second face sample image in the training sample specifically includes: According to the result obtained by the discriminant submodule and the second face sample image in the training sample, a loss function related to the image restoration initial model is calculated, and the parameter values ​​of the parameters in the image restoration initial model are adjusted according to the loss function.

8. The method according to claim 7, characterized in that The calculation of the loss function related to the image restoration initial model based on the result obtained by the discriminant submodule and the second face sample image in the training sample specifically includes: Calculate feature loss sub-function, adversarial loss sub-function and reconstruction loss sub-function; The feature loss sub-function is used to represent the difference between the feature information of the restored image of the first face sample image obtained by the complex atom module and the feature information of the second face sample image; the adversarial loss sub-function is used to represent the result of the discrimination sub-module discriminating whether the restored image of the first face sample image obtained by the complex atom module is true, and the expectation of the result of the discrimination sub-module discriminating whether the second face sample image is true; the reconstruction loss sub-function is used to represent the difference between the restored image of the first face sample image obtained by the complex atom module and the second face sample image; The function calculation value of the feature loss sub-function, the adversarial loss sub-function and the reconstruction loss sub-function is used as the loss function.

9. The method according to claim 7, characterized in that When the number of times the parameter value is adjusted is equal to a preset number, or when the difference between the currently adjusted parameter value and the last adjusted parameter value is less than a threshold, the adjustment of the parameter value is stopped.

10. An image restoration system, characterized in that: include: A coefficient acquisition unit, used to acquire a target face image, wherein some pixels in the target face image are missing or blocked, and to acquire three-dimensional coefficients of a face contained in the target face image; the three-dimensional coefficients describe the face in complete pixels; A three-dimensional image unit, used for acquiring a three-dimensional image of the target face image according to the three-dimensional coefficient of the target face image; A model calling unit, used to call a pre-trained image restoration model; a restoration unit, configured to adjust the features of the target facial image by using the three-dimensional image in the image restoration model to obtain adjusted features, obtain a restored image of the target facial image according to the adjusted features, and output the restored image, wherein the restored image includes pixels that are missing or blocked in the target facial image; The restoration unit is specifically used for the mapping module in the image restoration model to determine latent variables according to the data of the target face image, and the latent variables are used to adjust the convolution weights of the convolution calculation involved in the decoding module in the image restoration model; and extract a spatial feature of a resolution of the target face image through the encoding module in the image restoration model; The fusion module in the image restoration model adjusts a spatial feature of a resolution extracted by the encoding module according to the three-dimensional image to obtain an adjusted spatial feature of a resolution; The second decoding module among the multiple decoding modules in the image restoration model performs convolution calculation based on the latent variable, the adjusted spatial features of one resolution obtained by a fusion module, and the feature information of the one resolution obtained by a previous decoding module of the second decoding module to obtain feature information of another resolution; the output module in the image restoration model obtains the restored image of the target face image based on the feature information obtained by the last one of the second decoding modules.

11. An image restoration method, characterized in that: include: Determine the initial model for image restoration; The image restoration initial model includes: a restoration submodule and a discrimination submodule; Determine a training sample, wherein the training sample includes a plurality of sample groups, each sample group includes a first face sample image with missing or blocked pixels and a corresponding second face sample image with complete pixels and a three-dimensional face image; In the complex submodule, the features of the first face sample image are adjusted by the three-dimensional face image to obtain adjusted features, a restored image of the first face sample image is obtained according to the adjusted features, and the discrimination submodule discriminates whether the restored image obtained by the complex submodule is true; According to the result obtained by the discriminant submodule and the second face sample image in the training sample, the image restoration initial model is adjusted, and the restoration submodule in the adjusted image restoration initial model is used to restore the image with some pixels missing or blocked; Among them, the complex atomic module is used to determine the latent variable according to the data of the first face sample image through the mapping module, and the latent variable is used to adjust the convolution weight of the convolution calculation involved in the decoding module; the spatial feature of a resolution of the first face sample image is extracted through the encoding module; the fusion module adjusts the spatial feature of a resolution extracted by the encoding module according to the three-dimensional face image to obtain an adjusted spatial feature of a resolution; the second decoding module among the multiple decoding modules performs convolution calculation according to the latent variable, the adjusted spatial feature of a resolution obtained by a fusion module and the feature information of the resolution obtained by the previous decoding module of the second decoding module to obtain feature information of another resolution; the output module obtains the restored image of the first face sample image according to the feature information obtained by the last one of the second decoding modules.

12. The method according to claim 11, characterized in that The determining of the initial model for image restoration specifically includes: Determine that the image restoration initial model includes a restoration atomic module with the following structure: The complex atom module includes: a mapping module, multiple encoding modules, multiple fusion modules, multiple decoding modules and an output module, wherein: The output of an encoding module is connected to an input of a decoding module through a fusion module, and the output of the mapping module is respectively connected to another input of the multiple decoding modules; the multiple encoding modules are connected in series, the multiple decoding modules are connected in series, and the output of the last decoding module among the multiple decoding modules is connected to the output module.

13. The method according to claim 11 or 12, characterized in that: The adjusting the image restoration initial model according to the result obtained by the discriminant submodule and the second face sample image in the training sample specifically includes: Calculate feature loss sub-function, adversarial loss sub-function and reconstruction loss sub-function; The feature loss sub-function is used to represent the difference between the feature information of the restored image of the first face sample image obtained by the complex atom module and the feature information of the second face sample image; the adversarial loss sub-function is used to represent the result of the discrimination sub-module discriminating whether the restored image of the first face sample image obtained by the complex atom module is true, and the expectation of the result of the discrimination sub-module discriminating whether the second face sample image is true; the reconstruction loss sub-function is used to represent the difference between the restored image of the first face sample image obtained by the complex atom module and the second face sample image; The function calculation value of the feature loss sub-function, the adversarial loss sub-function and the reconstruction loss sub-function is used as the loss function; The parameter values ​​of the parameters in the initial image restoration model are adjusted according to the loss function.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of computer programs, and the computer programs are suitable for being loaded by a processor and executing the image restoration method according to any one of claims 1 to 9, or the image restoration method according to any one of claims 11 to 13.

15. A terminal device, characterized in that: including a processor and a memory; The memory is used to store multiple computer programs, and the computer programs are used to be loaded by the processor and execute the image restoration method as described in any one of claims 1 to 9, or the image restoration method as described in any one of claims 11 to 13; the processor is used to implement each computer program in the multiple computer programs.

Images