Face image restoration processing and model training method and device, equipment and medium

Abstract

The embodiment of the invention relates to a face image restoration processing and model training method and device, equipment and a medium, and the method comprises the steps: constructing an initial face image restoration model which comprises a preset number of densely connected residual blocks which are connected in series and a restoration image output module, extracting a first face image feature map of the first sample face image through a preset number of densely connected residual blocks which are connected in series, and obtaining and outputting an actual restoration image through an initial face image restoration model according to the first face image feature map; judging whether a target loss value corresponding to the model is greater than a preset loss threshold; and when the loss value is greater than a preset loss threshold value, modifying model parameters of the initial face image restoration model until the corresponding loss value is less than the preset loss threshold value, and obtaining a trained face image restoration model. According to the technical scheme, the quality of the face image is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for facial image restoration and model training. Background Art

[0002] The driving technology of digital human models based on images or videos has become one of the most popular directions in the field of artificial intelligence. Users only need to upload a video or an image, and they can use audio as a driving source to generate images corresponding to facial expressions and lip movements that match the audio content. Digital human driving technology is widely used in live streaming, distance learning, online meetings and other fields.

[0003] In the related technology, a face image and corresponding audio information are input into a face drive model, and the face drive model outputs multiple frames of face images, where the mouth shapes of the face regions in the multiple frames correspond to the audio information. With the performance improvement of terminal smart chip devices and the continuous updating of model compression technology, deploying face drive models on high-performance chips has become a new trend.

[0004] However, the computing power of high-performance chips is limited, resulting in low-quality generated facial images, which may appear blurry, jagged, and other low-quality issues. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides a facial image restoration processing and model training method, device, equipment and medium.

[0006] The present disclosure provides a method for training a face image restoration model, comprising the following steps:

[0007] Constructing an initial face image restoration model, wherein the initial face image restoration model includes a preset number of series-connected densely connected residual blocks and a restoration image output module, wherein the restoration image output module is connected to the preset number of series-connected densely connected residual blocks; obtaining training sample pairs, wherein each of the training sample pairs includes a first sample face image and a second sample face image, wherein the image quality of the second sample face image is higher than that of the first sample face image; extracting a first face image feature map of the first sample face image through the preset number of series-connected densely connected residual blocks, wherein the input of the first densely connected residual block in the preset number of series-connected densely connected residual blocks is the input of the first densely connected residual block. The features obtained after convolution calculation of a face image are the features output by the previous densely connected residual block, the input of the non-first densely connected residual block in the preset number of series-connected densely connected residual blocks is the features output by the previous densely connected residual block; the feature extraction processing is performed on the feature map of the first face image through the repaired image output module, and the extracted features are decoded to obtain and output the actual repaired image; the target loss value of the actual repaired image and the second sample face image is calculated, and it is determined whether the target loss value is greater than a preset loss threshold; when the loss value is greater than the preset loss threshold, the model parameters of the initial face image repair model are modified until the corresponding loss value is less than the preset loss threshold, and the trained face image repair model is obtained.

[0008] In order to implement the above-mentioned embodiments, the present disclosure also proposes a method for repairing a facial image, comprising: obtaining a current facial image and current audio information, and inputting the current facial image and current audio information into a preset facial drive model; obtaining a current reference facial image sequence output by the preset facial drive model, wherein the current reference facial image sequence includes multiple frames of third reference facial images, and the mouth shapes in the multiple frames of third reference facial images correspond to the current audio information; inputting each frame of the third reference facial image into a pre-trained facial image repair model, and obtaining a repaired fourth reference facial image output by the facial image repair model, wherein the facial image repair model is obtained according to the training method of the facial image repair model as described above.

[0009] In order to implement the above embodiment, the present disclosure also proposes a training device for a face image restoration model, comprising: a construction module, used to construct an initial face image restoration model, wherein the initial face image restoration model includes a preset number of series-connected densely connected residual blocks and a restoration image output module, wherein the restoration image output module is connected to the preset number of series-connected densely connected residual blocks; a first acquisition module, used to acquire training sample pairs, wherein each of the training sample pairs includes a first sample face image and a second sample face image, wherein the image quality of the second sample face image is higher than that of the first sample face image; a second acquisition module, used to extract a first face image feature map of the first sample face image through the preset number of series-connected densely connected residual blocks, wherein the first densely connected residual block in the preset number of series-connected densely connected residual blocks is a first sample face image feature map of the first sample face image. The input of the densely connected residual block is the feature obtained after the convolution calculation of the first sample face image, and the input of the non-first densely connected residual block in the preset number of series-connected densely connected residual blocks is the feature output by the previous densely connected residual block; a repaired image acquisition module is used to perform feature extraction processing on the feature map of the first face image through the repaired image output module, and decode the extracted features to obtain and output the actual repaired image; a loss calculation module is used to calculate the target loss value of the actual repaired image and the second sample face image; a judgment module is used to judge whether the target loss value is greater than a preset loss threshold; a training module is used to modify the model parameters of the initial face image repair model when the loss value is greater than the preset loss threshold until the corresponding loss value is less than the preset loss threshold, and then obtain the trained face image repair model.

[0010] The disclosed embodiment also provides a facial image restoration device, the device comprising: a third acquisition module, used to acquire a current facial image and current audio information, and input the current facial image and current audio information into a preset facial drive model; a fourth acquisition module, used to acquire a current reference facial image sequence output by the preset facial drive model, wherein the current reference facial image sequence includes multiple frames of third reference facial images, and the mouth shapes in the multiple frames of third reference facial images correspond to the current audio information; a restoration processing module, used to input each frame of the third reference facial image into a pre-trained facial image restoration model, and acquire a restored fourth reference facial image output by the facial image restoration model, wherein the facial image restoration model is obtained according to the above facial image restoration model training method.

[0011] An embodiment of the present disclosure also provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement a training method for a face image restoration model or a face image restoration method as provided in an embodiment of the present disclosure.

[0012] The embodiments of the present disclosure also provide a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the training method of the face image restoration model or the face image restoration method provided in the embodiments of the present disclosure.

[0013] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0014] The technical solution provided by the embodiment of the present disclosure constructs an initial face image restoration model, wherein the initial face image restoration model includes a preset number of densely connected residual blocks connected in series and a restoration image output module, wherein the restoration image output module is connected to a preset number of densely connected residual blocks connected in series, and a training sample pair is obtained, wherein each training sample pair includes a first sample face image and a second sample face image, wherein the image quality of the second sample face image is higher than that of the first sample face image, and then, a first face image feature map of the first sample face image is extracted through a preset number of densely connected residual blocks connected in series, wherein the first densely connected residual block in the preset number of densely connected residual blocks connected in series is The input of the block is the feature obtained after convolution calculation of the first sample face image, and the input of the non-first densely connected residual block in the preset number of series-connected densely connected residual blocks is the feature output by the previous densely connected residual block. The feature map of the first face image is subjected to feature extraction processing through the repaired image output module, and the extracted features are decoded to obtain and output the actual repaired image, and the target loss value of the actual repaired image and the second sample face image is calculated, and it is determined whether the target loss value is greater than the preset loss threshold. When the loss value is greater than the preset loss threshold, the model parameters of the initial face image repair model are modified until the corresponding loss value is less than the preset loss threshold, and the trained face image repair model is obtained. In the present technical scheme, when training a face image restoration model, dense residual blocks are used to extract face image features of relevant sample face images in training sample pairs, and based on the face image features, an actual restoration image of the sample image is obtained. The face image restoration model is trained in a closed loop based on the image loss value of the actual restoration image and the standard sample image corresponding to the relevant sample face image. This ensures that the trained face image restoration model can extract super-resolution face image features of face images, improves the quality of face images output by the face image restoration model, and the number of dense residual blocks is a preset number, which meets the limitation on the computing power consumption of the face image restoration model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0016] Figure 1 A schematic diagram of face restoration interaction provided by an embodiment of the present disclosure;

[0017] Figure 2 A flowchart of a method for training a face image restoration model provided by an embodiment of the present disclosure;

[0018] Figure 3 A schematic diagram of the structure of a residual dense module provided in an embodiment of the present disclosure;

[0019] Figure 4 A schematic diagram of the structure of a face image restoration model provided by an embodiment of the present disclosure;

[0020] Figure 5 A flowchart of another method for training a face image restoration model provided by an embodiment of the present disclosure;

[0021] Figure 6 A flowchart of a method for restoring a facial image provided by an embodiment of the present disclosure;

[0022] Figure 7 A schematic diagram of the structure of a face image restoration model device provided by an embodiment of the present disclosure;

[0023] Figure 8 A schematic diagram of the structure of a facial image restoration device provided by an embodiment of the present disclosure;

[0024] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0026] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0027] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0028] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0029] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0030] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0031] To solve the above problems, refer to Figure 1 The present disclosure introduces a lightweight face image restoration model, which restores the face image output by the face driving model to improve the quality of the face image, and then maps the restored face image to the corresponding character digital model, wherein the face driving model includes a preset number of series-connected densely connected residual blocks (Residual-in-Residual Dense Block, RRDB), each densely connected residual block includes multiple dense residual blocks, each dense residual block includes multiple convolutional layers, so that the preset number of series-connected densely connected residual blocks includes a large number of convolutional layers, which can realize super-resolution processing of face images and improve the quality of face images. In addition, the number of series-connected densely connected residual blocks is pre-calibrated, which reduces the consumption of computing power.

[0032] The following first describes the training method of the face image restoration model of the embodiment of the present disclosure.

[0033] Figure 2The present invention provides a flow chart of a method for training a face image restoration model, which can be performed by a training device for a face image restoration model, wherein the device can be implemented by software and / or hardware, and can generally be integrated in an electronic device. Figure 2 As shown, the method includes:

[0034] Step 201, constructing an initial face image restoration model, wherein the initial face image restoration model includes a preset number of series-connected densely connected residual blocks and a restoration image output module, wherein the restoration image output module is connected to a preset number of series-connected densely connected residual blocks.

[0035] Among them, the preset number can be determined according to the chip capacity of the processor integrated with the face image repair model. In some possible embodiments, when the face image repair model is integrated in the processor in the vehicle, the preset number can be 12, etc. In some possible embodiments, the face image repair model can be integrated in the same processor as the facial drive model. Or, in some possible embodiments, in order to further reduce the computing power consumption of the processor where the facial drive model is located, the processor integrated with the face image repair model can be different from the processor integrated with the facial drive model, so that the low-quality face image output by the facial drive model is sent to the face image repair model of another processor to obtain the repaired face image.

[0036] In this embodiment, an initial face image restoration model is constructed, wherein the basic skeleton of the initial face image restoration model can be any neural network, or can be a combination of any multiple neural networks, for example, it can be a convolutional neural network (CNN), etc. The basic module of the initial face image restoration model is a series of residual dense modules, wherein the initial face image restoration model also includes a restoration image output module, wherein the restoration image output module is connected to a preset number of series of densely connected residual blocks.

[0037] Among them, in some possible embodiments, such as Figure 3As shown in the figure, the residual dense module includes three densely connected residual blocks RDB (Residual Density Block, RDB), where RDB uses a multi-layer network structure for feature extraction. Each RDB module contains multiple convolutional layers Conv and activation layers ReLU. RDB concatenates multiple layers of features (Concat layer), and finally undergoes a 1*1 convolution operation (1*1Conv) to fuse the features of multiple channels to achieve dimensionality reduction fusion of the features. Finally, the initial input features (if the current RDB is not the first RDB, the input features are the features output by the previous RDB. If the current RDB is the first RDB, the input features are the features output by the upper network layer of RRDB in the initial face restoration model) and the fused features are fused to obtain the features output by the current RDB.

[0038] In some possible embodiments, in order to further reduce the computing power consumption of the densely connected residual block, the number of RDBs included in the densely connected residual block can be reduced, for example, the number of RDBs included in the densely connected residual block can be changed to 2.

[0039] In this embodiment, the number of convolution kernels of each convolution layer in the RDB module can also be modified. For example, the number of convolution kernels of each layer can be reduced to a preset percentage of the initial convolution kernel (the preset percentage is less than 1), thereby reducing the computing power consumption of the densely connected residual block.

[0040] Step 202: Acquire training sample pairs, wherein each training sample pair includes a first sample face image and a second sample face image, wherein the image quality of the second sample face image is higher than that of the first sample face image.

[0041] The training sample pair in this implementation may include a first sample face image and a second sample face image, wherein the first sample face image and the second sample face image are essentially the same face image, and the image content (including lip shape) of the two are the same, and the difference is that the image quality of the second sample face image is higher than that of the first sample face image.

[0042] It should be noted that in different application scenarios, the methods for obtaining training sample pairs are different. The examples are as follows:

[0043] In some possible examples, the original facial image and audio information are input into a preset facial drive model to generate a reference facial image sequence, wherein the reference image sequence includes multiple frames of first reference facial images, and the mouth shape in the first reference facial image corresponds to the audio information. As mentioned in the above embodiments, the quality of the first reference facial image output by the preset facial drive model is not high. Therefore, in this embodiment, the first reference facial image can be directly determined as the first sample image. For example, some or all of the multiple first reference facial images can be selected as the first sample image.

[0044] Furthermore, based on the first preset super-resolution algorithm, super-resolution calculation is performed on the first sample face image to obtain a second sample face image, wherein the first sample face image and the second sample face image constitute a training sample pair, wherein the first preset super-resolution algorithm can be any algorithm that can improve image quality, including but not limited to nearest neighbor interpolation, bilinear interpolation and bicubic interpolation.

[0045] In this embodiment, when the original facial image is a high-definition facial image of a certain person, multiple high-definition facial images corresponding to the certain person can be obtained, a silent note (including a note, etc.) is obtained, and each silent note and the high-definition facial image are input into a preset facial drive model to obtain corresponding multiple first sample facial images. Since the first sample facial image is an object synthesized after inputting the preset facial drive model, the image quality of the first sample facial image is relatively low. The multiple first sample facial images and the corresponding multiple high-definition facial images are used as second sample facial images.

[0046] The training sample pairs in this embodiment are derived from the face images actually output by the preset face driving model. Therefore, the face image restoration model trained based on the training sample pairs is more in line with the actual application scenario and has higher reliability.

[0047] In some possible examples, a second reference face image is extracted from a preset face image dataset, wherein the preset face image dataset includes a large number of real person face images, and then, super-resolution is performed on the second reference face image based on a second preset super-resolution algorithm to obtain a second sample face image, wherein the second preset super-resolution algorithm may be any algorithm that can improve image quality, including but not limited to nearest neighbor interpolation, bilinear interpolation, and bicubic interpolation.

[0048] In this example, the second reference face image corresponding to the second sample face image is not directly used as the first sample image. This is because the object repaired by the face image repair model is the processed face image output by the preset face driving model, not the real face image. Therefore, in this embodiment, the second sample face image is calculated according to a plurality of preset high-order image degradation algorithms to obtain the first sample face image, wherein the first sample face image and the second sample face image constitute a training sample pair, wherein the plurality of preset high-order image degradation algorithms include multiple of Gaussian blur, downsampling algorithm, and JPEG degradation.

[0049] In this embodiment, on the one hand, a plurality of preset high-order image degradation algorithms are used to obtain the first sample face image, which greatly expands the data domain distribution of the low-quality data set and can cover the data domain of most low-quality images, thereby improving the generalization ability of the subsequent training face image restoration model; on the other hand, the first sample face image is calculated based on the second sample face image, so the matching of the sample pair is guaranteed, and because the first sample image comes from the degradation calculation of the second sample face image, rather than the real face image, the face image restoration model trained based on the training sample pair is more in line with the real application scenario and has higher reliability. Step 203, extracting the first face image feature map of the first sample face image through a preset number of densely connected residual blocks connected in series, wherein the input of the first densely connected residual block in the preset number of densely connected residual blocks connected in series is the feature obtained after the convolution calculation of the first sample face image, and the input of the non-first densely connected residual block in the preset number of densely connected residual blocks connected in series is the feature output by the previous densely connected residual block.

[0050] Step 204: perform feature extraction processing on the first facial image feature map through the restored image output module, and decode the extracted features to obtain and output the actual restored image.

[0051] After obtaining the training sample pair, the first sample face image is input into the initial face image restoration model, and the first face image feature map of the first sample face image is extracted through a preset number of densely connected residual blocks connected in series, wherein the input of the first densely connected residual block in the preset number of densely connected residual blocks connected in series is the feature obtained after convolution calculation of the first sample face image, and the input of the non-first densely connected residual block in the preset number of densely connected residual blocks connected in series is the feature output by the previous densely connected residual block.

[0052] In some possible embodiments, in order to balance the computing power consumption of the processor and the image quality of the actual repaired image, refer to Figure 4The restored image output module includes an upsampling layer, a first convolution module, a second convolution module and a decoding layer, wherein the input end of the first convolution module is connected to the output end of the series-connected densely connected residual block, the output end of the first convolution module is connected to the input end of the upsampling layer, and the output end of the upsampling layer is connected to the second convolution module.

[0053] Among them, the first convolution module and the second convolution module may both include one or more convolution layers, wherein, regardless of including one or more convolution layers, in order to avoid exceeding the computing power limit of the processor, in some possible examples, the number of each convolution layer may be less than a preset convolution kernel threshold, wherein the preset convolution kernel threshold may be determined according to the computing power capability of the processor.

[0054] In this example, feature extraction is performed on the first face image feature map through the repaired image output module, and the extracted features are decoded to obtain and output the actual repaired image, including: convolution calculation is performed on the first face image feature map through the first convolution module to obtain a second face image feature map, that is, further feature extraction is performed on the first face image features to further optimize the image quality. In this implementation, in order to further optimize the image details, the second face image feature map is also upsampled according to the upsampling layer to obtain a third face image feature map, wherein the sampling resolution of the upsampling layer can be set according to the scene requirements.

[0055] Furthermore, in order to perform convolution calculation on the third facial image feature map through the second convolution module to obtain a fourth facial image feature map, wherein the number of channels of the second convolution module is less than a preset channel number threshold, wherein the channel number threshold can be set according to the algorithm consumption of the processor, in this embodiment, the upsampled third facial feature image is further processed to further optimize the image details, and the actual repaired image is obtained and output according to the fourth facial image feature map through the decoding layer, wherein the decoding layer is related to the basic skeleton of the initial facial image repair model.

[0056] in, Figure 4 The following is only one possible structure of the face image restoration model. Figure 4 , the model structure may also include other convolutional layers, etc. For example, in some possible scenarios, it may also include other convolutional modules connected to the input end of the series residual dense module, etc. Among them, other convolutional modules may include one or more convolutional layers, which can be set according to the needs of the scenario.

[0057] In some possible embodiments, the actual repaired image can also be obtained directly according to the first face image feature map output by the densely connected residual block. That is, the first face image feature layer is input into the network layer connected to the output end of the densely connected residual block in the initial face image repair model, and the network layer may include a pooling layer and an activation layer, etc., to obtain the output actual repaired image.

[0058] Step 205, calculate the target loss value of the actual restoration image and the second sample face image, and determine whether the target loss value is greater than a preset loss threshold.

[0059] In this embodiment, the target loss values ​​of the actual restored image and the second sample face image are calculated, and it is determined whether the target loss values ​​are greater than a preset loss threshold, wherein the preset loss threshold can be set according to the scene requirements.

[0060] In some possible embodiments, in order to further reduce computing power consumption, only the target loss value of the mouth area of ​​the actual repaired image and the second sample face image may be calculated.

[0061] Step 206, when the loss value is greater than the preset loss threshold, the model parameters of the initial face image restoration model are modified until the corresponding loss value is less than the preset loss threshold, and the trained face image restoration model is obtained.

[0062] When the loss value is greater than a preset loss threshold, the model parameters of the initial face image restoration model are modified until the corresponding loss value is less than the preset loss threshold, and the trained face image restoration model is obtained, wherein the model parameters include but are not limited to the model parameters of the densely connected residual block, etc.

[0063] In summary, the training method of the face image restoration model of the embodiment of the present disclosure constructs an initial face image restoration model, wherein the initial face image restoration model includes a preset number of series-connected densely connected residual blocks and a restoration image output module, wherein the restoration image output module is connected to a preset number of series-connected densely connected residual blocks, and a training sample pair is obtained, wherein each training sample pair includes a first sample face image and a second sample face image, wherein the image quality of the second sample face image is higher than that of the first sample face image, and then, a first face image feature map of the first sample face image is extracted through a preset number of series-connected densely connected residual blocks, wherein the first densely connected residual block in the preset number of series-connected densely connected residual blocks is a first sample face image feature map of the first sample face image. The input of the densely connected residual block is the feature obtained after convolution calculation of the first sample face image, and the input of the non-first densely connected residual block in the preset number of series-connected densely connected residual blocks is the feature output by the previous densely connected residual block. The feature map of the first face image is subjected to feature extraction processing through the repaired image output module, and the extracted features are decoded to obtain and output the actual repaired image, and the target loss value of the actual repaired image and the second sample face image is calculated, and it is determined whether the target loss value is greater than the preset loss threshold. When the loss value is greater than the preset loss threshold, the model parameters of the initial face image repair model are modified until the corresponding loss value is less than the preset loss threshold, and the trained face image repair model is obtained. In the present technical scheme, when training a face image restoration model, a dense residual block is used to extract face image features of a relevant sample face image in a training sample pair, and an actual restoration image of the sample image is obtained based on the face image features. The face image restoration model is trained in a closed loop based on the image loss value of the standard sample image corresponding to the actual restoration image and the relevant sample face image, thereby ensuring that the trained face image restoration model can extract super-resolution face image features of the face image, thereby improving the quality of the face image output by the face image restoration model, and the number of dense residual blocks is a preset number, which satisfies the limitation on the computing power consumption of the face image restoration model. Based on the above embodiments, in different application scenarios, the algorithms for calculating the target loss value of the actual restoration image and the second sample face image are different, as shown below:

[0064] In some possible embodiments, Figure 5 As shown, the target loss value of the actual repaired image and the second sample face image is calculated, including:

[0065] Step 501, calculating target texture loss values ​​of the actual restored image and the second sample face image.

[0066] In this implementation, the texture loss value that captures texture information is used as a type of loss value, and the style loss value is considered to train the face image restoration model to avoid the style shift problem when the face image restoration model restores the image.

[0067] In a disclosed embodiment, the target texture loss value of the actual repaired image and the second sample face image can be calculated by a pre-trained computer vision group (Visual Geometry Group, VGG) model. In this embodiment, VGG includes at least one convolution layer. In this embodiment, the actual repaired image and the second sample face image are input into the pre-trained VGG model, and a first vector feature map corresponding to the actual repaired image obtained by convolution of each preset convolution layer is obtained, and a second vector feature map corresponding to the second sample face image obtained by convolution of each preset convolution layer is obtained.

[0068] Among them, the preset convolutional layer can be all the convolutional layers in VGG, and can also be part of the convolutional layers of VGG. When it is a part of the convolutional layer of VGG, the partial convolutional layers can be continuous or discontinuous.

[0069] The first vector feature map can be regarded as a style representation of the actual restored image, and the second vector feature map can be regarded as a style representation of the second sample face image. In some possible embodiments, the first vector feature map and the second vector feature map can be calculated by a Gram matrix. In this embodiment, the first vector feature map can be expressed as the following formula (1):

[0070]

[0071] in, is the first vector feature map, i and j are the row and column of the feature points in the l-layer preset convolution layer, F is the corresponding feature vector, and k is the size of the feature map in the l-layer preset convolution layer. In this embodiment, the algorithm for the second vector feature map can also be obtained by calculating the inner product of the vectorized features of the second sample image in the l-layer preset convolution layer through the Gram matrix.

[0072] In this implementation, the first vector feature map and the second vector feature map are calculated according to a preset texture loss function to obtain a candidate texture loss value for each preset convolution layer. In some possible embodiments, the preset texture loss function may refer to the following formula (2), where E l is the candidate texture loss value, l is the number of preset convolutional layers in the VGG model, N is the number of channels in the l-layer preset convolutional layer, M is the convolution channel size of the l-layer preset convolutional layer, i and j are the rows and columns of feature points in the l-layer preset convolutional layer, respectively. and They are the first vector feature map and the second vector feature map of the l-layer preset convolutional layer respectively:

[0073]

[0074] Furthermore, according to at least one candidate texture loss value corresponding to at least one preset convolutional layer, a target texture loss value of the actual restored image and the second sample face image is calculated.

[0075] In some possible embodiments, the target texture loss value may be calculated based on the following formula (3), where a is the second sample image, x is the actual repaired image, L is the maximum number of convolution layers of the preset convolution layer, and w l The preset weight value of the candidate texture loss value of the preset convolutional layer at layer l, is the target texture loss value.

[0076]

[0077] In some possible embodiments, the average of all candidate texture loss values ​​may be calculated and used as the target texture loss value.

[0078] Step 502: Calculate the classification loss value and content loss value of the actual restored image and the second sample face image.

[0079] The classification loss value may be calculated by any algorithm in the prior art for calculating the classification loss, including but not limited to the cross entropy loss algorithm, the mean square error loss algorithm, etc. The content loss value is used to measure the content loss value of the actual restoration image and the second sample face image. The content loss value may be calculated by any algorithm including but not limited to the Euclidean distance, the mean square error algorithm, etc.

[0080] Step 503: Calculate the texture loss value, the classification loss value, and the content loss value to obtain a target loss value.

[0081] In one embodiment of the present disclosure, the texture loss value, the classification loss value and the content loss value are calculated to obtain a target loss value. The target loss value takes into account three loss functions, which can better evaluate the loss values ​​of the actual restoration image and the second sample face image.

[0082] Among them, in different application scenarios, the methods of calculating the texture loss value, classification loss value and content loss value to obtain the target loss value are different, as shown in the following examples:

[0083] In some possible examples, a first product value of a classification loss value and a first preset coefficient may be calculated, a second product value of a content loss value and a second preset coefficient may be calculated, and a texture loss value, a first product value, and a second product value may be summed to obtain a target loss value. The first preset coefficient and the second preset coefficient may be set according to scene requirements, for example, the first preset coefficient and the second preset coefficient are both less than 1, thereby achieving a heavier consideration of the texture loss function, and further avoiding style shift of the trained face image restoration model.

[0084] In this example, the corresponding target loss value can be calculated by the following formula (4), where L loss is the target loss value, L percep is the texture loss value, the first preset coefficient is γ, the second preset coefficient is σ, L G is the classification loss value, and L1 is the content loss value:

[0085] L loss =L percep +γL G +σL1 formula (4)

[0086] In some possible embodiments, the average of the texture loss value, the classification loss value, and the content loss value can be directly calculated, and the average is used as the target loss value. That is, the target loss value is determined by equally considering the three loss functions. Of course, in other possible embodiments, the target loss value of the actual repaired image and the second sample face image can also be calculated based on any loss function in the prior art.

[0087] In summary, the training method of the face image restoration model in the embodiment of the present disclosure introduces texture loss in addition to classification loss and content loss, and trains the face image restoration model based on the corresponding loss value, thereby ensuring that the image obtained based on the face image restoration model does not undergo image style shift.

[0088] The following describes the method for restoring a facial image according to an embodiment of the present disclosure with reference to the embodiments.

[0089] Figure 6 A flowchart of a method for restoring a facial image provided by an embodiment of the present disclosure is shown in FIG. Figure 6 As shown, the method includes:

[0090] Step 601, obtain the current face image and the current audio information, and input the current face image and the current audio information into a preset facial drive model.

[0091] Step 602, obtaining a current reference facial image sequence output by a preset facial drive model, wherein the current reference facial image sequence includes multiple frames of third reference facial images, and the lip shapes in the multiple frames of the third reference facial images correspond to the current audio information.

[0092] In this embodiment, a current face image and current audio information are obtained, wherein the current face image may be a real face image, and the current audio information may be audio information in any form, including but not limited to music.

[0093] Among them, the preset facial driving technology is based on the driving technology of the digital human model, and can output the current reference facial image sequence based on the current facial image and the current audio information, wherein the current reference facial image sequence includes multiple frames of third reference facial images, and the mouth shapes in the multiple frames of third reference facial images correspond to the current audio information.

[0094] For example, if the input audio information is a piece of music "I love my hometown", then the output lip shape areas correspond to the audio of "I", "love", "me", "of", "home", and "hometown" respectively.

[0095] Step 603: input each frame of the third reference face image into a pre-trained face image restoration model to obtain a restored fourth reference face image output by the face image restoration model.

[0096] As mentioned in the above embodiment, the image quality of the third reference face image may not be high. Therefore, in one embodiment of the present disclosure, each frame of the third reference face image is input into the face image restoration model pre-trained based on the above embodiment to obtain the restored fourth reference face image output by the face image restoration model. The image quality of the fourth reference face image is higher than that of the third reference face image.

[0097] Furthermore, in one embodiment of the present disclosure, after obtaining the fourth reference facial image, a preset character digital model can be obtained. The character digital model can be a two-dimensional digital model, a three-dimensional digital model, etc. The character digital model can be a predefined default digital model, or a three-dimensional model generated based on multi-angle facial images of the character corresponding to the current facial image, etc.

[0098] In this embodiment, after the preset human digital model is acquired, the face region of the human digital model is identified, and then, mapping processing is performed on the fourth reference human face image in the face region.

[0099] In some possible embodiments, if the texture size of the character digital model is preset (the texture size may be the same as the size of the current face image, etc.), it is identified whether the fourth reference face image is consistent with the texture size. Usually, the fourth reference face image is an image after super-resolution processing and may have been up-sampled, etc. Therefore, the size of the fourth reference face image is larger than the texture size. Therefore, the fourth reference face image is down-sampled to the texture size, and the down-sampled fourth face image is mapped to the character digital model. Thus, the fourth reference face image after continuous mapping downsampling can achieve the visual effect of the task digital model voice broadcasting "audio information".

[0100] In summary, the facial image repair method of the disclosed embodiment obtains the current facial image and the current audio information, and inputs the current facial image and the current audio information into a preset facial drive model, and then obtains the current reference facial image sequence output by the preset facial drive model, wherein the current reference facial image sequence includes multiple frames of third reference facial images, and the lip shapes in the multiple frames of third reference facial images correspond to the current audio information, and each frame of the third reference facial image is input into a pre-trained facial image repair model to obtain a repaired fourth reference facial image output by the facial image repair model. In the present technical solution, the reference facial image output by the preset facial drive model is input into the facial image repair model to obtain the repaired facial image output by the facial image repair model. The facial image repair model is pre-trained to extract super-resolution facial image features, and a repaired image is obtained based on the super-resolution facial image features, thereby obtaining a higher quality facial image.

[0101] In order to implement the above embodiment, the present disclosure also proposes a training device for a face image restoration model.

[0102] Figure 7 This is a schematic diagram of the structure of a face image restoration model device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 7 As shown, the device includes: a construction module 710, a first acquisition module 720, a second acquisition module 730, a repaired image acquisition module 740, a loss calculation module 750, a judgment module 760, and a training module 770, wherein:

[0103] A construction module 710 is used to construct an initial face image restoration model, wherein the initial face image restoration model includes a preset number of series-connected densely connected residual blocks and a restoration image output module, wherein the restoration image output module is connected to the preset number of series-connected densely connected residual blocks;

[0104] A first acquisition module 720, configured to acquire training sample pairs, wherein each training sample pair includes a first sample face image and a second sample face image, wherein the second sample face image has a higher image quality than the first sample face image;

[0105] The second acquisition module 730 is used to extract a first facial image feature map of the first sample facial image through a preset number of series-connected densely connected residual blocks, wherein the input of the first densely connected residual block in the preset number of series-connected densely connected residual blocks is the feature obtained after convolution calculation of the first sample facial image, and the input of the non-first densely connected residual block in the preset number of series-connected densely connected residual blocks is the feature output of the previous densely connected residual block

[0106] The repaired image acquisition module 740 is used to perform feature extraction processing on the first face image feature map through the repaired image output module, and decode the extracted features to obtain and output the actual repaired image.

[0107] A loss calculation module 750, used to calculate target loss values ​​of the actual restored image and the second sample face image;

[0108] A determination module 760 is used to determine whether the target loss value is greater than a preset loss threshold;

[0109] The training module 770 is used to modify the model parameters of the initial face image restoration model when the loss value is greater than a preset loss threshold until the corresponding loss value is less than the preset loss threshold, thereby obtaining the trained face image restoration model.

[0110] The training device for the facial image restoration model provided in the embodiments of the present disclosure can execute the training method for the facial image restoration model provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0111] Figure 8 This is a schematic diagram of the structure of a facial image restoration device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. Figure 8 As shown, the device includes: a third acquisition module 810, a fourth acquisition module 820, and a repair processing module 830, wherein:

[0112] The third acquisition module 810 is used to acquire the current face image and the current audio information, and input the current face image and the current audio information into a preset facial drive model;

[0113] A fourth acquisition module 820 is used to acquire a current reference face image sequence output by a preset face drive model, wherein the current reference face image sequence includes a plurality of frames of third reference face images, and the lip shapes in the plurality of frames of the third reference face images correspond to the current audio information;

[0114] The repair processing module 830 is used to input each frame of the third reference face image into a pre-trained face image repair model to obtain a repaired fourth reference face image output by the face image repair model, wherein the face image repair model is obtained according to the training method of the face image repair model in the above embodiment.

[0115] The training device for the facial image restoration model provided in the embodiments of the present disclosure can execute the training method for the facial image restoration model provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0116] The facial image restoration device provided in the embodiments of the present disclosure can execute the facial image restoration method provided in any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0117] In order to implement the above embodiments, the present disclosure also proposes a computer program product, including a computer program / instruction, which, when executed by a processor, implements the training method of the face image restoration model in the above embodiments, or the face image restoration method.

[0118] In order to implement the above embodiments, the present disclosure also proposes a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the training method of the face image restoration model in the above embodiments, or the face image restoration method.

[0119] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure.

[0120] The following specific reference Fig. 9 , which shows a schematic diagram of the structure of an electronic device 900 suitable for implementing the embodiment of the present disclosure. The electronic device 900 in the embodiment of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig. 9 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0121] like Fig. 9 As shown, the electronic device 900 may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a memory 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0122] Typically, the following devices may be connected to the I / O interface 905: an input device 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a memory 908 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig. 9 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0123] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 909, or installed from the memory 908, or installed from the ROM 902. When the computer program is executed by the processor 901, the above-mentioned functions defined in the training method of the face image repair model or the face image repair method of the embodiment of the present disclosure are executed.

[0124] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0125] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0126] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0127] The electronic device may be written in one or more programming languages ​​or a combination thereof to write computer program code for performing the operations of the present disclosure, including but not limited to object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0128] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0129] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0130] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0131] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0132] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0133] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0134] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A training method for a face image restoration model, characterized in that: The following steps are involved: Constructing an initial face image restoration model, wherein the initial face image restoration model includes a preset number of densely connected residual blocks connected in series and a restoration image output module, wherein the restoration image output module is connected to the preset number of densely connected residual blocks connected in series; Acquire training sample pairs, wherein each of the training sample pairs includes a first sample face image and a second sample face image, wherein the second sample face image has higher image quality than the first sample face image; Extracting a first facial image feature map of the first sample facial image through the preset number of densely connected residual blocks connected in series, wherein the input of the first densely connected residual block among the preset number of densely connected residual blocks connected in series is a feature obtained after convolution calculation of the first sample facial image, and the input of the non-first densely connected residual blocks among the preset number of densely connected residual blocks connected in series is a feature output by a previous densely connected residual block; Performing feature extraction processing on the first face image feature map by the repaired image output module, and decoding the extracted features to obtain and output an actual repaired image; Calculating target loss values ​​of the actual restored image and the second sample face image, and determining whether the target loss values ​​are greater than a preset loss threshold; When the loss value is greater than a preset loss threshold, the model parameters of the initial face image restoration model are modified until the corresponding loss value is less than the preset loss threshold, thereby obtaining a trained face image restoration model.

2. The method according to claim 1, characterized in that The obtaining of training sample pairs comprises: Inputting the original facial image and the audio information into a preset facial drive model to generate a reference facial image sequence, wherein the reference facial image sequence includes a plurality of frames of first reference facial images, and the mouth shape in the first reference facial image corresponds to the audio information; Determining the first sample face image in the plurality of frames of first reference face images; The first sample face image is super-resolved based on a first preset super-resolution algorithm to obtain the second sample face image, wherein the first sample face image and the second sample face image constitute the training sample pair.

3. The method according to claim 1, characterized in that The obtaining of training sample pairs comprises: Extracting a second reference face image from a preset face image data set; Performing super-resolution calculation on the second reference face image based on a second preset super-resolution algorithm to obtain the second sample face image; The second sample face image is calculated according to a plurality of preset high-order image degradation algorithms to obtain the first sample face image, wherein the first sample face image and the second sample face image constitute the training sample pair.

4. The method according to claim 1, characterized in that The repaired image output module includes an upsampling layer, a first convolution module, a second convolution module and a decoding layer. The repaired image output module performs feature extraction processing on the first face image feature map, and decodes the extracted features to obtain and output an actual repaired image, including: Performing convolution calculation on the first facial image feature map through the first convolution module to obtain a second facial image feature map; Performing upsampling processing on the second facial image feature map according to the upsampling layer to obtain a third facial image feature map; Performing convolution calculation on the third facial image feature map through the second convolution module to obtain a fourth facial image feature map; The actual repaired image is acquired and outputted through the decoding layer according to the fourth facial image feature map.

5. The method according to any one of claims 1 to 4, characterized in that: The calculating target loss values ​​of the actual restored image and the second sample face image includes: Calculating target texture loss values ​​of the actual restored image and the second sample face image; Calculating classification loss values ​​and content loss values ​​of the actual restored image and the second sample face image; The texture loss value, the classification loss value and the content loss value are calculated to obtain the target loss value.

6. The method according to claim 5, characterized in that The calculating target texture loss values ​​of the actual restoration image and the second sample face image includes: Inputting the actual restored image and the second sample face image into a pre-trained VGG model, wherein the VGG model includes at least one preset convolutional layer; Obtaining a first vector feature map corresponding to the actual restored image obtained by convolution of each preset convolution layer, and obtaining a second vector feature map corresponding to the second sample face image obtained by convolution of each preset convolution layer; Calculate the first vector feature map and the second vector feature map according to a preset texture loss calculation function to obtain a candidate texture loss value of each preset convolution layer; According to at least one of the candidate texture loss values ​​corresponding to the at least one preset convolutional layer, target texture loss values ​​of the actual restored image and the second sample face image are calculated.

7. The method according to claim 5, characterized in that The calculating the texture loss value, the classification loss value and the content loss value to obtain the target loss value includes: Calculate a first product value of the classification loss value and a first preset coefficient; Calculate a second product value of the content loss value and a second preset coefficient; The texture loss value, the first product value, and the second product value are summed to obtain the target loss value.

8. The method according to claim 6, characterized in that The preset texture loss calculation function includes: Among them, E l is the candidate texture loss value, l is the number of preset convolutional layers in the VGG model, N is the number of channels of the l-th preset convolutional layer, M is the convolution channel size of the l-th preset convolutional layer, i and j are the rows and columns of feature points in the l-th preset convolutional layer, respectively. and They are the first vector feature map and the second vector feature map corresponding to the l-layer preset convolutional layer respectively.

9. A method for restoring a facial image, characterized in that: include: Acquire a current face image and current audio information, and input the current face image and current audio information into a preset face drive model; Acquire a current reference facial image sequence output by the preset facial drive model, wherein the current reference facial image sequence includes a plurality of frames of third reference facial images, and the lip shapes in the plurality of frames of third reference facial images correspond to the current audio information; Input each frame of the third reference facial image into a pre-trained facial image restoration model to obtain a restored fourth reference facial image output by the facial image restoration model, wherein the facial image restoration model is obtained according to the training method of the facial image restoration model as described in any one of claims 1 to 8 above.

10. The method according to claim 9, characterized in that Also includes: Acquire a preset character digital model, and identify a face area of ​​the character digital model; The fourth reference face image is mapped in the face area.

11. A training device for a face image restoration model, characterized in that: include: A construction module, used to construct an initial face image restoration model, wherein the initial face image restoration model includes a preset number of densely connected residual blocks connected in series and a restoration image output module, wherein the restoration image output module is connected to the preset number of densely connected residual blocks connected in series; A first acquisition module, configured to acquire training sample pairs, wherein each of the training sample pairs comprises a first sample face image and a second sample face image, wherein the second sample face image has a higher image quality than the first sample face image; A second acquisition module is used to extract a first facial image feature map of the first sample facial image through the preset number of densely connected residual blocks connected in series, wherein the input of the first densely connected residual block in the preset number of densely connected residual blocks connected in series is a feature obtained after convolution calculation of the first sample facial image, and the input of the non-first densely connected residual block in the preset number of densely connected residual blocks connected in series is a feature output by a previous densely connected residual block; A repaired image acquisition module, used to perform feature extraction processing on the first face image feature map through the repaired image output module, and decode the extracted features to obtain and output an actual repaired image; A loss calculation module, used for calculating target loss values ​​of the actual restoration image and the second sample face image; A judgment module, used to judge whether the target loss value is greater than a preset loss threshold; The training module is used to modify the model parameters of the initial face image restoration model when the loss value is greater than a preset loss threshold until the corresponding loss value is less than the preset loss threshold, thereby obtaining the trained face image restoration model.

12. A facial image restoration device, characterized in that: include: A third acquisition module, used to acquire a current face image and current audio information, and input the current face image and current audio information into a preset facial drive model; a fourth acquisition module, configured to acquire a current reference facial image sequence output by the preset facial drive model, wherein the current reference facial image sequence includes a plurality of frames of third reference facial images, and the lip shapes in the plurality of frames of third reference facial images correspond to the current audio information; A restoration processing module is used to input each frame of the third reference face image into a pre-trained face image restoration model to obtain a restored fourth reference face image output by the face image restoration model, wherein the face image restoration model is obtained according to the training method of the face image restoration model as described in any one of claims 1 to 8 above.

13. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the training method of the facial image restoration model described in any one of claims 1 to 8, or the facial image restoration method described in claim 9 or 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to execute the training method of the facial image restoration model described in any one of claims 1 to 8, or the facial image restoration method described in claim 9 or 10.

Images