Video processing method and device, electronic equipment and storage medium

Abstract

This disclosure relates to a video processing method, apparatus, electronic device, and computer-readable storage medium, belonging to the field of deep learning technology. The method includes: identifying and extracting the corresponding face image from each frame of the original video and inputting it into a face correction model; obtaining three-dimensional detail parameters of the face image through a first feature extraction network and a modeling network; obtaining three-dimensional global parameters of the face image through a second feature extraction network and a tracking network, wherein the neural network parameters in the first and second feature extraction networks share weights; inputting the three-dimensional detail parameters and the three-dimensional global parameters into a renderer to obtain a corrected face image, and obtaining a reconstructed video based on the corrected face image. This disclosure, by introducing three-dimensional prior data and rationally decomposing the three-dimensional information of the face into detail parameters and global parameters, can achieve real-time three-dimensional face correction technology, achieving efficient and realistic face correction results.

Description

Technical Field

[0001] This disclosure relates to the field of deep learning technology, and more specifically, to a video processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the development of the internet, applications of facial retouching technology are becoming increasingly common. However, since existing face-shaping technologies are primarily designed for two-dimensional images, they cannot robustly handle faces from different angles and with different expressions, and it is also difficult to achieve both real-time performance and stability. Therefore, these applications are generally used in shooting mode and rarely in video mode.

[0003] Therefore, there is an urgent need in this field for a method that can realize real-time three-dimensional face correction technology to improve the face correction effect in video mode.

[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The purpose of this disclosure is to provide a video processing method, apparatus, electronic device, and storage medium, which can at least to some extent realize three-dimensional real-time face correction technology and improve the face correction effect in video mode.

[0006] According to a first aspect of this disclosure, a video processing method is provided, comprising:

[0007] Identify and extract the corresponding face image from each frame of the original video, and input the corresponding face image from each frame of the original video into the pre-trained face correction model in sequence;

[0008] The three-dimensional detail parameters of the face image corresponding to each frame of the original video are obtained through the first feature extraction network and the modeling network in the face correction model.

[0009] The three-dimensional global parameters of the face image corresponding to each frame of the original video are obtained through the second feature extraction network and the tracking network in the face correction model, wherein the weights of the neural network parameters in the first feature extraction network and the second feature extraction network are shared.

[0010] The three-dimensional detail parameters and the three-dimensional global parameters are input into the renderer to obtain the corrected face image, and the original video is reconstructed after correction based on each frame of the corrected face image.

[0011] In one exemplary embodiment of this disclosure, obtaining the three-dimensional detail parameters of the face image corresponding to each frame of the original video through the first feature extraction network and the modeling network in the face correction model includes:

[0012] The face images corresponding to a predetermined number of specified frames in the original video are input into the first feature extraction network and the modeling network in the face correction model to obtain the three-dimensional detail parameters of the face images corresponding to the predetermined number of specified frames.

[0013] The three-dimensional detail parameters of the face image in each frame of the original video are determined based on the three-dimensional detail parameters of the face image corresponding to the specified number of frames.

[0014] In one exemplary embodiment of this disclosure, the three-dimensional detail parameters include the shape parameters and albedo of the face image, and the three-dimensional global parameters include the expression parameters, pose parameters, and lighting parameters of the face image. The step of inputting the three-dimensional detail parameters and the three-dimensional global parameters into a renderer to obtain the corrected face image includes:

[0015] The shape parameter and albedo in the three-dimensional detail parameters, and the expression parameter in the three-dimensional global parameters are input into the three-dimensional deformation model to obtain the three-dimensional parameters corresponding to the shape parameter, the albedo and the expression parameter, respectively.

[0016] Based on the three-dimensional parameters corresponding to the shape parameters, the albedo, and the expression parameters, as well as the pose parameters and the illumination parameters in the three-dimensional global parameters, the three-dimensional information corresponding to the face image is obtained;

[0017] The three-dimensional information corresponding to the face image is input into the renderer to obtain the corrected face image.

[0018] In one exemplary embodiment of this disclosure, obtaining the reconstructed video after correction of the original video based on each frame of the corrected face image includes:

[0019] The corrected face image of each frame is fused with the corresponding video background to obtain the reconstructed video image corresponding to each frame, and the original video is corrected and reconstructed based on the reconstructed video image of each frame.

[0020] In one exemplary embodiment of this disclosure, the training method of the face correction model includes:

[0021] Obtain a first training dataset and a second training dataset, and obtain a mixed training dataset based on the first training dataset and the second training dataset, wherein the first training dataset includes face images of a first category, and the second training dataset includes face images of a second category;

[0022] The training images in the hybrid training dataset are respectively input into the first feature extraction network and the second feature extraction network in the initial face correction model to obtain the corresponding first feature vector and second feature vector, wherein the weights of the neural network parameters in the first feature extraction network and the second feature extraction network are shared.

[0023] The first feature vector is input into the modeling network of the initial face correction model to obtain the three-dimensional detail sample parameters corresponding to the training image, and the second feature vector is input into the tracking network of the initial face correction model to obtain the three-dimensional global sample parameters corresponding to the training image. The three-dimensional detail sample parameters include the sample shape parameters and sample albedo of the face image, and the three-dimensional global sample parameters include the sample expression parameters, sample pose parameters, and sample illumination parameters of the face image.

[0024] Based on the three-dimensional detail sample parameters and the three-dimensional global sample parameters, the three-dimensional sample information corresponding to the training image is obtained, and the three-dimensional sample information is input into the renderer in the initial face correction model to obtain the two-dimensional output sample image corresponding to the training image.

[0025] Reconstruction loss and semantic loss are obtained based on the training images and the two-dimensional output sample images, and the training images and the two-dimensional output sample images are input into the discriminator to obtain the discriminator loss;

[0026] The target loss is obtained based on the discriminator loss, the reconstruction loss, and the semantic loss;

[0027] The neural network parameters in the initial face correction model are iteratively updated based on the target loss to obtain the trained face correction model.

[0028] In one exemplary embodiment of this disclosure, obtaining the 3D sample information corresponding to the training image based on the 3D detail sample parameters and the 3D global sample parameters includes:

[0029] The sample shape parameter and sample albedo in the three-dimensional detail sample parameters, and the sample expression parameter in the three-dimensional global sample parameters are input into the three-dimensional deformation model to obtain the three-dimensional sample parameters corresponding to the sample shape parameter, the sample albedo and the sample expression parameter, respectively.

[0030] Based on the three-dimensional sample parameters corresponding to the sample shape parameters, the sample albedo, and the sample expression parameters, as well as the sample pose parameters and the sample illumination parameters in the three-dimensional global sample parameters, the three-dimensional sample information corresponding to the training image is obtained.

[0031] In one exemplary embodiment of this disclosure, obtaining the reconstruction loss and semantic loss based on the training image and the two-dimensional output sample image includes:

[0032] The semantic loss is obtained based on the semantic feature vectors corresponding to the training images and the semantic feature vectors corresponding to the two-dimensional output sample images.

[0033] A two-dimensional output target image is obtained based on the two-dimensional output sample image and the face image mask, and a reconstruction loss is obtained based on the loss value of each pixel between the training image and the two-dimensional output target image.

[0034] According to a second aspect of this disclosure, a video processing apparatus is provided, comprising:

[0035] The face image acquisition module is configured to perform recognition and capture the corresponding face image in each frame of the original video, and input the corresponding face image in each frame of the original video into a pre-trained face correction model in sequence.

[0036] The detail parameter determination module is configured to execute the first feature extraction network and the modeling network in the face correction model to obtain the three-dimensional detail parameters of the face image corresponding to each frame of the original video.

[0037] The global parameter determination module is configured to execute the second feature extraction network and the tracking network in the face correction model to obtain the three-dimensional global parameters of the face image corresponding to each frame of the original video, wherein the weights of the neural network parameters in the first feature extraction network and the second feature extraction network are shared.

[0038] The reconstructed video generation module is configured to input the three-dimensional detail parameters and the three-dimensional global parameters into the renderer to obtain the corrected face image, and to obtain the reconstructed video after correcting the original video based on each frame of the corrected face image.

[0039] In one exemplary embodiment of this disclosure, the detailed parameter determination module includes:

[0040] The specified frame parameter determination unit is configured to input the face images corresponding to a preset number of specified frames in the original video into the first feature extraction network and the modeling network in the face correction model to obtain the three-dimensional detail parameters of the face images corresponding to the preset number of specified frames.

[0041] The detail parameter determination unit is configured to perform the determination of the three-dimensional detail parameters of the face image corresponding to each frame of the original video based on the three-dimensional detail parameters of the face image corresponding to the preset number of specified frames.

[0042] In one exemplary embodiment of this disclosure, the reconstructed video generation module includes:

[0043] The three-dimensional parameter calculation unit is configured to input the shape parameter and albedo in the three-dimensional detail parameters and the expression parameter in the three-dimensional global parameters into the three-dimensional deformation model, and obtain the three-dimensional parameters corresponding to the shape parameter, the albedo and the expression parameter respectively.

[0044] The three-dimensional information determination unit is configured to perform an operation to obtain three-dimensional information corresponding to the face image based on the three-dimensional parameters corresponding to the shape parameters, the albedo and the expression parameters, as well as the pose parameters and the illumination parameters in the three-dimensional global parameters.

[0045] The three-dimensional information input unit is configured to input the three-dimensional information corresponding to the face image into a renderer to obtain the corrected face image.

[0046] In one exemplary embodiment of this disclosure, the reconstructed video generation module further includes:

[0047] The video background fusion unit is configured to perform a fusion process on the corrected face image of each frame and the corresponding video background to obtain a reconstructed video image corresponding to each frame, and to obtain the original video after correction based on the reconstructed video image of each frame.

[0048] In one exemplary embodiment of this disclosure, the video processing apparatus further includes a face correction model training module, the face correction model training module comprising:

[0049] The training dataset acquisition unit is configured to acquire a first training dataset and a second training dataset, and obtain a mixed training dataset based on the first training dataset and the second training dataset, wherein the first training dataset includes face images of a first category, and the second training dataset includes face images of a second category.

[0050] The feature vector determination unit is configured to input the training images in the mixed training dataset into the first feature extraction network and the second feature extraction network in the initial face correction model to obtain the corresponding first feature vector and second feature vector, wherein the weights of the neural network parameters in the first feature extraction network and the second feature extraction network are shared.

[0051] The sample parameter determination unit is configured to input the first feature vector into the modeling network of the initial face correction model to obtain the three-dimensional detail sample parameters corresponding to the training image, and input the second feature vector into the tracking network of the initial face correction model to obtain the three-dimensional global sample parameters corresponding to the training image. The three-dimensional detail sample parameters include the sample shape parameters and sample albedo of the face image, and the three-dimensional global sample parameters include the sample expression parameters, sample pose parameters, and sample illumination parameters of the face image.

[0052] The output sample determination unit is configured to execute the process of obtaining the three-dimensional sample information corresponding to the training image based on the three-dimensional detail sample parameters and the three-dimensional global sample parameters, and inputting the three-dimensional sample information into the renderer in the initial face correction model to obtain the two-dimensional output sample image corresponding to the training image;

[0053] The loss function determination unit is configured to perform operations to obtain reconstruction loss and semantic loss based on the training image and the two-dimensional output sample image, and input the training image and the two-dimensional output sample image into the discriminator to obtain the discriminator loss;

[0054] The target loss determination unit is configured to perform a target loss determination based on the discriminator loss, the reconstruction loss, and the semantic loss;

[0055] The correction model iteration unit is configured to perform iterative updates on the neural network parameters in the initial face correction model based on the target loss to obtain the trained face correction model.

[0056] In one exemplary embodiment of this disclosure, the output image determining unit includes:

[0057] The three-dimensional sample parameter calculation unit is configured to input the sample shape parameter and the sample albedo in the three-dimensional detail sample parameters, and the sample expression parameter in the three-dimensional global sample parameters into the three-dimensional deformation model, and obtain the three-dimensional sample parameters corresponding to the sample shape parameter, the sample albedo and the sample expression parameter respectively.

[0058] The three-dimensional sample information determination unit is configured to perform operations to obtain the three-dimensional sample information corresponding to the training image based on the three-dimensional sample parameters corresponding to the sample shape parameters, the sample albedo and the sample expression parameters, as well as the sample pose parameters and the sample illumination parameters in the three-dimensional global sample parameters.

[0059] In one exemplary embodiment of this disclosure, the loss function determination unit includes:

[0060] The semantic loss calculation unit is configured to perform semantic loss calculation based on the semantic feature vector corresponding to the training image and the semantic feature vector corresponding to the two-dimensional output sample image;

[0061] The reconstruction loss calculation unit is configured to perform operations to obtain a two-dimensional output target image based on the two-dimensional output sample image and the face image mask, and to obtain a reconstruction loss based on the loss value of each pixel between the training image and the two-dimensional output target image.

[0062] According to a third aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the video processing method described in any of the preceding claims.

[0063] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the video processing method described in any of the preceding claims.

[0064] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the video processing method described in any one of the preceding claims.

[0065] The exemplary embodiments disclosed herein can have the following beneficial effects:

[0066] In the video processing method of this exemplary embodiment, the face image of each frame of the video is sequentially input into a pre-trained face correction model. Two different neural networks are used to obtain the three-dimensional detail parameters and three-dimensional global parameters of the face image, respectively. The rendered image is then input into a decision processor to obtain the corrected face image, and finally, the corrected reconstructed video is generated. This video processing method, in its exemplary embodiment, achieves three-dimensional real-time face correction technology by reasonably decomposing the three-dimensional information of the face into detail parameters and global parameters. This technology can robustly correct faces under different postures and expressions, achieving efficient and natural face correction results. Furthermore, through information separation and weight sharing, end-to-end real-time stability can be achieved. Moreover, by introducing three-dimensional prior data, two-dimensional correction is extended to three-dimensional correction, eliminating the need for expensive three-dimensional supervision data and enabling real-time stable correction of face images under different angles and expressions.

[0067] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. Attached Figure Description

[0068] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0069] Figure 1 A schematic diagram of a general framework for face correction according to a related embodiment of the present disclosure is shown;

[0070] Figure 2 A flowchart illustrating a video processing method according to an exemplary embodiment of the present disclosure is shown;

[0071] Figure 3 A flowchart illustrating the corrected face image according to an exemplary embodiment of this disclosure is shown;

[0072] Figure 4 A flowchart illustrating a training method for a face correction model according to an exemplary embodiment of the present disclosure is shown.

[0073] Figure 5 A schematic diagram of the process for determining the three-dimensional sample information corresponding to the training image in an exemplary embodiment of this disclosure is shown.

[0074] Figure 6 A schematic diagram of a face correction model according to a specific embodiment of the present disclosure is shown;

[0075] Figure 7 A block diagram of a video processing apparatus according to an exemplary embodiment of the present disclosure is shown;

[0076] Figure 8 A schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present disclosure is shown. Detailed Implementation

[0077] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0078] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein.

[0079] The following exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0080] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0081] With the development of the internet, there are more and more applications that enable portrait beautification technology. However, these applications are generally used in shooting mode and rarely in video mode.

[0082] In some related embodiments, it can be achieved through, for example Figure 1 The general framework for face correction shown implements face correction. The specific content of this general framework for face correction is as follows:

[0083] 1) Input the uncorrected input image into the generator to obtain the corrected output image. The reconstruction loss function prompts the generator to generate an image with the same detailed information as the input image; the semantic loss function prompts the generator to generate an image that visually represents the same person as the input image.

[0084] 2) Achieve facial enhancement through the discriminator loss function. Specifically, a batch of face images with good facial structure can be collected as training set A, and a batch of face images with poor facial structure can be collected as training set B. The discriminator aims to correctly distinguish training set A as real and training set B as fake, while the generator aims to confuse the discriminator and make it misjudge the generated images as real.

[0085] The face correction methods described in the above embodiments have the following drawbacks:

[0086] 1) The above-mentioned face correction methods cannot robustly handle face images under different angles and expressions because the face is three-dimensional, while the existing face-shaping technology is oriented towards 2D. Due to the loss of one dimension of control information, unreasonable visual phenomena such as curve distortion and abrupt transitions often appear under large angles and large expressions.

[0087] 2) Existing face-shaping technology struggles to achieve real-time stability and smooth operation because it is typically implemented using GANs (Generated Adversarial Networks). This method is computationally intensive and is generally only used on high-end machines or servers. It often fails to achieve real-time stability on mobile devices.

[0088] Based on the above problems, this example implementation first provides a video processing method. (Reference) Figure 2 As shown, the above video processing method may include the following steps:

[0089] Step S210. Identify and extract the corresponding face image in each frame of the original video, and input the corresponding face image in each frame of the original video into the pre-trained face correction model in sequence.

[0090] Step S220. Obtain the three-dimensional detail parameters of the face image corresponding to each frame of the original video through the first feature extraction network and the modeling network in the face correction model.

[0091] Step S230. Obtain the three-dimensional global parameters of the face image corresponding to each frame of the original video through the second feature extraction network and the tracking network in the face correction model.

[0092] Step S240. Input the 3D detail parameters and 3D global parameters into the renderer to obtain the corrected face image, and obtain the reconstructed video after correcting the original video based on each frame of the corrected face image.

[0093] In the video processing method of this exemplary embodiment, the face image of each frame of the video is sequentially input into a pre-trained face correction model. Two different neural networks are used to obtain the three-dimensional detail parameters and three-dimensional global parameters of the face image, respectively. The rendered image is then input into a decision processor to obtain the corrected face image, and finally, the corrected reconstructed video is generated. This video processing method, in its exemplary embodiment, achieves three-dimensional real-time face correction technology by reasonably decomposing the three-dimensional information of the face into detail parameters and global parameters. This technology can robustly correct faces under different postures and expressions, achieving efficient and natural face correction results. Furthermore, through information separation and weight sharing, end-to-end real-time stability can be achieved. Moreover, by introducing three-dimensional prior data, two-dimensional correction is extended to three-dimensional correction, eliminating the need for expensive three-dimensional supervision data and enabling real-time stable correction of face images under different angles and expressions.

[0094] Below, in conjunction with Figures 3 to 5 The steps described above in this example implementation will be explained in more detail.

[0095] In step S210, the corresponding face image in each frame of the original video is identified and extracted, and the corresponding face image in each frame of the original video is sequentially input into the pre-trained face correction model.

[0096] In this example implementation, the original video refers to the unprocessed original video captured or uploaded by the user. First, facial recognition technology can be used to identify and extract the corresponding facial image from each frame of the original video. Then, the facial image corresponding to each frame is sequentially input into a pre-trained facial correction model to perform real-time facial correction on each frame.

[0097] In this example implementation, the face correction model may include a feature extraction network, a modeling network, a tracking network, and a discriminator. The aforementioned neural networks are essentially parameter regression CNNs (Convolutional Neural Networks), such as ResNet (Residual Network), which can regress a one-dimensional vector from an image.

[0098] In step S220, the three-dimensional detail parameters of the face image corresponding to each frame of the original video are obtained through the first feature extraction network and the modeling network in the face correction model.

[0099] In this example implementation, the first feature extraction network and the modeling network in the face correction model can be used to extract three-dimensional detail parameters of the face image. These three-dimensional detail parameters represent high-frequency detail information of the face image and may include shape parameters and albedo. Shape parameters refer to the shape of the 3D vertices of the face image, and albedo refers to the color values ​​of the vertices, such as RGB (color mode).

[0100] For the same person in a video, the shape and texture of the face are generally constant. Therefore, the 3D detail parameters of the face image, such as shape parameters and albedo, can be obtained from just one or a few frames of the video that contain the face or where the face is clear, without needing to calculate them for every single frame of the video. Specifically, the face images corresponding to a predetermined number of frames in the original video can be input into the first feature extraction network and the modeling network in the face correction model to obtain the 3D detail parameters of the face images corresponding to the predetermined number of frames. Based on the 3D detail parameters of the face images corresponding to the predetermined number of frames, the 3D detail parameters of the face image corresponding to each frame of the original video can be determined.

[0101] The aforementioned preset number of specified frames can refer to one or more frames in the video, such as the first frame, the Nth frame, or the first few frames. To ensure real-time video correction, the 3D detail parameters of the face image can generally be calculated directly from the first frame of the video. These calculated parameters can then be used directly in all subsequent video frames without recalculation, significantly improving computational efficiency. If the first frame does not contain a face or does not have a clear face, the Nth frame, which first contains a face or has a clear face, can be selected.

[0102] If the 3D detail parameters obtained from only the first or Nth frame of the video are deemed insufficiently accurate, multiple specified frames of face images can be input into the first feature extraction network and the modeling network of the face correction model to obtain the 3D detail parameters of the face images corresponding to the specified frames. Based on these 3D detail parameters, the 3D detail parameters of the face image corresponding to each frame of the original video can then be determined. For example, to ensure real-time video correction, several different sets of 3D detail parameters can be obtained from the first few specified frames of the video, and then the average value can be calculated according to certain weights to obtain the final 3D detail parameters.

[0103] In step S230, the three-dimensional global parameters of the face image corresponding to each frame of the original video are obtained through the second feature extraction network and the tracking network in the face correction model.

[0104] In this example implementation, the second feature extraction network and the tracking network in the face correction model can be used to extract three-dimensional global parameters of the face image. These three-dimensional global parameters represent low-frequency global information of the face image and may include facial expression parameters, pose parameters, and illumination parameters. Facial expression parameters refer to the facial expression coefficients, pose parameters refer to the displacement and scaling parameters of the 3D vertex coordinates of the face image, and illumination parameters may include quantization parameters for illumination intensity and direction, etc.

[0105] For each frame of the video, the pose, expression, and lighting of the face are changing. Therefore, the face image corresponding to each frame of the video needs to be processed by a second feature extraction network and a tracking network to obtain the corresponding three-dimensional global parameters.

[0106] Furthermore, in this example embodiment, the neural network parameters in the first and second feature extraction networks share weights. Through information separation and weight sharing, end-to-end real-time stability can be achieved, and during model training, the joint backpropagation update of the two parameter information is more beneficial to the learning of the feature extraction network.

[0107] In step S240, the three-dimensional detail parameters and three-dimensional global parameters are input into the renderer to obtain the corrected face image, and the original video is reconstructed based on each frame of the corrected face image.

[0108] In this example implementation, the three-dimensional detail parameters and three-dimensional global parameters obtained through the above steps are processed by a renderer, and the rendered two-dimensional image is the corrected face image.

[0109] In this example implementation, as Figure 3 As shown, inputting the 3D detail parameters and 3D global parameters into the renderer to obtain the corrected face image can specifically include the following steps:

[0110] Step S310. Input the shape parameter and albedo in the three-dimensional detail parameters, and the expression parameter in the three-dimensional global parameters into the three-dimensional deformation model to obtain the three-dimensional parameters corresponding to the shape parameter, albedo and expression parameter respectively.

[0111] In this example implementation, the shape parameters and albedo in the 3D detail parameters, as well as the expression parameters in the 3D global parameters, need to be processed by a 3D Morphable Model (3DMM) to obtain the corresponding 3D parameter information. The pose parameters and lighting parameters in the 3D global parameters do not need to be processed by the aforementioned 3D Morphable Model.

[0112] Step S320. Obtain the three-dimensional information corresponding to the face image based on the three-dimensional parameters corresponding to the shape parameters, albedo, and expression parameters, as well as the pose parameters and illumination parameters in the three-dimensional global parameters.

[0113] Based on the three-dimensional parameters obtained after processing the shape parameters, albedo, and expression parameters using 3DMM, as well as the pose and lighting parameters in the three-dimensional global parameters, the three-dimensional information corresponding to the face image can be obtained.

[0114] Step S330. Input the 3D information corresponding to the face image into the renderer to obtain the corrected face image.

[0115] Finally, the 3D information corresponding to the face image is input into the renderer, and the rendered 2D image is the corrected face image.

[0116] In this example implementation, by introducing three-dimensional prior data, two-dimensional correction is extended to three-dimensional correction. Without the need for expensive three-dimensional supervision data, real-time and stable correction effects can be achieved for facial images under different angles and expressions.

[0117] By fusing each frame of the corrected face image with the corresponding video background, the reconstructed video image corresponding to each frame can be obtained. Then, based on each frame of the reconstructed video image, the original video can be corrected and reconstructed.

[0118] In this example implementation, since the computational cost of the tracking network is much smaller than that of the modeling network, the processing speed is very fast except for the first frame of the video or the first few frames of a specified video, which may be slower, and the subsequent processing speed is very fast, achieving the effect of real-time correction.

[0119] In addition, this example embodiment of the video processing method also provides a training method for a face correction model, such as... Figure 4 As shown, the training method for this face correction model can specifically include the following steps:

[0120] Step S410. Obtain the first training dataset and the second training dataset, and obtain the mixed training dataset based on the first training dataset and the second training dataset.

[0121] In this example implementation, the first training dataset includes face images of a first category, and the second training dataset includes face images of a second category. The first category of face images refers to face images with good facial structure, and the second category of face images refers to face images with poor facial structure. The first and second training datasets are randomly mixed to obtain a mixed training dataset.

[0122] Taking real-time profile correction as an example, in the data preparation stage, a large number of face images with good profile structures, such as the nose tip, lip philtrum, and chin forming a straight line, can be prepared as the first training dataset, and face images with poor profile structures, such as the nose tip, lip philtrum, and chin forming a broken line, can be prepared as the second training dataset.

[0123] Step S420. Input the training images from the mixed training dataset into the first feature extraction network and the second feature extraction network in the initial face correction model to obtain the corresponding first feature vector and second feature vector.

[0124] In the model preparation stage, an initial face correction model can be built based on the feature extraction network, modeling network, tracking network, and discriminator. The feature extraction network, modeling network, tracking network, and discriminator are essentially parametric regression CNNs, which can regress a one-dimensional vector from the image.

[0125] In this example implementation, the neural network parameters in the first and second feature extraction networks share weights. This is because although the 3D detail parameters and 3D global parameters of a face image carry different information, they are both obtained through convolution of the input image and have certain commonalities.

[0126] Step S430. Input the first feature vector into the modeling network of the initial face correction model to obtain the three-dimensional detail sample parameters corresponding to the training image, and input the second feature vector into the tracking network of the initial face correction model to obtain the three-dimensional global sample parameters corresponding to the training image.

[0127] The feature vectors are input into the modeling network and the tracking network in the initial face correction model, respectively. The modeling network can obtain the three-dimensional detail sample parameters corresponding to the training image, such as sample shape parameters and sample albedo. The tracking network can obtain the three-dimensional global sample parameters corresponding to the training image, such as sample expression parameters, sample pose parameters and sample illumination parameters.

[0128] Shape parameters and albedo represent high-frequency detail information of a face image, while expression parameters, pose parameters, and illumination parameters represent low-frequency global information of a face image. The joint backpropagation update of both types of parameter information is more beneficial to the learning of the feature extraction network.

[0129] Step S440. Obtain the 3D sample information corresponding to the training image based on the 3D detail sample parameters and the 3D global sample parameters, and input the 3D sample information into the renderer in the initial face correction model to obtain the 2D output sample image corresponding to the training image.

[0130] In this example implementation, as Figure 5As shown, obtaining the 3D sample information corresponding to the training image based on the 3D detail sample parameters and the 3D global sample parameters can specifically include the following steps:

[0131] Step S510. Input the sample shape parameter and sample albedo from the 3D detail sample parameters, and the sample expression parameter from the 3D global sample parameters into the 3D deformation model to obtain the 3D sample parameters corresponding to the sample shape parameter, sample albedo and sample expression parameter respectively.

[0132] The sample shape parameters and sample albedo in the 3D detail sample parameters, as well as the sample expression parameters in the 3D global sample parameters, need to be processed by 3DMM (3D Morphable Model) to obtain the corresponding 3D sample parameters.

[0133] Step S520. Based on the three-dimensional sample parameters corresponding to the sample shape parameters, sample albedo, and sample expression parameters, as well as the sample pose parameters and sample illumination parameters in the three-dimensional global sample parameters, obtain the three-dimensional sample information corresponding to the training image.

[0134] Based on the three-dimensional sample parameters obtained after processing the sample shape parameters, sample albedo, and sample expression parameters through 3DMM, as well as the sample pose parameters and sample illumination parameters in the three-dimensional global sample parameters, the five three-dimensional sample parameters corresponding to the training image can be obtained.

[0135] By inputting the parameters of the five 3D samples corresponding to the training image into the renderer, a rendered 2D output sample image can be obtained.

[0136] Step S450. Obtain reconstruction loss and semantic loss based on training images and two-dimensional output sample images, and input training images and two-dimensional output sample images into the discriminator to obtain discriminator loss.

[0137] In this example implementation, the semantic loss can be obtained based on the semantic feature vectors corresponding to the training images and the semantic feature vectors corresponding to the two-dimensional output sample images. The semantic feature vector is a multi-dimensional feature vector obtained from the image; for example, it can be a 512-dimensional vector. The corresponding semantic loss can be obtained by calculating either L1 loss or L2 loss based on the semantic feature vectors corresponding to the training images and the two-dimensional output sample images. Here, L1 loss is the mean absolute error, and L2 loss is the mean squared error.

[0138] For reconstruction loss, a 2D output target image can be obtained from the 2D output sample image and the face image mask, and the reconstruction loss can be obtained based on the loss value at each pixel between the training image and the 2D output target image. Specifically, the L1 or L2 loss at each pixel between the training image and the 2D output target image can be calculated, and then the reconstruction loss can be obtained based on the loss value at each pixel.

[0139] Since GAN loss (discriminator loss) alters the shape of the face, pixel-level reconstruction loss requires a mask that is identical to the pixels used in the reconstruction loss. The goal is to make the upper half of the face as consistent as possible with the input image, while the lower half can differ slightly. This step is unnecessary for higher-level semantic losses. To achieve this, a circle can be drawn with the lips as the center and the length from the lips to the nose as the radius. A filtering algorithm can then be introduced to smooth the boundary transition, resulting in the face image mask corresponding to the reconstruction loss. Therefore, for the reconstruction loss, the 2D output sample image can be processed first using the face image mask, masking the lower half of the face and selecting the upper half as the region of interest, resulting in the processed 2D output target image. The reconstruction loss is then calculated based on this processed 2D output target image.

[0140] Simultaneously, the rendered 2D output sample images are fed into a discriminator. The discriminator aims to correctly distinguish between real and fake training images in the first training dataset and training images in the second training dataset. The 3D reconstruction network, composed of feature extraction, modeling, and tracking networks, aims to mislead the discriminator, causing it to misclassify the rendered 2D output sample images as real, thus achieving face correction. The discriminator loss can be obtained from the discriminator's output; for example, "0" can represent "fake" and "1" can represent "real." The discriminator loss can achieve an image beautification effect.

[0141] Step S460. Obtain the target loss based on the discriminator loss, reconstruction loss, and semantic loss.

[0142] The target loss of the initial face correction model can be obtained based on the discriminator loss, reconstruction loss, and semantic loss.

[0143] Step S470. Iteratively update the neural network parameters in the initial face correction model based on the target loss to obtain the trained face correction model.

[0144] In this example implementation, since the renderer is differentiable, the neural network parameters in the feature extraction network, modeling network, and tracking network can be updated through backpropagation of reconstruction loss, semantic loss, and discriminator loss, thereby enabling the training process of the face correction model.

[0145] The face correction model trained through the above steps can achieve real-time 3D face correction technology by reasonably decomposing the 3D information of the face into detailed parameters and global parameters. It can robustly correct faces under different postures and expressions, achieving efficient and realistic and natural face correction results.

[0146] like Figure 6 The diagram shown is a schematic representation of a face correction model according to a specific embodiment of this disclosure, illustrating a specific application scenario of the above steps in this example embodiment. Figure 6 In the face correction model shown, the two feature extraction networks share weights. The face image corresponding to each frame of the original video is used as the input image and sequentially fed into the face correction model. The shape parameters and albedo of the face image are obtained through the first feature extraction network and the modeling network in the face correction model. The expression parameters, pose parameters, and illumination parameters of the face image are obtained through the second feature extraction network and the tracking network in the face correction model. Then, the three-dimensional parameter information corresponding to the above five parameters is input into a differentiable renderer. The output image obtained by the renderer is the corrected face image. Finally, each frame of the corrected face image is fused with the corresponding video background to obtain the reconstructed video image corresponding to each frame. Then, based on each frame of the reconstructed video image, the reconstructed video after correction of the original video can be obtained.

[0147] In such Figure 6 The training process of the face correction model shown can be achieved by building an initial face correction model based on a feature extraction network, a modeling network, a tracking network, and a discriminator. The neural network parameters in the first and second feature extraction networks share weights. After building the initial face correction model, the first feature extraction network and the modeling network in the initial face correction model obtain the sample shape parameters and sample albedo corresponding to the training images. The second feature extraction network and the tracking network in the initial face correction model obtain the sample expression parameters, sample pose parameters, and sample illumination parameters corresponding to the training images. Then, the 3D sample information corresponding to the above five sample parameters is input into a differentiable renderer to obtain the rendered 2D output sample images. The reconstruction loss, semantic loss, and GAN loss (discriminator loss) of the model can be obtained from the 2D output sample images output by the renderer. The neural network parameters in the initial face correction model are iteratively updated using the discriminator loss, reconstruction loss, and semantic loss to obtain the trained face correction model.

[0148] The specific training method of the face correction model and the specific method of face correction using the face correction model have been described in detail above, and will not be repeated here.

[0149] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0150] Furthermore, this disclosure also provides a video processing apparatus. (See reference) Figure 7 As shown, the video processing apparatus may include a face image acquisition module 710, a detail parameter determination module 720, a global parameter determination module 730, and a reconstructed video generation module 740. Wherein:

[0151] The face image acquisition module 710 is configured to identify and capture the corresponding face image in each frame of the original video, and input the corresponding face image in each frame of the original video into the pre-trained face correction model in sequence.

[0152] The detail parameter determination module 720 is configured to obtain the three-dimensional detail parameters of the face image corresponding to each frame of the original video through the first feature extraction network and the modeling network in the face correction model;

[0153] The global parameter determination module 730 is configured to obtain the three-dimensional global parameters of the face image corresponding to each frame of the original video through the second feature extraction network and the tracking network in the face correction model, wherein the weights of the neural network parameters in the first feature extraction network and the second feature extraction network are shared.

[0154] The reconstructed video generation module 740 is configured to input three-dimensional detail parameters and three-dimensional global parameters into the renderer to obtain the corrected face image, and to obtain the reconstructed video after correcting the original video based on each frame of the corrected face image.

[0155] In some exemplary embodiments of this disclosure, the detail parameter determination module 720 may include a specified frame parameter determination unit and a detail parameter determination unit. Wherein:

[0156] The specified frame parameter determination unit is configured to input the face images corresponding to a preset number of specified frames in the original video into the first feature extraction network and the modeling network in the face correction model to obtain the three-dimensional detail parameters of the face images corresponding to the preset number of specified frames.

[0157] The detail parameter determination unit is configured to determine the three-dimensional detail parameters of the face image corresponding to each frame of the original video based on the three-dimensional detail parameters of the face image corresponding to a preset number of specified frames.

[0158] In some exemplary embodiments of this disclosure, the reconstructed video generation module 740 may include a three-dimensional parameter calculation unit, a three-dimensional information determination unit, and a three-dimensional information input unit. Wherein:

[0159] The three-dimensional parameter calculation unit is configured to input the shape parameter and albedo in the three-dimensional detail parameters, as well as the expression parameter in the three-dimensional global parameters, into the three-dimensional deformation model to obtain the three-dimensional parameters corresponding to the shape parameter, albedo, and expression parameter, respectively.

[0160] The three-dimensional information determination unit is configured to obtain the three-dimensional information corresponding to the face image based on the three-dimensional parameters corresponding to the shape parameters, albedo and expression parameters, as well as the pose parameters and illumination parameters in the three-dimensional global parameters.

[0161] The 3D information input unit is configured to input the 3D information corresponding to the face image into the renderer to obtain the corrected face image.

[0162] In some exemplary embodiments of this disclosure, the reconstructed video generation module 740 may further include a video background fusion unit, configured to fuse each frame of corrected face image with the corresponding video background to obtain a reconstructed video image corresponding to each frame, and to obtain the original video corrected reconstructed video based on each frame of reconstructed video image.

[0163] In some exemplary embodiments of this disclosure, a video processing apparatus provided by this disclosure may further include a face correction model training module. This face correction model training module may include a training dataset acquisition unit, a feature vector determination unit, a sample parameter determination unit, an output sample determination unit, a loss function determination unit, and a correction model iteration unit. Wherein:

[0164] The training dataset acquisition unit is configured to acquire a first training dataset and a second training dataset, and to obtain a mixed training dataset based on the first training dataset and the second training dataset, wherein the first training dataset includes face images of a first category and the second training dataset includes face images of a second category.

[0165] The feature vector determination unit is configured to input the training images in the mixed training dataset into the first feature extraction network and the second feature extraction network in the initial face correction model, respectively, to obtain the corresponding first feature vector and second feature vector, wherein the weights of the neural network parameters in the first feature extraction network and the second feature extraction network are shared.

[0166] The sample parameter determination unit is configured to input the first feature vector into the modeling network in the initial face correction model to obtain the three-dimensional detail sample parameters corresponding to the training image, and input the second feature vector into the tracking network in the initial face correction model to obtain the three-dimensional global sample parameters corresponding to the training image. The three-dimensional detail sample parameters include the sample shape parameters and sample albedo of the face image, and the three-dimensional global sample parameters include the sample expression parameters, sample pose parameters, and sample illumination parameters of the face image.

[0167] The output sample determination unit is configured to obtain the 3D sample information corresponding to the training image based on the 3D detail sample parameters and the 3D global sample parameters, and input the 3D sample information into the renderer in the initial face correction model to obtain the 2D output sample image corresponding to the training image.

[0168] The loss function determination unit is configured to obtain reconstruction loss and semantic loss based on training images and 2D output sample images, and input the training images and 2D output sample images into the discriminator to obtain the discriminator loss;

[0169] The target loss determination unit is configured to obtain the target loss based on the discriminator loss, reconstruction loss, and semantic loss;

[0170] The correction model iteration unit is configured to iteratively update the neural network parameters in the initial face correction model based on the target loss to obtain a trained face correction model.

[0171] In some exemplary embodiments of this disclosure, the output image determination unit may include a three-dimensional sample parameter calculation unit and a three-dimensional sample information determination unit. Wherein:

[0172] The three-dimensional sample parameter calculation unit is configured to input the sample shape parameter and sample albedo in the three-dimensional detail sample parameters, as well as the sample expression parameter in the three-dimensional global sample parameters, into the three-dimensional deformation model to obtain the three-dimensional sample parameters corresponding to the sample shape parameter, sample albedo, and sample expression parameter, respectively.

[0173] The three-dimensional sample information determination unit is configured to obtain the three-dimensional sample information corresponding to the training image based on the three-dimensional sample parameters corresponding to the sample shape parameters, sample albedo and sample expression parameters, as well as the sample pose parameters and sample illumination parameters in the three-dimensional global sample parameters.

[0174] In some exemplary embodiments of this disclosure, the loss function determination unit may include a semantic loss calculation unit and a reconstruction loss calculation unit. Wherein:

[0175] The semantic loss calculation unit is configured to obtain the semantic loss based on the semantic feature vectors corresponding to the training images and the semantic feature vectors corresponding to the two-dimensional output sample images.

[0176] The reconstruction loss calculation unit is configured to obtain a two-dimensional output target image based on the two-dimensional output sample image and the face image mask, and to obtain the reconstruction loss based on the loss value of each pixel between the training image and the two-dimensional output target image.

[0177] The specific details of each module / unit in the above video processing device have been described in detail in the corresponding method embodiment section, and will not be repeated here.

[0178] Figure 8 A schematic diagram of a computer system suitable for implementing embodiments of the present invention is shown.

[0179] It should be noted that, Figure 8 The computer system 800 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0180] like Figure 8 As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 802 or programs loaded from storage section 808 into random access memory (RAM) 803. The RAM 803 also stores various programs and data required for system operation. The CPU 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0181] The following components are connected to I / O interface 805: an input section 806 including a keyboard, mouse, etc.; an output section 807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 808 including a hard disk, etc.; and a communication section 809 including a network interface card such as a LAN card, modem, etc. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to I / O interface 805 as needed. A removable medium 811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 810 as needed so that computer programs read from it can be installed into storage section 808 as needed.

[0182] In particular, according to embodiments of the present invention, the processes described below with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 809, and / or installed from removable medium 811. When the computer program is executed by central processing unit (CPU) 801, it performs various functions defined in the system of this application.

[0183] It should be noted that the computer-readable medium disclosed herein may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0184] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0185] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0186] It should be noted that although several modules for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0187] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein.

[0188] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video processing method, characterized in that, include: Identify and extract the corresponding face image from each frame of the original video, and input the corresponding face image from each frame of the original video into the pre-trained face correction model in sequence; The three-dimensional detail parameters of the face image corresponding to each frame of the original video are obtained through the first feature extraction network and the modeling network in the face correction model. The three-dimensional detail parameters include the shape parameters and albedo of the face image. The three-dimensional global parameters of the face image corresponding to each frame of the original video are obtained through the second feature extraction network and the tracking network in the face correction model. The three-dimensional global parameters include the expression parameters, pose parameters and illumination parameters of the face image. The weights of the neural network parameters in the first feature extraction network and the second feature extraction network are shared. The three-dimensional detail parameters and the three-dimensional global parameters are input into the renderer to obtain the corrected face image, and the original video is reconstructed after correction based on each frame of the corrected face image. The step of inputting the three-dimensional detail parameters and the three-dimensional global parameters into the renderer to obtain the corrected face image includes: inputting the shape parameter and albedo from the three-dimensional detail parameters, and the expression parameter from the three-dimensional global parameters into a three-dimensional deformation model to obtain the three-dimensional parameters corresponding to the shape parameter, the albedo, and the expression parameter, respectively; obtaining the three-dimensional information corresponding to the face image based on the three-dimensional parameters corresponding to the shape parameter, the albedo, and the expression parameter, and the pose parameter and the lighting parameter from the three-dimensional global parameters; and inputting the three-dimensional information corresponding to the face image into the renderer to obtain the corrected face image.

2. The video processing method according to claim 1, characterized in that, The process involves obtaining the three-dimensional detail parameters of the face image corresponding to each frame of the original video through the first feature extraction network and the modeling network in the face correction model, including: The face images corresponding to a predetermined number of specified frames in the original video are input into the first feature extraction network and the modeling network in the face correction model to obtain the three-dimensional detail parameters of the face images corresponding to the predetermined number of specified frames. The three-dimensional detail parameters of the face image in each frame of the original video are determined based on the three-dimensional detail parameters of the face image corresponding to the specified number of frames.

3. The video processing method according to claim 1, characterized in that, The step of obtaining the reconstructed video after correction of the original video based on each frame of the corrected face image includes: The corrected face image of each frame is fused with the corresponding video background to obtain the reconstructed video image corresponding to each frame, and the original video is corrected and reconstructed based on the reconstructed video image of each frame.

4. The video processing method according to claim 1, characterized in that, The training method for the face correction model includes: Obtain a first training dataset and a second training dataset, and obtain a mixed training dataset based on the first training dataset and the second training dataset, wherein the first training dataset includes face images of a first category, and the second training dataset includes face images of a second category; The training images in the hybrid training dataset are respectively input into the first feature extraction network and the second feature extraction network in the initial face correction model to obtain the corresponding first feature vector and second feature vector, wherein the weights of the neural network parameters in the first feature extraction network and the second feature extraction network are shared. The first feature vector is input into the modeling network of the initial face correction model to obtain the three-dimensional detail sample parameters corresponding to the training image, and the second feature vector is input into the tracking network of the initial face correction model to obtain the three-dimensional global sample parameters corresponding to the training image. The three-dimensional detail sample parameters include the sample shape parameters and sample albedo of the face image, and the three-dimensional global sample parameters include the sample expression parameters, sample pose parameters, and sample illumination parameters of the face image. Based on the three-dimensional detail sample parameters and the three-dimensional global sample parameters, the three-dimensional sample information corresponding to the training image is obtained, and the three-dimensional sample information is input into the renderer in the initial face correction model to obtain the two-dimensional output sample image corresponding to the training image. Reconstruction loss and semantic loss are obtained based on the training images and the two-dimensional output sample images, and the training images and the two-dimensional output sample images are input into the discriminator to obtain the discriminator loss; The target loss is obtained based on the discriminator loss, the reconstruction loss, and the semantic loss; The neural network parameters in the initial face correction model are iteratively updated based on the target loss to obtain the trained face correction model.

5. The video processing method according to claim 4, characterized in that, The step of obtaining the 3D sample information corresponding to the training image based on the 3D detail sample parameters and the 3D global sample parameters includes: The sample shape parameter and sample albedo in the three-dimensional detail sample parameters, and the sample expression parameter in the three-dimensional global sample parameters are input into the three-dimensional deformation model to obtain the three-dimensional sample parameters corresponding to the sample shape parameter, the sample albedo and the sample expression parameter, respectively. Based on the three-dimensional sample parameters corresponding to the sample shape parameters, the sample albedo, and the sample expression parameters, as well as the sample pose parameters and the sample illumination parameters in the three-dimensional global sample parameters, the three-dimensional sample information corresponding to the training image is obtained.

6. The video processing method according to claim 4, characterized in that, The process of obtaining reconstruction loss and semantic loss based on the training images and the two-dimensional output sample images includes: The semantic loss is obtained based on the semantic feature vectors corresponding to the training images and the semantic feature vectors corresponding to the two-dimensional output sample images. A two-dimensional output target image is obtained based on the two-dimensional output sample image and the face image mask, and a reconstruction loss is obtained based on the loss value of each pixel between the training image and the two-dimensional output target image.

7. A video processing apparatus, characterized in that, include: The face image acquisition module is configured to perform recognition and capture the corresponding face image in each frame of the original video, and input the corresponding face image in each frame of the original video into a pre-trained face correction model in sequence. The detail parameter determination module is configured to execute the first feature extraction network and the modeling network in the face correction model to obtain the three-dimensional detail parameters of the face image corresponding to each frame of the original video. The three-dimensional detail parameters include the shape parameters and albedo of the face image. The global parameter determination module is configured to execute the second feature extraction network and the tracking network in the face correction model to obtain the three-dimensional global parameters of the face image corresponding to each frame of the original video. The three-dimensional global parameters include the expression parameters, pose parameters and illumination parameters of the face image. The weights of the neural network parameters in the first feature extraction network and the second feature extraction network are shared. The reconstructed video generation module is configured to input the three-dimensional detail parameters and the three-dimensional global parameters into the renderer to obtain the corrected face image, and to obtain the reconstructed video after correcting the original video based on each frame of the corrected face image; The step of inputting the three-dimensional detail parameters and the three-dimensional global parameters into the renderer to obtain the corrected face image includes: inputting the shape parameter and albedo from the three-dimensional detail parameters, and the expression parameter from the three-dimensional global parameters into a three-dimensional deformation model to obtain the three-dimensional parameters corresponding to the shape parameter, the albedo, and the expression parameter, respectively; obtaining the three-dimensional information corresponding to the face image based on the three-dimensional parameters corresponding to the shape parameter, the albedo, and the expression parameter, and the pose parameter and the lighting parameter from the three-dimensional global parameters; and inputting the three-dimensional information corresponding to the face image into the renderer to obtain the corrected face image.

8. The video processing apparatus according to claim 7, characterized in that, The detailed parameter determination module includes: The specified frame parameter determination unit is configured to input the face images corresponding to a preset number of specified frames in the original video into the first feature extraction network and the modeling network in the face correction model to obtain the three-dimensional detail parameters of the face images corresponding to the preset number of specified frames. The detail parameter determination unit is configured to perform the determination of the three-dimensional detail parameters of the face image corresponding to each frame of the original video based on the three-dimensional detail parameters of the face image corresponding to the preset number of specified frames.

9. The video processing apparatus according to claim 7, characterized in that, The reconstructed video generation module also includes: The video background fusion unit is configured to perform a fusion process on the corrected face image of each frame and the corresponding video background to obtain a reconstructed video image corresponding to each frame, and to obtain the original video after correction based on the reconstructed video image of each frame.

10. The video processing apparatus according to claim 7, characterized in that, The video processing device further includes a face correction model training module, which includes: The training dataset acquisition unit is configured to acquire a first training dataset and a second training dataset, and obtain a mixed training dataset based on the first training dataset and the second training dataset, wherein the first training dataset includes face images of a first category, and the second training dataset includes face images of a second category. The feature vector determination unit is configured to input the training images in the mixed training dataset into the first feature extraction network and the second feature extraction network in the initial face correction model to obtain the corresponding first feature vector and second feature vector, wherein the weights of the neural network parameters in the first feature extraction network and the second feature extraction network are shared. The sample parameter determination unit is configured to input the first feature vector into the modeling network of the initial face correction model to obtain the three-dimensional detail sample parameters corresponding to the training image, and input the second feature vector into the tracking network of the initial face correction model to obtain the three-dimensional global sample parameters corresponding to the training image. The three-dimensional detail sample parameters include the sample shape parameters and sample albedo of the face image, and the three-dimensional global sample parameters include the sample expression parameters, sample pose parameters, and sample illumination parameters of the face image. The output sample determination unit is configured to execute the process of obtaining the three-dimensional sample information corresponding to the training image based on the three-dimensional detail sample parameters and the three-dimensional global sample parameters, and inputting the three-dimensional sample information into the renderer in the initial face correction model to obtain the two-dimensional output sample image corresponding to the training image; The loss function determination unit is configured to perform operations to obtain reconstruction loss and semantic loss based on the training image and the two-dimensional output sample image, and input the training image and the two-dimensional output sample image into the discriminator to obtain the discriminator loss; The target loss determination unit is configured to perform a target loss determination based on the discriminator loss, the reconstruction loss, and the semantic loss; The correction model iteration unit is configured to perform iterative updates on the neural network parameters in the initial face correction model based on the target loss to obtain the trained face correction model.

11. The video processing apparatus according to claim 10, characterized in that, The output sample determination unit includes: The three-dimensional sample parameter calculation unit is configured to input the sample shape parameter and the sample albedo in the three-dimensional detail sample parameters, and the sample expression parameter in the three-dimensional global sample parameters into the three-dimensional deformation model, and obtain the three-dimensional sample parameters corresponding to the sample shape parameter, the sample albedo and the sample expression parameter respectively. The three-dimensional sample information determination unit is configured to perform operations to obtain the three-dimensional sample information corresponding to the training image based on the three-dimensional sample parameters corresponding to the sample shape parameters, the sample albedo and the sample expression parameters, as well as the sample pose parameters and the sample illumination parameters in the three-dimensional global sample parameters.

12. The video processing apparatus according to claim 10, characterized in that, The loss function determination unit includes: The semantic loss calculation unit is configured to perform semantic loss calculation based on the semantic feature vector corresponding to the training image and the semantic feature vector corresponding to the two-dimensional output sample image; The reconstruction loss calculation unit is configured to perform operations to obtain a two-dimensional output target image based on the two-dimensional output sample image and the face image mask, and to obtain a reconstruction loss based on the loss value of each pixel between the training image and the two-dimensional output target image.

13. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the video processing method as described in any one of claims 1 to 6.

14. A computer-readable storage medium, wherein instructions in the computer-readable storage medium, when executed by a processor of an electronic device, enable the electronic device to perform the video processing method as described in any one of claims 1 to 6.

15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the video processing method as described in any one of claims 1 to 6.

Images