Video processing method, model training method and device, equipment, storage medium, product

Abstract

The embodiment of the present application discloses a video processing method, a model training method and apparatus, a device, a storage medium and a product. The video processing method includes: obtaining the facial identity features of the first object and the video to be processed, extracting the displacement features of the facial image frame set associated with each facial video frame in the video to be processed, obtaining the displacement features associated with each facial video frame, calling the face-changing model to perform the first face-changing processing on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, and generating a face-changing video corresponding to the video to be processed based on the face-changing results of M facial video frames. It can be seen that in the process of the first face-changing processing, by adding the displacement features associated with the facial video frames and the facial image frame set associated with the facial video frames, the correlation between the face-changing results of adjacent facial video frames can be improved, thereby improving the stability of the face-changing video.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and specifically to a video processing method, a model training method, a video processing device, a model training device, a computer device, a computer-readable storage medium, and a video processing product and a model training product. Background Art

[0002] With the advancement of scientific and technological research, face-changing technology has developed rapidly. Face-changing technology can be understood as: replacing the facial features of the first object in the source image (source) with the face of the second object in the template image (template) to obtain a facial image frame (fake). The facial image frame retains information such as the background in the template image, the expression of the second object, and the facial angle. Research has found that although the face-changing effect of a single image is relatively ideal, when face-changing technology is applied to a video, the stability of the face-changing video will be poor (such as incoherence between the previous and next frames). Summary of the invention

[0003] The embodiments of the present application provide a video processing method, a model training method, an apparatus, a device, a computer-readable storage medium and a product, which can improve the stability of face-changing videos.

[0004] On the one hand, an embodiment of the present application provides a video processing method, including:

[0005] Obtaining a facial identity feature of a first object and a video to be processed, wherein the facial identity feature of the first object is obtained by performing facial feature recognition on a source image, the source image includes the face of the first object, and the facial identity feature is a facial feature used to distinguish the identity of the object; the video to be processed includes M facial video frames, each facial video frame includes the face of the second object, and M is an integer greater than 1;

[0006] Extracting displacement features of a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame, wherein the set of facial image frames associated with any facial video frame includes N facial image frames, where N is an integer greater than 1 and less than M;

[0007] Calling the face-swapping model to perform a first face-swapping process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-swapping result for each facial video frame;

[0008] Based on the face-swapping results of the M facial video frames, a face-swapping video corresponding to the video to be processed is generated.

[0009] In the embodiment of the present application, the facial identity features of the first object and the video to be processed are obtained, and the displacement features of the facial image frame set associated with each facial video frame in the video to be processed are extracted to obtain the displacement features associated with each facial video frame, and the face-changing model is called to perform the first face-changing processing on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object to obtain the face-changing result of each facial video frame, and the face-changing video corresponding to the video to be processed is generated based on the face-changing results of M facial video frames. It can be seen that in the process of the first face-changing processing, by adding the displacement features associated with the facial video frames and the facial image frame set associated with the facial video frames, the correlation between the face-changing results of adjacent facial video frames can be improved, thereby improving the stability of the face-changing video.

[0010] On the one hand, an embodiment of the present application provides a model training method, comprising:

[0011] Obtaining a facial identity feature of a first object and a sample video, wherein the facial identity feature of the first object is obtained by performing facial feature recognition on a source image, the source image includes the face of the first object, and the facial identity feature is a facial feature used to distinguish the identity of the object; the sample video includes M facial video frames, each facial video frame includes the face of the first object, and M is an integer greater than 1;

[0012] Extracting displacement features from a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame, wherein the set of facial image frames associated with any facial video frame includes N facial image frames, each facial image frame includes a face of the second object, and N is an integer greater than 1 and less than M;

[0013] Calling the face-changing model to be optimized to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-changing result for each facial video frame;

[0014] The face-changing model to be optimized is optimized through the face-changing results of each facial video frame to obtain an optimized face-changing model.

[0015] In the embodiment of the present application, the facial identity features and sample videos of the first object are obtained, and the displacement features of the facial image frame set associated with each facial video frame in the sample video are extracted to obtain the displacement features associated with each facial video frame, and the face-changing model to be optimized is called to perform the first face-changing processing on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object to obtain the face-changing result of each facial video frame, and the face-changing model to be optimized is optimized by the face-changing results of each facial video frame to obtain the optimized face-changing model. It can be seen that adding the displacement features associated with the facial video frames and the facial image frame set associated with the facial video frames in the process of the first face-changing processing can improve the quality of the face-changing results and improve the training effect of the face-changing model to be optimized.

[0016] On the one hand, an embodiment of the present application provides a video processing device, the video processing device comprising:

[0017] an acquisition unit, configured to acquire a facial identity feature of a first object and a video to be processed, wherein the facial identity feature of the first object is obtained by performing facial feature recognition on a source image, the source image includes the face of the first object, and the facial identity feature is a facial feature used to distinguish the identity of the object; the video to be processed includes M facial video frames, each facial video frame includes the face of the second object, and M is an integer greater than 1;

[0018] A processing unit, configured to extract displacement features from a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame, wherein the set of facial image frames associated with any facial video frame includes N facial image frames, where N is an integer greater than 1 and less than M;

[0019] and for calling the face-changing model to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-changing result for each facial video frame;

[0020] And used to generate a face-changing video corresponding to the video to be processed based on the face-changing results of the M facial video frames.

[0021] In one embodiment, the processing unit calls the face-swapping model to perform a first face-swapping process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, and the process of obtaining the face-swapping result of each facial video frame includes:

[0022] Down-sample and encode the set of facial image frames associated with the i-th facial video frame to obtain the latent space features corresponding to the i-th facial video frame, where i is a positive integer less than or equal to M;

[0023] splicing the displacement feature associated with the i-th facial video frame with the facial identity feature of the first object to obtain a splicing feature of the i-th facial video frame;

[0024] Fusing the splicing features of the i-th facial video frame and the latent space features corresponding to the i-th facial video frame to obtain the fusion features of the i-th facial video frame;

[0025] The fused features of the i-th facial video frame are upsampled and decoded to obtain the face-changing result of the i-th facial video frame.

[0026] In one embodiment, the processing unit extracts displacement features from a set of facial image frames associated with each facial video frame to obtain the displacement features associated with each facial video frame, including:

[0027] Obtain a set of facial image frames associated with the i-th facial video frame, where i is a positive integer less than or equal to M;

[0028] Extract key points from each face image frame in the face image frame set associated with the i-th face video frame to obtain position features of key points of each face image frame;

[0029] Through the temporal relationship of the facial image frames associated with the ith facial video frame and the positional features of the key points of each facial image frame, the displacement features of each key point are extracted to obtain the displacement features associated with the ith facial video frame.

[0030] In one embodiment, the processing unit is used to extract displacement features of each key point through the temporal relationship of the facial image frames associated with the i-th facial video frame and the position features of the key points of each facial image frame, and obtain the displacement features associated with the i-th facial video frame, specifically for:

[0031] By using the temporal relationship of the facial image frames associated with the i-th facial video frame and the positional features of the key points of each facial image frame, displacement features of each key point are extracted to obtain a feature extraction result;

[0032] The feature extraction result is transformed into a dimension to obtain the displacement feature associated with the i-th facial video frame.

[0033] In one implementation, the M facial video frames are continuous video frames in the video to be processed; the processing unit is used to obtain a facial image frame set associated with the i-th facial video frame, specifically to:

[0034] If i is less than N, a facial image frame set associated with the i-th facial video frame is generated based on the first i facial video frames, and there are at least two identical facial video frames in the facial image frame set associated with the i-th facial video frame;

[0035] If i is greater than or equal to N, then the i-N+1th facial video frame to the i-th facial video frame in the M facial video frames are added to the same set to obtain a facial image frame set associated with the i-th facial video frame.

[0036] On the one hand, an embodiment of the present application provides a model training device, the model training device comprising:

[0037] an acquisition unit, configured to acquire a facial identity feature of a first object and a sample video, wherein the facial identity feature of the first object is obtained by performing facial feature recognition on a source image, the source image includes the face of the first object, and the facial identity feature is a facial feature used to distinguish the identity of the object; the sample video includes M facial video frames, each facial video frame includes the face of the first object, and M is an integer greater than 1;

[0038] a processing unit, configured to extract displacement features from a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame, wherein the set of facial image frames associated with any facial video frame includes N facial image frames, each facial image frame includes a face of a second object, and N is an integer greater than 1 and less than M;

[0039] and for calling the face-changing model to be optimized to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-changing result for each facial video frame;

[0040] And it is used to optimize the face-changing model to be optimized through the face-changing results of each facial video frame to obtain the optimized face-changing model.

[0041] In one implementation, the M facial video frames are continuous video frames in a sample video; the processing unit extracts displacement features from a set of facial image frames associated with each facial video frame, and the process of obtaining the displacement features associated with each facial video frame includes:

[0042] Obtaining a facial video frame set associated with the i-th facial video frame, the facial video frame set associated with the i-th facial video frame includes N facial video frames, the N facial video frames are consecutive video frames among the M facial video frames, and i is a positive integer less than or equal to M;

[0043] Using the facial image frames of the second object, performing a second face-changing process on each facial video frame associated with the i-th facial video frame to obtain N facial image frames;

[0044] Adding N facial image frames to the same set to obtain a facial image frame set associated with the i-th facial video frame;

[0045] Based on a set of facial image frames associated with the ith facial video frame, a displacement feature associated with the ith facial video frame is extracted.

[0046] In one embodiment, the processing unit is used to optimize the face-changing model to be optimized through the face-changing results of each face video frame to obtain the optimized face-changing model, specifically for:

[0047] The face-swapping result of each face video frame is discriminated by a discriminator to obtain M discrimination results of the face-swapping result; the discrimination result of any face-swapping result is used to indicate whether the face-swapping result is a synthetic image;

[0048] Based on the discrimination results of the M face-changing results and the M face-changing results, the generative adversarial network is optimized to obtain an optimized face-changing model. The generative adversarial network includes a discriminator and a face-changing model to be optimized.

[0049] In one embodiment, the processing unit optimizes the generative adversarial network based on the discrimination results of the M face-changing results and the M face-changing results to obtain the optimized face-changing model, including:

[0050] The discrimination loss is constructed by the difference between the discrimination result of the j-th face-swapping result and the discrimination result of the j-th face video frame, where j is a positive integer less than or equal to M;

[0051] Based on the difference between the jth face-swapping result and the jth face video frame, construct the face-swapping loss;

[0052] The generative adversarial network is optimized based on the discriminative loss and face-changing loss.

[0053] In one embodiment, the processing unit is used to construct a face-swapping loss based on the difference between the j-th face-swapping result and the j-th face video frame, specifically for:

[0054] The features of the j-th face-changing result and the j-th face video frame at at least two levels are respectively extracted by the convolutional neural network, and the features at different levels are output by different convolutional neural network layers in the convolutional neural network;

[0055] The feature differences between the j-th face-swapping result and the j-th face video frame at each level are summed to obtain the feature loss between the j-th face-swapping result and the j-th face video frame;

[0056] Based on the pixel value difference between the jth face-swapping result and the jth face video frame, construct the pixel-level loss between the jth face-swapping result and the jth face video frame;

[0057] The identity loss of the jth face-swapping result is calculated by the cosine similarity between the facial identity feature of the jth face-swapping result and the jth face video frame;

[0058] The feature loss, pixel-level loss, identity loss and adversarial loss are weightedly summed to obtain the face-changing loss; the adversarial loss is constructed based on the discrimination result of the j-th face-changing result.

[0059] Accordingly, the present application provides a computer device, the computer device comprising:

[0060] a memory, wherein a computer program is stored in the memory;

[0061] A processor is used to load a computer program to implement the above-mentioned video processing method, or to implement the above-mentioned model training method.

[0062] Accordingly, the present application provides a computer-readable storage medium, which stores a computer program, which is suitable for being loaded by a processor and executing the above-mentioned video processing method, or is suitable for being loaded by a processor and executing the above-mentioned model training method.

[0063] Accordingly, the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned video processing method or the above-mentioned model training method.

[0064] In the embodiment of the present application, during video processing, displacement features associated with facial video frames and a set of facial image frames associated with facial video frames are added during face-changing processing to improve the correlation between face-changing results of adjacent facial video frames, thereby improving the stability of the face-changing video. During model training, displacement features associated with facial video frames and a set of facial image frames associated with facial video frames are added during the first face-changing processing to improve the quality of the face-changing results and improve the training effect of the face-changing model to be optimized. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1a A face-changing schematic diagram provided in an embodiment of the present application;

[0067] Figure 1b A data processing scenario diagram provided for an embodiment of the present application;

[0068] Figure 2 A flowchart of a video processing method provided in an embodiment of the present application;

[0069] Figure 3 A video processing architecture diagram provided for an embodiment of the present application;

[0070] Figure 4 A flowchart of a model training method provided in an embodiment of the present application;

[0071] Figure 5a A schematic diagram of extracting features at different levels provided in an embodiment of the present application;

[0072] Figure 5b A model training architecture diagram provided for an embodiment of the present application;

[0073] Figure 6 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application;

[0074] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0075] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0077] This application involves face-changing technology and generative adversarial networks. The following is a brief introduction to face-changing technology and generative adversarial networks:

[0078] Face swapping can be understood as replacing the facial features of the second object (such as a human face) in the template image (template) with the facial features of the first object (such as a human face) in the source image (source) to obtain a facial image frame (fake). The facial image frame retains the background in the template image, as well as the facial expression and facial angle of the second object. Figure 1a A face-changing schematic diagram provided in an embodiment of the present application. Figure 1aAs shown, the background in the template image, as well as information such as the facial expression and facial angle of the second object are retained, and the facial identity features of the second object in the template image are replaced with the facial identity features of the first object in the source image to obtain a facial image frame; wherein the facial identity features refer to features contained in the facial features for identifying the identity of the object (such as facial features, etc.).

[0079] Generative Adversarial Network (GAN): Generative adversarial network is an unsupervised learning method that learns by letting two neural networks compete with each other. Generative adversarial network consists of a generative network and a discriminative network; in this application, the generative network is the face-changing model, and the discriminative network is the discriminator. Specifically, the face-changing model performs a first face-changing process on the displacement features associated with each facial video frame, the set of facial image frames associated with the facial video frame, and the facial identity features of the first object to obtain the face-changing result of each facial video frame, and its output result needs to imitate the real sample as much as possible (that is, try to let the discriminator determine that the output result of the face-changing model is a non-synthetic image). The input data of the discriminator is the real sample (label map) or the output result of the face-changing model, and its purpose is to distinguish whether the input image is a synthetic image (an image obtained by face-changing processing). The face-changing model and the discriminator compete with each other during the training process, and the parameters of the two models are continuously adjusted for alternating optimization, and finally the optimized face-changing model is obtained.

[0080] Based on the above-mentioned face-changing technology and generative adversarial network, the embodiments of the present application provide a video processing solution and a model training solution, which can improve the stability of face-changing videos. Figure 1b A data processing scenario diagram provided in an embodiment of the present application, such as Figure 1bAs shown, the video processing scenario provided by the present application includes a terminal device 101 and a server 102. The video processing solution and model training solution provided by the present application can be executed by the terminal device 101 or the server 102; when the video processing solution and model training solution provided by the present application are both executed by the terminal device 101, the video processing scenario may not include the server 102. Among them, the terminal device may include but is not limited to: smart phones (such as Android phones, IOS phones, etc.), tablet computers, portable personal computers, mobile Internet devices (Mobile Internet Devices, MID), intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, wearable devices, etc., which are not limited in the embodiments of the present application; the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms, which are not limited in the embodiments of the present application.

[0081] It should be noted that Figure 1b The number of terminal devices and servers is only used as an example and does not constitute an actual limitation of the present application. The terminal device 101 and the server 102 can be connected via wired or wireless means, and the present application does not limit this.

[0082] The video processing solution and model training solution provided in the present application can be independent of each other or combined with each other; for example, the model training solution is executed in the server 102, and the trained face-changing model is deployed in the terminal device 101, and then the video processing solution is executed in the terminal device 101; for another example, the model training solution and the video processing solution are executed in the server 102.

[0083] Taking the server 102 executing the video processing solution as an example, the general process of the video processing solution provided by this application is as follows:

[0084] (1) The server 102 obtains the facial identity features of the first object and the video to be processed; wherein the facial identity features of the first object are obtained by performing facial feature recognition on a source image (source), the source image includes the face of the first object, and the facial identity features refer to facial features that can be used to distinguish the identity of the object. In one embodiment, the server 102 performs facial feature recognition on the source image through a facial recognition model to obtain the facial identity features of the first object. The video to be processed includes M facial video frames, each facial video frame includes the face of the second object, and M is an integer greater than 1; in one embodiment, the M facial video frames are continuous facial video frames.

[0085] (2) The server 102 extracts displacement features from the set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame; wherein the set of facial image frames associated with any facial video frame includes N facial image frames, where N is an integer greater than 1 and less than M. The displacement features are used to characterize changes in the face of the second object within a corresponding time period.

[0086] In one embodiment, the server 102 obtains a set of facial image frames associated with the i-th facial video frame, where i is a positive integer less than or equal to M. After obtaining the set of facial image frames associated with the i-th facial video frame, the server 102 extracts key points from each facial image frame in the set of facial image frames associated with the i-th facial video frame, and obtains the position features of the key points of each facial image frame. Then, the server 102 extracts displacement features from each key point through the temporal relationship of the facial image frames associated with the i-th facial video frame and the position features of the key points of each facial image frame, and obtains the displacement features associated with the i-th facial video frame. That is, the displacement features associated with the i-th facial video frame include the displacement features of each key point in the face, and the displacement features of each key point include the position features of the key point at different times (each time corresponds to a facial video frame), and the motion features (such as motion trajectory, moving speed, etc.) in the corresponding time period.

[0087] (3) The server 102 calls the face-changing model to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, and obtains the face-changing result of each facial video frame. The specific process includes: the server 102 downsamples and encodes the facial image frame set associated with the i-th facial video frame to obtain the latent space features corresponding to the i-th facial video frame, where i is a positive integer less than or equal to M. Then the server 102 splices the displacement features associated with the i-th facial video frame with the facial identity features of the first object to obtain the splicing features of the i-th facial video frame. After obtaining the splicing features of the i-th facial video frame, the server 102 fuses the splicing features of the i-th facial video frame and the latent space features corresponding to the i-th facial video frame to obtain the fused features of the i-th facial video frame, and upsamples and decodes the fused features of the i-th facial video frame to obtain the face-changing result of the i-th facial video frame.

[0088] (4) The server 102 generates a face-swapped video corresponding to the video to be processed based on the face-swapped results of the M facial video frames. In one embodiment, the server 102 replaces the M facial video frames in the video to be processed with the face-swapped results of the M facial video frames to obtain a face-swapped video corresponding to the video to be processed.

[0089] Taking the server 102 executing the model training solution as an example, the general process of the model training solution provided in this application is as follows:

[0090] The server 102 obtains the facial identity features and sample video of the first object, and obtains the face-swapping result of each facial video frame in the sample video based on the facial identity features and sample video of the first object. The specific implementation method is different from (1)-(4) in the above-mentioned video processing scheme in that: the sample video includes M facial video frames, and the M facial video frames all contain the face of the first object; in addition, during the training process, the N facial image frames associated with the i-th facial video frame are obtained by performing a second face-swapping process on each facial video frame associated with the i-th facial video frame through the facial image of the second object. During the video processing process, the N facial image frames associated with the i-th facial video frame are directly obtained from the facial video frame associated with the i-th facial video frame (without the need for a second face-swapping process). After obtaining the face-swapping result of each facial video frame in the sample video, the server 102 optimizes the face-swapping model to be optimized through the face-swapping results of each facial video frame, and obtains the optimized face-swapping model. In one embodiment, the server 102 discriminates the face-swapping results of each facial video frame through a discriminator (used during model training) to obtain the discrimination results of M face-swapping results; wherein the discrimination result of any face-swapping result is used to indicate whether the face-swapping result is a synthetic image. After obtaining the discrimination results of the M face-swapping results, the server 102 optimizes the generative adversarial network based on the discrimination results of the M face-swapping results and the M face-swapping results to obtain an optimized face-swapping model; wherein the generative adversarial network includes a discriminator and a face-swapping model to be optimized. In one embodiment, the optimization process of the generative adversarial network includes: on the one hand, the server 102 constructs a discrimination loss by the difference between the discrimination result of the j-th face-swapping result and the discrimination result of the j-th facial video frame (as a label map), where j is a positive integer less than or equal to M. On the other hand, the server 102 constructs a face-swapping loss based on the difference between the j-th face-swapping result and the j-th facial video frame. After obtaining the discrimination loss and the face-changing loss, the server 102 optimizes the generative adversarial network according to the discrimination loss and the face-changing loss.

[0091] In the embodiment of the present application, during video processing, displacement features associated with facial video frames and a set of facial image frames associated with facial video frames are added during face-changing processing to improve the correlation between face-changing results of adjacent facial video frames, thereby improving the stability of the face-changing video. During model training, displacement features associated with facial video frames and a set of facial image frames associated with facial video frames are added during the first face-changing processing to improve the quality of the face-changing results and improve the training effect of the face-changing model to be optimized.

[0092] Based on the above-mentioned video processing scheme and model training scheme, the embodiments of the present application propose a more detailed video processing method and model training method. The video processing method and model training method proposed in the embodiments of the present application will be introduced in detail below in conjunction with the accompanying drawings.

[0093] See also Figure 2 , Figure 2 A flowchart of a video processing method provided in an embodiment of the present application, the video processing method can be executed by a computer device; for example, Figure 1b The terminal device 101 or the server 102 shown in FIG. Figure 2 As shown, the video processing method may include the following steps S201-S204:

[0094] S201: Obtain facial identity features of a first subject and a video to be processed.

[0095] The facial identity features of the first object are obtained by performing facial feature recognition on a source image (source), and the source image contains the face of the first object; for example, the computer device may perform facial feature recognition on the source image through a face recognition model to obtain the facial identity features of the first object. Optionally, before performing facial feature recognition on the source image, the computer device may first detect the facial area in the source image, align the detected facial area through facial key points, and then crop the aligned facial area to obtain a cropped source image. The facial area in the source image may be adjusted to an appropriate scale by cropping for subsequent processing.

[0096] The video to be processed includes M facial video frames, each facial video frame includes the face of the second object, and M is an integer greater than 1. In one embodiment, the M facial video frames are continuous video frames in the video to be processed. If the M facial video frames are non-continuous video frames, the video to be processed can be divided into a plurality of sub-videos, so that the facial video frames included in each sub-video are continuous video frames, and then the video processing method provided by the present application is used to perform face-changing processing on each sub-video.

[0097] The facial identity feature is a facial feature used to distinguish the identity of an object (ie, the facial identity feature is included in the facial features); for example, the facial identity feature may include the facial features of the object.

[0098] The video processing method provided in this application aims to replace all the faces of the second object in the video to be processed with the face of the first object (in the same scene, same action, and same expression) to obtain a face-swapped video. The object can be a person, a virtual character, a robot, etc.

[0099] S202: Extract displacement features of a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame.

[0100] The facial image frame set associated with any facial video frame includes N facial image frames, where N is an integer greater than 1 and less than M. The displacement feature is used to characterize the changes in the face of the second object in the corresponding time period; for example, the displacement feature associated with the i-th facial video frame is the displacement feature extraction of the facial image frame set associated with the i-th facial video frame, and the facial image frame set associated with the i-th facial video frame includes the i-th-N+1th facial video frame to the i-th facial video frame in the M facial video frames, then the displacement feature associated with the i-th facial video frame is used to characterize the changes in the face of the second object in the time period t0-t1, t0 is the time corresponding to the i-N+1th facial video frame, and t1 is the time corresponding to the i-th facial video frame.

[0101] The computer device extracts displacement features from a set of facial image frames associated with each facial video frame to obtain a process of displacement features associated with each facial video frame, including: the computer device obtains a set of facial image frames associated with the i-th facial video frame, where i is a positive integer less than or equal to M.

[0102] In one embodiment, if i is less than N, the computer device generates a facial image frame set associated with the i-th facial video frame based on the first i facial video frames, and there are at least two identical facial video frames in the facial image frame set associated with the i-th facial video frame. For example, assuming N=4 and i=1, the computer device can add 4 repeated first facial video frames to the same set as the facial image frame set associated with the first facial video frame; assuming N=4 and i=2, the computer device can add 2 repeated first facial video frames and 2 repeated second facial video frames to the same set as the facial image frame set associated with the second facial video frame; or add the first facial video frame and 3 repeated second facial video frames to the same set as the facial image frame set associated with the second facial video frame; or add 3 repeated first facial video frames and second facial video frames to the same set as the facial image frame set associated with the second facial video frame. Correspondingly, if i is greater than or equal to N, the computer device can directly add the [i-N+1, i]th facial video frame to the same set (i.e., directly use the facial video frame as the facial image frame) to obtain a facial image frame set associated with the i-th facial video frame.

[0103] Furthermore, after obtaining the set of facial image frames associated with the ith facial video frame, the computer device extracts key points from each facial image frame in the set of facial image frames associated with the ith facial video frame, and obtains the position features of the key points of each facial image frame. In one embodiment, the computer device can extract key points from each facial image frame in the set of facial image frames associated with the ith facial video frame through a trained key point network (such as a two-dimensional key point network, a three-dimensional key point network, etc.), and obtain the position features of the key points of each facial image frame. The position features of the key points in the image frame can be used to locate the coordinates of the contour, facial features, pupil, etc. of the object's face. In practical applications, the key point network and the number of key points can be adjusted according to the requirements of accuracy, scene, etc., and this application does not impose any restrictions on this.

[0104] After obtaining the positional features of the key points of N facial image frames, the computer device extracts the displacement features of each key point through the temporal relationship of the facial image frames associated with the ith facial video frame and the positional features of the key points of each facial image frame, and obtains the displacement features associated with the ith facial video frame. In other words, the displacement features associated with the ith facial video frame include the displacement features of each key point in the face, and the displacement features of each key point include the positional features of the key point at different times (each time corresponds to a facial video frame), as well as the motion features (such as motion trajectory, moving speed, etc.) in the corresponding time period.

[0105] In one embodiment, the computer device may extract the displacement features associated with the ith facial video frame through a multi-frame key point coding network. The multi-frame key point coding network includes at least one network layer (transformer layer). The computer device calls at least one network layer to extract the displacement features of each key point in N facial image frames through the temporal relationship of the facial image frames associated with the ith facial video frame and the position features of the key points of each facial image frame to obtain a feature extraction result, and then (such as through a fully connected layer) performs a dimension conversion on the feature extraction result to obtain the displacement features associated with the ith facial video frame.

[0106] S203, calling the face-changing model to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-changing result for each facial video frame.

[0107] The difference between the first face-changing process and the second face-changing process is that the input data of the first face-changing process is the facial identity features of the first object, the displacement features associated with the facial video frame, and the facial image frame set (including multiple facial image frames) associated with the facial video frame. The input data of the second face-changing process is a facial video frame and a target facial video frame. It is understandable that, under the premise of different input data, there are differences in the results obtained by performing the second face-changing process and the first face-changing process on the same facial video frame.

[0108] In one embodiment, the face swapping process of the i-th facial video frame includes: the computer device performs downsampling encoding on the facial image frame set associated with the i-th facial video frame to obtain the latent space feature (which can be expressed as swap_features) corresponding to the i-th facial video frame, where i is a positive integer less than or equal to M. The number of downsampling encoding is at least one, and the resolution of the facial image frame after each downsampling encoding is less than the resolution before the downsampling encoding, and the number of channels of the facial image frame after each downsampling encoding is greater than the number of channels before the downsampling encoding. In one embodiment, the computer device splices the facial image frames in the facial image frame set, and then performs downsampling encoding on the spliced ​​facial image frames through a convolution layer to obtain the latent space feature corresponding to the i-th facial video frame. Then the computer device splices the displacement feature associated with the i-th facial video frame with the facial identity feature of the first object to obtain the splicing feature of the i-th facial video frame, and the dimension of the splicing feature is the sum of the dimension of the displacement feature associated with the i-th facial video frame and the dimension of the facial identity feature of the first object.

[0109] Furthermore, the computer device fuses the splicing features of the ith facial video frame and the latent space features corresponding to the ith facial video frame to obtain the fused features of the ith facial video frame. The specific fusion method may include but is not limited to: fusion based on the attention mechanism, fusion based on adaptive instance normalization (Adaptive Instance Normalization, AdaIN). In one embodiment, the computer device performs a difference calculation on the latent space features corresponding to the ith facial video frame and the mean of the latent space features corresponding to the ith facial video frame to obtain a target difference. Then, the quotient of the target difference and the variance of the latent space features corresponding to the ith facial video frame is calculated to obtain a first intermediate calculation result. Then, the variance of the splicing features of the ith facial video frame and the product of the first intermediate calculation result are calculated to obtain a second intermediate calculation result. After obtaining the second intermediate calculation result, the computer device sums the second intermediate calculation result and the mean of the splicing features of the ith facial video frame to obtain the fused features of the ith facial video frame. It can be specifically expressed as:

[0110]

[0111] in, Adaptive instance-based normalization fusion and The fusion features obtained later are and are the latent space features corresponding to the i-th facial video frame and the splicing features of the i-th facial video frame, respectively. express The mean of express The variance of .

[0112] After obtaining the fusion features of the i-th facial video frame, the computer device upsamples and decodes the fusion features of the i-th facial video frame to obtain the face-swapping result of the i-th facial video frame. The number of upsampling decoding corresponds to the number of downsampling encoding. After each upsampling decoding, the resolution of the fusion features of the i-th facial video frame is greater than the resolution before the upsampling decoding, and the number of channels of the fusion features of the i-th facial video frame after each upsampling decoding is less than the number of channels before the upsampling decoding. The result of the last upsampling decoding is the face-swapping result of the i-th facial video frame.

[0113] S204: Generate a face-swapped video corresponding to the video to be processed based on the face-swapped results of the M facial video frames.

[0114] In one implementation, a computer device replaces M facial video frames in a video to be processed with face-changing results of M facial video frames to obtain a face-changing video corresponding to the video to be processed.

[0115] Figure 3 A video processing architecture diagram provided in an embodiment of the present application. Figure 3 As shown in the figure, the face-swapping generation network can be generally divided into two parts: the encoder and the decoder. The encoder performs downsampling encoding through the convolutional layer. Each downsampling encoding reduces the resolution of the input data (N facial image frames spliced ​​together) (e.g., halved each time) and increases the number of RGB channels. For example, assuming that the resolution of each facial image frame is 512 512, the number of RGB channels is 3, N=4, then the dimension of the input data is 512 512 (3 4). After that, it passes through the convolution layer and is gradually encoded into 256 256 32,128 128 64,64 64 128,32 32 256, and finally the latent space features are obtained, which can be expressed as swap_features. On the other hand, the position features of the facial key points of each facial image frame in the facial image frame set are extracted through a key point network (such as a 2D key point network). The dimension of the facial key points of each facial image frame is 256 2. Assuming N=4, the dimension of the splicing features of the facial key points of the facial image frame is 4 256 2, which can be expressed as template_pts_list. Then, template_pts_list is sent to the multi-frame key point encoding network. The multi-frame key point encoding network is a transformer structure, which has at least 1 transformer layer and 1 fully connected layer; wherein, the transformer layer is used to extract the displacement features of each facial key point in template_pts_list, and the fully connected layer is used to perform dimension conversion on the features to obtain the displacement features associated with the i-th frame, which can be expressed as refine_template_pts, and the dimension of the displacement features associated with the i-th frame is 512. On the other hand, facial feature recognition is performed on the source image (source) to obtain the facial identity features of the first object, which can be expressed as src_id_features, and the dimension of the facial identity features is 1024. The displacement features associated with the i-th frame and the facial identity features of the first object are spliced ​​together to obtain spliced ​​features, which can be expressed as id_pts_features, and the dimension of the spliced ​​features is 1536. Furthermore, based on the adaptive instance normalization method, id_pts_features and swap_features are fused to obtain the fused features of the i-th frame. Then the fused features are input to the decoder. The decoder mainly performs deconvolution operations (up-sampling decoding). Each up-sampling decoding increases the resolution of the input data (such as doubling) and reduces the number of RGB channels. For example, the fused features of the i-th frame are gradually decoded into 32 32 256, 64 64 128, 128 128 64, 256 256 32,512 512 3. Finally, the face-changing result (fake) of the i-th frame is obtained.

[0116] Optionally, the computer device can also input the face-changing video into a trained optimization model to further optimize the face-changing video (such as adjusting the area containing the face in the target video based on adjacent video frames) to obtain an optimized face-changing result.

[0117] In the embodiment of the present application, the facial identity features of the first object and the video to be processed are obtained, and the displacement features of the facial image frame set associated with each facial video frame in the video to be processed are extracted to obtain the displacement features associated with each facial video frame, and the face-changing model is called to perform the first face-changing processing on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object to obtain the face-changing result of each facial video frame, and the face-changing video corresponding to the video to be processed is generated based on the face-changing results of M facial video frames. It can be seen that in the process of the first face-changing processing, by adding the displacement features associated with the facial video frames and the facial image frame set associated with the facial video frames, the correlation between the face-changing results of adjacent facial video frames can be improved, thereby improving the stability of the face-changing video.

[0118] See also Figure 4 , Figure 4 A flowchart of a model training method provided in an embodiment of the present application, the model training method can be executed by a computer device; for example, Figure 1b The terminal device 101 or the server 102 shown in FIG. Figure 4 As shown, the model training method may include the following steps S401-S404:

[0119] S401: Obtain facial identity features and sample video of a first subject.

[0120] The specific implementation of step S401 can be found in Figure 2 The implementation of step S201 in the example is not described in detail here. The difference is that, among the M face video frames included in the sample video, each face video frame includes the face of the first object, and M is an integer greater than 1. In one implementation, the M face video frames are continuous video frames in the sample video.

[0121] The model training method provided in the present application aims to optimize the face-changing model to be optimized by using the difference between the face-changing results of the facial video frames in the sample video frames and the corresponding facial video frames in the sample video frames to obtain an optimized face-changing model.

[0122] S402: Extract displacement features of a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame.

[0123] The set of facial image frames associated with any facial video frame includes N facial image frames, which are obtained by performing a second face-changing process on N (continuous) facial video frames in the sample video based on the facial image of a second object (any object other than the first object), and each facial image frame contains the face of the second object, and N is an integer greater than 1 and less than M. In addition, the Nth facial video frame among the N (continuous) facial video frames can be used as a labeled image (without the second face-changing process). For the discriminator, it is expected that the discrimination result of the N facial image frames is a synthetic image, and the discrimination result of the Nth facial video frame (labeled image) is a non-synthetic image. The discriminator can be initialized and trained through the set of facial image frames associated with any facial video frame and the corresponding labeled image.

[0124] In one embodiment, a computer device obtains a facial video frame set associated with the ith facial video frame, wherein the facial video frame set associated with the ith facial video frame includes N facial video frames, wherein the N facial video frames may be adjacent video frames of the ith facial video frame. The N facial video frames may be any N consecutive facial video frames in the sample video (in playback order) that include the ith facial video frame.

[0125] In one implementation, if i is less than N, the computer device generates a facial video frame set associated with the i-th facial video frame based on the first i facial video frames; wherein there are at least two identical facial video frames in the facial video frame set associated with the i-th facial video frame. For example, assuming N=4 and i=1, the computer device may add 4 repeated first facial video frames to the same set as a facial video frame set associated with the first facial video frame; assuming N=4 and i=2, the computer device may add 2 repeated first facial video frames and 2 repeated second facial video frames to the same set as a facial video frame set associated with the second facial video frame; or add the first facial video frame and 3 repeated second facial video frames to the same set as a facial video frame set associated with the second facial video frame; or add 3 repeated first facial video frames and second facial video frames to the same set as a facial video frame set associated with the second facial video frame. Correspondingly, if i is greater than or equal to N, the computer device adds the i-N+1th facial video frame to the i-th facial video frame among the M facial video frames to the same set to obtain a facial video frame set associated with the i-th facial video frame.

[0126] Then, the computer device uses the facial image of the second object to perform a second face-changing process on each facial video frame associated with the ith facial video frame, respectively, to obtain N facial image frames; for example, the computer device can use a trained facial processing model to perform a second face-changing process on each facial video frame associated with the ith facial video frame and the target facial video frame, to obtain facial image frames of each facial video frame associated with the ith facial video frame. During a second face-changing process, the input data of the trained facial processing model is any facial video frame associated with the ith facial video frame and the facial image of the second object, and the output result is a facial image frame corresponding to the input facial video frame.

[0127] After obtaining N facial image frames, the computer device adds the N facial image frames to the same set to obtain a facial image frame set associated with the ith facial video frame. Taking N=4 as an example, the facial image frame set associated with the ith facial video frame can be expressed as: {fake_template_i-3, fake_template_i-2, fake_template_i-1, fake_template_i}; the 4 facial image frames associated with the ith facial video frame are obtained by performing a second face-changing process on the facial video frames associated with the ith facial video frame (which can be expressed as {A_i-3, A_i-2, A_i-1, A_i}) based on the facial image of the second object (which can be expressed as A_j).

[0128] As can be seen from the above, during the training process, the N facial image frames associated with the ith facial video frame are obtained by performing the second face-changing process on each facial video frame associated with the ith facial video frame through the facial image of the second object. During the video processing process, the N facial image frames associated with the ith facial video frame are directly obtained from the facial video frame associated with the ith facial video frame (without the need for the second face-changing process).

[0129] Further, after obtaining the set of facial image frames associated with the ith facial video frame, the computer device extracts the displacement feature associated with the ith facial video frame based on the set of facial image frames associated with the ith facial video frame. For a specific implementation method, please refer to Figure 2 The implementation of step S202 will not be repeated here.

[0130] S403, calling the face-changing model to be optimized to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-changing result for each facial video frame.

[0131] The specific implementation of step S403 can be found in Figure 2Through the above steps S402 and S403, the computer device can optimize the face-changing model to be optimized by the difference between the face-changing result of the face video frame in the sample video frame and the corresponding face video frame in the sample video frame, so as to achieve unsupervised training.

[0132] S404, optimizing the face-changing model to be optimized through the face-changing results of each facial video frame to obtain an optimized face-changing model.

[0133] In one embodiment, the computer device discriminates the face-swapping results of each face video frame through a discriminator to obtain M discrimination results of the face-swapping results. The discrimination result of any face-swapping result is used to indicate whether the face-swapping result is a synthetic image.

[0134] Furthermore, the computer device optimizes the generative adversarial network based on the discrimination results of the M face-changing results and the M face-changing results to obtain an optimized face-changing model; wherein the generative adversarial network includes a discriminator and a face-changing model to be optimized. In the optimization process of the generative adversarial network, on the one hand, by adjusting the parameters of the discriminator, the discriminator can judge as accurately as possible whether the input image is a synthetic image; on the other hand, by adjusting the parameters of the face-changing model to be optimized, the image output by the face-changing model is recognized by the discriminator as a non-synthetic image.

[0135] In one embodiment, the computer device optimizes the generative adversarial network based on the discrimination results of the M face-changing results and the M face-changing results to obtain the optimized face-changing model, including:

[0136] On the one hand, the computer device constructs a discrimination loss through the difference between the discrimination result of the j-th face-changing result and the discrimination result of the j-th face video frame, where j is a positive integer less than or equal to M. Specifically, it can be expressed as: D_loss = -logD(A_T) - log(1 – D(fake)); where D_loss represents the discrimination loss, D(fake) represents the discrimination result of the j-th face-changing result, and D(A_T) represents the discrimination result of the j-th face video frame.

[0137] On the other hand, the computer device constructs a face-swapping loss based on the difference between the jth face-swapping result and the jth face video frame. In one implementation, on the one hand, the computer device extracts the features of the jth face-swapping result and the jth face video frame at at least two levels through a convolutional neural network (such as alexnet); wherein the features at different levels are output by different convolutional neural network layers in the convolutional neural network. Then, the feature differences at each level between the jth face-swapping result and the jth face video frame are summed to obtain the feature loss between the jth face-swapping result and the jth face video frame. For example, the computer device can extract the features of the jth face-swapping result at 4 levels through a pre-trained alexnet network to obtain fake_fea1, fake_fea2, fake_fea3, and fake_fea4; and extract the features of the jth face video frame at 4 levels through a pre-trained alexnet network to obtain gt_fea1, gt_fea2, gt_fea3, and gt_fea4. Further, the feature loss between the j-th face swap result and the j-th face video frame can be expressed as: LPIPS_loss = | fake_fea1 - gt_fea1 | + | fake_fea2 - gt_fea2| + | fake_fea3 -gt_fea3 | + | fake_fea4 - gt_fea4 |.

[0138] Figure 5a A schematic diagram of extracting features at different levels provided in an embodiment of the present application. Figure 5a As shown in the figure, the number of convolutional layers in AlexNet is 8; the first convolutional layer can extract the edge features of the j-th face-changing result, the third convolutional layer can extract the texture features of the j-th face-changing result, the fifth convolutional layer can perceive the local and overall objects, and the eighth layer can perceive the category of the object. It should be noted that Figure 5a The number of convolutional layers included in the convolutional neural network is only used as an example. The specific number of convolutional layers can be dynamically adjusted based on demand, and this application does not impose any restrictions on this.

[0139] On the other hand, the computer device constructs a pixel-level loss between the j-th face-changing result and the j-th facial video frame based on the pixel value difference between the j-th face-changing result and the j-th facial video frame; for example, L1_loss = abs(fake–A_T); wherein L1_loss is the pixel-level loss between the j-th face-changing result and the j-th facial video frame, fake is the pixel value of each pixel in the j-th face-changing result, A_T is the pixel value of each pixel in the j-th facial video frame, and abs represents the absolute value.

[0140] On the other hand, the computer device calculates the identity loss between the jth face-swapping result and the jth face video frame through the cosine similarity between the facial identity features of the jth face-swapping result and the facial identity features of the jth face video frame. Specifically, it can be expressed as: ID_loss = 1 – cosine_similarity (fake_id_features, source_id_features); where fake_id_features is the facial identity features of the jth face-swapping result, source_id_features is the facial identity features of the jth face video frame, and the specific calculation method of cosine_similarity (A, B) is:

[0141]

[0142] in, and They represent the components of the facial identity features of the j-th face-changing result and the facial identity features of the j-th face video frame respectively.

[0143] On the other hand, the computer device constructs an adversarial loss based on the judgment result of the jth face-changing result. Specifically, it can be expressed as: G_loss = log (1-D(fake)); where D(fake) represents the judgment result of the jth face-changing result, and G_loss is the adversarial loss.

[0144] After obtaining the feature loss, pixel-level loss, identity loss and adversarial loss, the computer device performs a weighted summation of the feature loss, pixel-level loss, identity loss and adversarial loss to obtain the face-changing loss. Specifically, it can be expressed as: loss = a L1_loss + b LPIPS_loss + c ID_loss + G_loss; where a, b, c are weights (such as a=20, b=20, c=5), loss is the face-changing loss, L1_loss is the pixel-level loss between the j-th face-changing result and the j-th face video frame, LPIPS_loss is the feature loss between the j-th face-changing result and the j-th face video frame, ID_loss is the identity loss between the j-th face-changing result and the j-th face video frame, and G_loss is the adversarial loss.

[0145] Furthermore, after obtaining the discriminant loss and the face-changing loss, the computer device optimizes the generative adversarial network according to the discriminant loss and the face-changing loss.

[0146] Figure 5b A model training architecture diagram provided in an embodiment of the present application. Figure 5bAs shown, compared to Figure 3 For , the architectural difference lies in the inclusion of a discriminator during model training. Figure 3 On the basis of, based on the discrimination result of the face-swapping result of the i-th frame and the discrimination result of the i-th frame, a discrimination loss is constructed; and based on the face-swapping result of the i-th frame and the discrimination result of the i-th frame, a face-swapping loss is constructed. The construction process of the face-swapping loss includes: on the one hand, constructing feature loss through the features of the face-swapping result of the i-th frame and the i-th frame (label map) at different levels. On the other hand, based on the pixel value difference between the face-swapping result of the i-th frame and the i-th frame, constructing the pixel-level loss between the face-swapping result of the i-th frame and the i-th frame. On the other hand, facial feature recognition is performed on the face-swapping result of the i-th frame to obtain the facial identity features of the face-swapping result of the i-th frame, which can be expressed as fake_id_features. Through the cosine similarity between the facial identity features of the face-swapping result of the i-th frame and the facial identity features of the i-th frame, the identity loss between the face-swapping result of the i-th frame and the i-th frame is calculated. On the other hand, an adversarial loss is constructed based on the face-swapping result of the i-th frame. Finally, the feature loss, pixel-level loss, identity loss and adversarial loss are weighted and summed to obtain the face-changing loss. After obtaining the face-changing loss and the discriminant loss, the generative adversarial network consisting of the discriminator and the face-changing model is optimized based on the face-changing loss and the discriminant loss to obtain the optimized face-changing model.

[0147] Understandably, Figure 4 The model training method in can be used with Figure 2 The video processing methods in are independent of each other and can also be Figure 2 Executed before the video processing method in .

[0148] In the embodiment of the present application, the facial identity features and sample videos of the first object are obtained, and the displacement features of the facial image frame set associated with each facial video frame in the sample video are extracted to obtain the displacement features associated with each facial video frame, and the face-changing model to be optimized is called to perform the first face-changing processing on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object to obtain the face-changing result of each facial video frame, and the face-changing model to be optimized is optimized by the face-changing results of each facial video frame to obtain the optimized face-changing model. It can be seen that adding the displacement features associated with the facial video frames and the facial image frame set associated with the facial video frames in the process of the first face-changing processing can improve the quality of the face-changing results and improve the training effect of the face-changing model to be optimized.

[0149] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0150] See also Figure 6 , Figure 6 A schematic diagram of the structure of a video processing device provided in an embodiment of the present application is shown in FIG. Figure 6 The video processing apparatus shown may be mounted in a computer device, which may specifically be a terminal device or a server. Figure 6 The video processing device shown can be used to perform the above Figure 2 Some or all of the functions of the described method embodiments. Figure 6 , the video processing device comprises:

[0151] The acquisition unit 601 is used to acquire the facial identity feature of the first object and the video to be processed, wherein the facial identity feature of the first object is obtained by performing facial feature recognition on a source image, the source image includes the face of the first object, and the facial identity feature is a facial feature used to distinguish the identity of the object; the video to be processed includes M facial video frames, each facial video frame includes the face of the second object, and M is an integer greater than 1;

[0152] The processing unit 602 is used to extract displacement features from the set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame, wherein the set of facial image frames associated with any facial video frame includes N facial image frames, where N is an integer greater than 1 and less than M.

[0153] and for calling the face-changing model to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-changing result for each facial video frame;

[0154] And used to generate a face-changing video corresponding to the video to be processed based on the face-changing results of the M facial video frames.

[0155] In one embodiment, the processing unit 602 calls the face-swapping model to perform a first face-swapping process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, and the process of obtaining the face-swapping result of each facial video frame includes:

[0156] Down-sample and encode the set of facial image frames associated with the i-th facial video frame to obtain the latent space features corresponding to the i-th facial video frame, where i is a positive integer less than or equal to M;

[0157] splicing the displacement feature associated with the i-th facial video frame with the facial identity feature of the first object to obtain a splicing feature of the i-th facial video frame;

[0158] Fusing the splicing features of the i-th facial video frame and the latent space features corresponding to the i-th facial video frame to obtain the fusion features of the i-th facial video frame;

[0159] The fused features of the i-th facial video frame are upsampled and decoded to obtain the face-changing result of the i-th facial video frame.

[0160] In one implementation, the processing unit 602 extracts displacement features from the set of facial image frames associated with each facial video frame, and the process of obtaining the displacement features associated with each facial video frame includes:

[0161] Obtain a set of facial image frames associated with the i-th facial video frame, where i is a positive integer less than or equal to M;

[0162] Extract key points from each face image frame in the face image frame set associated with the i-th face video frame to obtain position features of key points of each face image frame;

[0163] Through the temporal relationship of the facial image frames associated with the ith facial video frame and the positional features of the key points of each facial image frame, the displacement features of each key point are extracted to obtain the displacement features associated with the ith facial video frame.

[0164] In one implementation, the processing unit 602 is configured to extract displacement features of each key point through the temporal relationship of the facial image frames associated with the i-th facial video frame and the position features of the key points of each facial image frame, and obtain the displacement features associated with the i-th facial video frame, specifically for:

[0165] By using the temporal relationship of the facial image frames associated with the i-th facial video frame and the positional features of the key points of each facial image frame, displacement features of each key point are extracted to obtain a feature extraction result;

[0166] The feature extraction result is transformed into a dimension to obtain the displacement feature associated with the i-th facial video frame.

[0167] In one implementation, the M facial video frames are continuous video frames in the video to be processed; the processing unit 602 is used to obtain a facial image frame set associated with the i-th facial video frame, specifically to:

[0168] If i is less than N, a facial image frame set associated with the i-th facial video frame is generated based on the first i facial video frames, and there are at least two identical facial video frames in the facial image frame set associated with the i-th facial video frame;

[0169] If i is greater than or equal to N, then the i-N+1th facial video frame to the i-th facial video frame in the M facial video frames are added to the same set to obtain a facial image frame set associated with the i-th facial video frame.

[0170] According to one embodiment of the present application, Figure 2 Some of the steps involved in the video processing method shown can be represented by Figure 6 The video processing device shown in the figure is executed by each unit. For example, Figure 2 The step S201 shown in FIG. 1 can be performed by Figure 6 The acquisition unit 601 shown in FIG. 6 is executed, and steps S202 to S204 can be performed by Figure 6 Processing unit 602 is shown executing. Figure 6 The various units in the video processing device shown can be separately or completely combined into one or several other units to form, or one (some) of the units can be further divided into multiple functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the video processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0171] According to another embodiment of the present application, the program can be executed by running a program on a general computing device such as a computer device including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 2 A computer program (including program code) for each step involved in the corresponding method shown in Figure 6 The video processing device shown in the embodiment of the present application is used to implement the video processing method of the embodiment of the present application. The computer program can be recorded on a computer-readable recording medium, for example, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0172] Based on the same inventive concept, the principles and beneficial effects of solving the problems by the video processing device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems by the video processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0173] See also Figure 7 , Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application, Figure 7 The model training apparatus shown may be mounted in a computer device, which may specifically be a terminal device or a server. Figure 7 The model training apparatus shown can be used to perform the above Figure 4 Some or all of the functions of the described method embodiments. Figure 7, the model training device comprises:

[0174] The acquisition unit 701 is used to acquire a facial identity feature of a first object and a sample video, wherein the facial identity feature of the first object is obtained by performing facial feature recognition on a source image, the source image includes the face of the first object, and the facial identity feature is a facial feature used to distinguish the identity of the object; the sample video includes M facial video frames, each facial video frame includes the face of the first object, and M is an integer greater than 1;

[0175] The processing unit 702 is configured to extract displacement features from a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame, wherein the set of facial image frames associated with any facial video frame includes N facial image frames, each facial image frame includes a face of a second object, and N is an integer greater than 1 and less than M.

[0176] and for calling the face-changing model to be optimized to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-changing result for each facial video frame;

[0177] And it is used to optimize the face-changing model to be optimized through the face-changing results of each facial video frame to obtain the optimized face-changing model.

[0178] In one implementation, the M facial video frames are continuous video frames in the sample video; the processing unit 702 extracts displacement features from a set of facial image frames associated with each facial video frame, and the process of obtaining the displacement features associated with each facial video frame includes:

[0179] Obtaining a facial video frame set associated with the i-th facial video frame, the facial video frame set associated with the i-th facial video frame includes N facial video frames, the N facial video frames are consecutive video frames among the M facial video frames, and i is a positive integer less than or equal to M;

[0180] Using the facial image frames of the second object, performing a second face-changing process on each facial video frame associated with the i-th facial video frame to obtain N facial image frames;

[0181] Adding N facial image frames to the same set to obtain a facial image frame set associated with the i-th facial video frame;

[0182] Based on a set of facial image frames associated with the ith facial video frame, a displacement feature associated with the ith facial video frame is extracted.

[0183] In one implementation, the processing unit 702 is used to optimize the face-changing model to be optimized by using the face-changing results of each facial video frame to obtain an optimized face-changing model, specifically for:

[0184] The face-swapping result of each face video frame is discriminated by a discriminator to obtain M discrimination results of the face-swapping result; the discrimination result of any face-swapping result is used to indicate whether the face-swapping result is a synthetic image;

[0185] Based on the discrimination results of the M face-changing results and the M face-changing results, the generative adversarial network is optimized to obtain an optimized face-changing model. The generative adversarial network includes a discriminator and a face-changing model to be optimized.

[0186] In one implementation, the processing unit 702 optimizes the generative adversarial network based on the discrimination results of the M face-changing results and the M face-changing results to obtain the optimized face-changing model, including:

[0187] The discrimination loss is constructed by the difference between the discrimination result of the j-th face-swapping result and the discrimination result of the j-th face video frame, where j is a positive integer less than or equal to M;

[0188] Based on the difference between the jth face-swapping result and the jth face video frame, construct the face-swapping loss;

[0189] The generative adversarial network is optimized based on the discriminative loss and face-changing loss.

[0190] In one implementation, the processing unit 702 is configured to construct a face-swapping loss based on the difference between the j-th face-swapping result and the j-th face video frame, specifically to:

[0191] The features of the j-th face-changing result and the j-th face video frame at at least two levels are respectively extracted by the convolutional neural network, and the features at different levels are output by different convolutional neural network layers in the convolutional neural network;

[0192] The feature difference between the j-th face-swapping result and the j-th face video frame at each level is summed up to obtain the feature loss between the j-th face-swapping result and the j-th face video frame;

[0193] Based on the pixel value difference between the j-th face-swapping result and the j-th face video frame, construct the pixel-level loss between the j-th face-swapping result and the j-th face video frame;

[0194] The identity loss of the jth face-swapping result is calculated by the cosine similarity between the facial identity features of the jth face-swapping result and the jth face video frame;

[0195] The feature loss, pixel-level loss, identity loss and adversarial loss are weightedly summed to obtain the face-changing loss; the adversarial loss is constructed based on the discrimination result of the j-th face-changing result.

[0196] According to one embodiment of the present application, Figure 4 Some of the steps involved in the model training method shown can be represented by Figure 7 The model training device shown is executed by each unit. For example, Figure 4 The step S401 shown in FIG. 4 can be performed by Figure 7 The acquisition unit 701 shown in FIG. 1 is executed, and steps S402 to S404 can be performed by Figure 7 Processing unit 702 is shown executing. Figure 7 The various units in the model training device shown can be separately or completely combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple functionally smaller units to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the model training device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0197] According to another embodiment of the present application, the program can be executed by running a program on a general computing device such as a computer device including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 4 A computer program (including program code) for each step involved in the corresponding method shown in Figure 7 The model training device shown in and the model training method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer readable recording medium, and loaded into the above-mentioned computing device through the computer readable recording medium and run therein.

[0198] Based on the same inventive concept, the principles and beneficial effects of solving the problems by the model training device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems by the model training method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0199] See also Figure 8 , Figure 8 The following is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a terminal device or a server. Figure 8As shown, the computer device at least includes a processor 801, a communication interface 802 and a memory 803. The processor 801, the communication interface 802 and the memory 803 can be connected via a bus or other means. The processor 801 (or central processing unit (CPU)) is the computing core and control core of the computer device, which can parse various instructions in the computer device and process various data of the computer device. For example, the CPU can be used to parse the power on and off instructions issued by the object to the computer device, and control the computer device to perform power on and off operations; for example, the CPU can transmit various interactive data between the internal structures of the computer device, and so on. The communication interface 802 can optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.), which can be used to send and receive data under the control of the processor 801; the communication interface 802 can also be used for the transmission and interaction of data within the computer device. The memory 803 (Memory) is a memory device in the computer device, which is used to store programs and data. It can be understood that the memory 803 here can include both the built-in memory of the computer device and the extended memory supported by the computer device. The memory 803 provides a storage space, which stores the operating system of the computer device, including but not limited to: Android system, iOS system, Windows Phone system, etc., which is not limited in this application.

[0200] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, a computer program suitable for being loaded and executed by the processor 801 is also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0201] In one embodiment, the processor 801 performs the following operations by running the computer program in the memory 803:

[0202] Obtaining a facial identity feature of a first object and a video to be processed, wherein the facial identity feature of the first object is obtained by performing facial feature recognition on a source image, the source image includes the face of the first object, and the facial identity feature is a facial feature used to distinguish the identity of the object; the video to be processed includes M facial video frames, each facial video frame includes the face of the second object, and M is an integer greater than 1;

[0203] Extracting displacement features of a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame, wherein the set of facial image frames associated with any facial video frame includes N facial image frames, where N is an integer greater than 1 and less than M;

[0204] Calling the face-swapping model to perform a first face-swapping process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-swapping result for each facial video frame;

[0205] Based on the face-swapping results of the M facial video frames, a face-swapping video corresponding to the video to be processed is generated.

[0206] As an optional embodiment, the processor 801 calls the face-changing model to perform a first face-changing process on the displacement feature associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity feature of the first object, and the process of obtaining the face-changing result of each facial video frame includes:

[0207] Down-sample and encode the set of facial image frames associated with the i-th facial video frame to obtain the latent space features corresponding to the i-th facial video frame, where i is a positive integer less than or equal to M;

[0208] splicing the displacement feature associated with the i-th facial video frame with the facial identity feature of the first object to obtain a splicing feature of the i-th facial video frame;

[0209] Fusing the splicing features of the i-th facial video frame and the latent space features corresponding to the i-th facial video frame to obtain the fusion features of the i-th facial video frame;

[0210] The fused features of the i-th facial video frame are upsampled and decoded to obtain the face-changing result of the i-th facial video frame.

[0211] As an optional embodiment, the processor 801 extracts displacement features from the set of facial image frames associated with each facial video frame, and the process of obtaining the displacement features associated with each facial video frame includes:

[0212] Obtain a set of facial image frames associated with the i-th facial video frame, where i is a positive integer less than or equal to M;

[0213] Extract key points from each face image frame in the face image frame set associated with the i-th face video frame to obtain position features of key points of each face image frame;

[0214] Through the temporal relationship of the facial image frames associated with the ith facial video frame and the positional features of the key points of each facial image frame, the displacement features of each key point are extracted to obtain the displacement features associated with the ith facial video frame.

[0215] As an optional embodiment, the processor 801 extracts displacement features of each key point through the time sequence relationship of the facial image frames associated with the i-th facial video frame and the position features of the key points of each facial image frame, and obtains the specific embodiment of the displacement features associated with the i-th facial video frame as follows:

[0216] By using the temporal relationship of the facial image frames associated with the i-th facial video frame and the positional features of the key points of each facial image frame, displacement features of each key point are extracted to obtain a feature extraction result;

[0217] The feature extraction result is transformed into a dimension to obtain the displacement feature associated with the i-th facial video frame.

[0218] As an optional embodiment, the M facial video frames are continuous video frames in the video to be processed; the specific embodiment in which the processor 801 obtains the facial image frame set associated with the i-th facial video frame is:

[0219] If i is less than N, a facial image frame set associated with the i-th facial video frame is generated based on the first i facial video frames, and there are at least two identical facial video frames in the facial image frame set associated with the i-th facial video frame;

[0220] If i is greater than or equal to N, then the i-N+1th facial video frame to the i-th facial video frame in the M facial video frames are added to the same set to obtain a facial image frame set associated with the i-th facial video frame.

[0221] In another embodiment, the processor 801 performs the following operations by running the computer program in the memory 803:

[0222] Obtaining a facial identity feature of a first object and a sample video, wherein the facial identity feature of the first object is obtained by performing facial feature recognition on a source image, the source image includes the face of the first object, and the facial identity feature is a facial feature used to distinguish the identity of the object; the sample video includes M facial video frames, each facial video frame includes the face of the first object, and M is an integer greater than 1;

[0223] Extracting displacement features from a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame, wherein the set of facial image frames associated with any facial video frame includes N facial image frames, each facial image frame includes a face of the second object, and N is an integer greater than 1 and less than M;

[0224] Calling the face-changing model to be optimized to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-changing result for each facial video frame;

[0225] The face-changing model to be optimized is optimized through the face-changing results of each facial video frame to obtain an optimized face-changing model.

[0226] As an optional embodiment, the M facial video frames are continuous video frames in the sample video; the processor 801 extracts displacement features from a facial image frame set associated with each facial video frame, and the process of obtaining the displacement features associated with each facial video frame includes:

[0227] Obtaining a facial video frame set associated with the i-th facial video frame, the facial video frame set associated with the i-th facial video frame includes N facial video frames, the N facial video frames are consecutive video frames among the M facial video frames, and i is a positive integer less than or equal to M;

[0228] Using the facial image frames of the second object, performing a second face-changing process on each facial video frame associated with the i-th facial video frame to obtain N facial image frames;

[0229] Adding N facial image frames to the same set to obtain a facial image frame set associated with the i-th facial video frame;

[0230] Based on a set of facial image frames associated with the ith facial video frame, a displacement feature associated with the ith facial video frame is extracted.

[0231] As an optional embodiment, the processor 801 optimizes the face-changing model to be optimized according to the face-changing results of each face video frame, and a specific embodiment of obtaining the optimized face-changing model is as follows:

[0232] The face-swapping result of each face video frame is discriminated by a discriminator to obtain M discrimination results of the face-swapping result; the discrimination result of any face-swapping result is used to indicate whether the face-swapping result is a synthetic image;

[0233] Based on the discrimination results of the M face-changing results and the M face-changing results, the generative adversarial network is optimized to obtain an optimized face-changing model. The generative adversarial network includes a discriminator and a face-changing model to be optimized.

[0234] As an optional embodiment, the processor 801 optimizes the generative adversarial network based on the discrimination results of the M face-changing results and the M face-changing results, and the process of obtaining the optimized face-changing model includes:

[0235] The discrimination loss is constructed by the difference between the discrimination result of the j-th face-swapping result and the discrimination result of the j-th face video frame, where j is a positive integer less than or equal to M;

[0236] Based on the difference between the jth face-swapping result and the jth face video frame, construct the face-swapping loss;

[0237] The generative adversarial network is optimized based on the discriminative loss and face-changing loss.

[0238] As an optional embodiment, the processor 801 constructs a specific embodiment of the face-swapping loss based on the difference between the j-th face-swapping result and the j-th face video frame as follows:

[0239] The features of the j-th face-changing result and the j-th face video frame at at least two levels are respectively extracted by the convolutional neural network, and the features at different levels are output by different convolutional neural network layers in the convolutional neural network;

[0240] The feature difference between the j-th face-swapping result and the j-th face video frame at each level is summed up to obtain the feature loss between the j-th face-swapping result and the j-th face video frame;

[0241] Based on the pixel value difference between the j-th face-swapping result and the j-th face video frame, construct the pixel-level loss between the j-th face-swapping result and the j-th face video frame;

[0242] The identity loss of the jth face-swapping result is calculated by the cosine similarity between the facial identity features of the jth face-swapping result and the jth face video frame;

[0243] The feature loss, pixel-level loss, identity loss and adversarial loss are weightedly summed to obtain the face-changing loss; the adversarial loss is constructed based on the discrimination result of the j-th face-changing result.

[0244] Based on the same inventive concept, the principles and beneficial effects of solving the problems provided by the computer device in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems provided by the data processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0245] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the data processing method of the above method embodiment.

[0246] The embodiment of the present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned data processing method.

[0247] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0248] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0249] In the embodiments of the present application, the "module" or "unit" referred to refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0250] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the readable storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0251] What is disclosed above is only a preferred embodiment of the present application, and it certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiment and equivalent changes made according to the claims of the present application are still within the scope covered by the application.

Claims

1. A model training method, characterized in that: The method comprises: Acquire a facial identity feature of a first object and a sample video, wherein the facial identity feature of the first object is obtained by performing facial feature recognition on a source image, the source image includes the face of the first object, and the facial identity feature is a facial feature used to distinguish the identity of the object; the sample video includes M facial video frames, each facial video frame includes the face of the first object, and M is an integer greater than 1; A displacement feature is extracted from a set of facial image frames associated with each facial video frame to obtain a displacement feature associated with each facial video frame, wherein the set of facial image frames associated with any facial video frame includes N facial image frames, where N is an integer greater than 1 and less than M. The process of determining the set of facial image frames associated with the ith facial video frame includes: obtaining a set of facial video frames associated with the ith facial video frame, wherein the set of facial video frames associated with the ith facial video frame includes N facial video frames, wherein the N facial video frames are continuous video frames among the M facial video frames, and i is a positive integer less than or equal to M; performing a second face-changing process on each facial video frame associated with the ith facial video frame using the facial image frames of the second object to obtain N facial image frames; and adding the N facial image frames to the same set to obtain the set of facial image frames associated with the ith facial video frame; Calling the face-changing model to be optimized to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-changing result for each facial video frame; The face-changing model to be optimized is optimized through the face-changing results of each facial video frame to obtain an optimized face-changing model.

2. The method according to claim 1, characterized in that The M facial video frames are continuous video frames in the sample video; the process of extracting displacement features from a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame includes: Based on the set of facial image frames associated with the i-th facial video frame, a displacement feature associated with the i-th facial video frame is extracted.

3. The method according to claim 1, characterized in that The step of optimizing the face-changing model to be optimized by using the face-changing results of each face video frame to obtain the optimized face-changing model comprises: The face-swapping result of each face video frame is discriminated by a discriminator to obtain M discrimination results of the face-swapping result; the discrimination result of any face-swapping result is used to indicate whether the face-swapping result is a synthetic image; Based on the discrimination results of the M face-changing results and the M face-changing results, a generative adversarial network is optimized to obtain an optimized face-changing model, wherein the generative adversarial network includes the discriminator and the face-changing model to be optimized.

4. The method according to claim 3, characterized in that The process of optimizing the generative adversarial network based on the discrimination results of the M face-changing results and the M face-changing results to obtain an optimized face-changing model includes: The discrimination loss is constructed by the difference between the discrimination result of the j-th face-swapping result and the discrimination result of the j-th face video frame, where j is a positive integer less than or equal to M; Constructing a face-swapping loss based on a difference between the j-th face-swapping result and the j-th face video frame; The generative adversarial network is optimized according to the discriminant loss and the face-changing loss.

5. The method according to claim 4, characterized in that The constructing the face-swapping loss based on the difference between the j-th face-swapping result and the j-th face video frame comprises: Extracting features of the j-th face-swapping result and the j-th face video frame at at least two levels respectively by using a convolutional neural network, wherein features at different levels are output by different convolutional neural network layers in the convolutional neural network; Summing up the feature differences between the j-th face-swapping result and the j-th face video frame at each level to obtain a feature loss between the j-th face-swapping result and the j-th face video frame; Based on the pixel value difference between the j-th face-swapping result and the j-th face video frame, construct a pixel-level loss between the j-th face-swapping result and the j-th face video frame; Calculating the identity loss of the j-th face-swapping result by using the facial identity feature of the j-th face-swapping result and the cosine similarity between the j-th face video frame; The feature loss, the pixel-level loss, the identity loss and the adversarial loss are weightedly summed to obtain the face-changing loss; the adversarial loss is constructed based on the discrimination result of the j-th face-changing result.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining a facial identity feature of a first object and a video to be processed, wherein the facial identity feature of the first object is obtained by performing facial feature recognition on a source image, the source image includes the face of the first object, and the facial identity feature is a facial feature used to distinguish the identity of the object; the video to be processed includes M facial video frames, each facial video frame includes the face of the second object, and M is an integer greater than 1; Extracting displacement features of a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame, wherein the set of facial image frames associated with any facial video frame includes N facial image frames, where N is an integer greater than 1 and less than M; Calling the optimized face-changing model to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-changing result for each facial video frame; Based on the face-swapping results of the M facial video frames, a face-swapping video corresponding to the video to be processed is generated.

7. The method according to claim 6, characterized in that The process of calling the optimized face-changing model to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object to obtain the face-changing result of each facial video frame includes: Down-sampling and encoding a set of facial image frames associated with the i-th facial video frame to obtain a latent space feature corresponding to the i-th facial video frame, where i is a positive integer less than or equal to M; splicing the displacement feature associated with the i-th facial video frame with the facial identity feature of the first object to obtain a splicing feature of the i-th facial video frame; fusing the splicing features of the i-th facial video frame and the latent space features corresponding to the i-th facial video frame to obtain the fused features of the i-th facial video frame; The fusion features of the i-th facial video frame are up-sampled and decoded to obtain a face-changing result of the i-th facial video frame.

8. The method according to claim 6, characterized in that The process of extracting displacement features from a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame includes: Obtain a set of facial image frames associated with the i-th facial video frame, where i is a positive integer less than or equal to M; Extract key points from each facial image frame in the facial image frame set associated with the i-th facial video frame to obtain position features of key points of each facial image frame; By using the temporal relationship of the facial image frames associated with the ith facial video frame and the positional features of the key points of each facial image frame, displacement features of each key point are extracted to obtain the displacement features associated with the ith facial video frame.

9. The method according to claim 8, characterized in that The method extracts displacement features of each key point through the temporal relationship of the facial image frames associated with the i-th facial video frame and the position features of the key points of each facial image frame to obtain the displacement features associated with the i-th facial video frame, including: Extracting displacement features of each key point based on the temporal relationship of the face image frames associated with the i-th face video frame and the position features of the key points of each face image frame to obtain a feature extraction result; The feature extraction result is dimensionally transformed to obtain a displacement feature associated with the i-th facial video frame.

10. The method according to claim 8, characterized in that The M facial video frames are continuous video frames in the video to be processed; and the step of obtaining a facial image frame set associated with the i-th facial video frame includes: If i is less than N, a facial image frame set associated with the i-th facial video frame is generated based on the first i facial video frames, and there are at least two identical facial video frames in the facial image frame set associated with the i-th facial video frame; If i is greater than or equal to N, then the i-N+1th facial video frame to the i-th facial video frame in the M facial video frames are added to the same set to obtain a facial image frame set associated with the i-th facial video frame.

11. A model training device, characterized in that: The model training device comprises: an acquisition unit, configured to acquire a facial identity feature of a first object and a sample video, wherein the facial identity feature of the first object is obtained by performing facial feature recognition on a source image, the source image includes the face of the first object, and the facial identity feature is a facial feature used to distinguish the identity of the object; the sample video includes M facial video frames, each facial video frame includes the face of the first object, and M is an integer greater than 1; The processing unit is used to extract displacement features from a set of facial image frames associated with each facial video frame to obtain displacement features associated with each facial video frame, wherein the set of facial image frames associated with any facial video frame includes N facial image frames, where N is an integer greater than 1 and less than M. The process of determining the set of facial image frames associated with the ith facial video frame includes: obtaining a set of facial video frames associated with the ith facial video frame, wherein the set of facial video frames associated with the ith facial video frame includes N facial video frames, wherein the N facial video frames are continuous video frames among the M facial video frames, and i is a positive integer less than or equal to M; performing a second face-changing process on each facial video frame associated with the ith facial video frame using facial image frames of a second object to obtain N facial image frames; and adding the N facial image frames to the same set to obtain the set of facial image frames associated with the ith facial video frame; and for calling the face-changing model to be optimized to perform a first face-changing process on the displacement features associated with each facial video frame, the facial image frame set associated with the facial video frame, and the facial identity features of the first object, to obtain a face-changing result for each facial video frame; And it is used to optimize the face-changing model to be optimized through the face-changing results of each facial video frame to obtain the optimized face-changing model.

12. A computer device, characterized in that: include: a memory, wherein a computer program is stored in the memory; A processor, used to load the computer program to implement the model training method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the model training method as described in any one of claims 1-10.

14. A computer program product, characterized in that The computer program product comprises a computer program, which is suitable for being loaded by a processor and executing the model training method as described in any one of claims 1-10.

Images