A live real-time face replacement method, medium and system

Abstract

The application provides a live real-time face replacement method, medium and system, and belongs to the technical field of face replacement. The live real-time face replacement method comprises the following steps: S10, obtaining a previous frame and a current frame in a live video stream, and obtaining a predicted frame according to the previous frame and the current frame; S20, performing face replacement on the predicted frame, that is, replacing a target face in the predicted frame with a required face, wherein the required face comes from a preset; S30, obtaining a next frame in the live video stream, if a non-face area similarity between the next frame and the predicted frame is greater than a similarity threshold, then replacing the next frame with the predicted frame; otherwise, performing face replacement on the next frame; and S40, repeating steps S10 to S30 until the live broadcast ends. The live real-time face replacement method, medium and system can solve the technical problem that serial live face replacement operations affect the real-time degree and smoothness of the entire live video stream.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of face replacement, and particularly relates to a live real-time face replacement method, medium and system. BACKGROUND

[0002] Face changing technology is an artificial intelligence image processing technology, which can extract the face information of a person, and then match it with the face information of another person, so as to generate a new face image with the features of both. In recent years, with the development of deep learning technology, face synthesis algorithms based on neural networks have gradually become mainstream.

[0003] The Chinese invention patent (application number CN201811163244.6) with the publication number CN110969673A discloses a live face changing interaction implementation method, storage medium, device and system. The method includes voting on multiple candidate faces based on a barrage voting method; selecting the candidate face with the most votes as the target face; performing face key point detection on the target face based on a face key point detection algorithm, and performing triangular partitioning on the detected key points to determine the triangular region of the target face; performing face key point detection on the anchor face to determine the triangular region of the anchor face; and covering the pixels of the triangular region of the target face to the triangular region of the anchor face through affine transformation of the triangle.

[0004] The above-mentioned invention patent provides face changing in the live process, but the face changing operation and the live stream are serial, which affects the real-time degree and smoothness of the entire live video stream. SUMMARY

[0005] Therefore, the application provides a live real-time face replacement method, medium and system, which can solve the technical problem that the serial live face changing operation affects the real-time degree and smoothness of the entire live video stream.

[0006] The application is implemented as follows:

[0007] The first aspect of the application provides a live real-time face replacement method, which includes the following steps:

[0008] S10, obtaining a previous frame and a current frame in a live video stream, and obtaining a predicted frame according to the previous frame and the current frame;

[0009] S20, performing face changing on the predicted frame, i.e., replacing the target face in the predicted frame with a required face, wherein the required face comes from a preset;

[0010] S30, obtaining a next frame in the live video stream, if a non-face area similarity of the next frame and the predicted frame is greater than a similarity threshold, using the predicted frame to replace the next frame, otherwise, replacing a target face in the next frame with a required face;

[0011] S40, repeating steps S10 to S30 until the live broadcast ends.

[0012] On the basis of the above technical solutions, the live real-time face replacement method can be further improved as follows:

[0013] The step of obtaining a previous frame and a current frame in the live video stream and obtaining a predicted frame according to the previous frame and the current frame specifically comprises:

[0014] obtaining the current frame and the previous frame from a live source;

[0015] preprocessing the current frame and the previous frame, the preprocessing including denoising, deformation correction, color correction, brightness adjustment, and contrast enhancement, wherein the previous frame after preprocessing is denoted as a first frame, and the current frame after preprocessing is denoted as a second frame;

[0016] extracting face features from the first frame and the second frame to obtain a face feature vector of the first frame and a face feature vector of the second frame, wherein the face feature vector of the first frame is denoted as a first face feature vector, and the face feature vector of the second frame is denoted as a second face feature vector;

[0017] calculating a change function from the previous frame to the current frame according to the first face feature vector and a face feature vector of the previous frame;

[0018] obtaining a predicted frame from the second frame by using the change function.

[0019] Further, the step of extracting face features from the first frame and the second frame specifically comprises:

[0020] in the first frame and the second frame, obtaining face recognition key points by using a Dlib face detection algorithm, and collecting coordinates of the face recognition key points as a vector as a face feature vector.

[0021] Further, in the step of calculating a change function from the previous frame to the current frame according to the first face feature vector and a face feature vector of the previous frame, the calculation formula of the change function is:

[0022] Fx = X2 - X1, wherein Fx is the change function, X1 represents the first face feature vector, and X2 represents the second face feature vector.

[0023] Further, the step of obtaining a predicted frame from the second frame by using the change function specifically comprises:

[0024] The third face feature vector is calculated according to the second face feature vector by using the change function;

[0025] The second frame is adjusted by using the third face feature vector to obtain a predicted frame;

[0026] The step of calculating the third face feature vector according to the second face feature vector by using the change function is expressed by a formula as follows:

[0027] X3 = X2 + Fx, wherein X3 represents the third feature vector.

[0028] Further, the step of adjusting the second frame by using the third face feature vector to obtain a predicted frame specifically includes:

[0029] In the second frame, a Dlib face detection algorithm is used to obtain face recognition key points, and the second frame is marked as a basic key point set;

[0030] According to the basic key point set, the second frame is divided into a plurality of triangular patches;

[0031] According to the numbering of the basic key point set, the coordinates of each point in the third face feature vector are adjusted to the coordinates of each key point in the basic key point set, and the adjustment of the plurality of triangular patches is driven;

[0032] The adjusted second frame is taken as the predicted frame.

[0033] Further, the step of using the predicted frame to replace the next frame if the similarity between the next frame and the predicted frame is greater than the similarity threshold includes the following specific steps of judging the similarity between the next frame and the predicted frame:

[0034] The next frame is preprocessed to obtain a fourth frame;

[0035] The face in the fourth frame is removed to obtain a background image of the fourth frame, denoted as a fourth background image;

[0036] The face in the predicted frame is removed to obtain a background image of the predicted frame, denoted as a third background image;

[0037] The similarity between the third background image and the fourth background image is calculated as the similarity between the next frame and the predicted frame.

[0038] Further, the face changing method is any one of DeepFake, FaceSwap and Face2Face algorithms.

[0039] The second aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores program instructions, and the program instructions are used to execute the live real-time face replacement method.

[0040] The third aspect of the present application provides a live real-time face replacement system, which comprises the computer readable storage medium.

[0041] Compared with the prior art, the live real-time face replacement method, medium and system provided by the present application have the following beneficial effects: since the face replacement is performed on each frame in the live video stream in real time, the real-time degree and smoothness of the entire live video stream are affected, therefore, the predicted frame is obtained in advance according to the current frame, the face replacement is performed on the predicted frame, when the live video stream is played to the next frame, the similarity analysis of the non-face area is performed on the next frame and the predicted frame, and when the similarity condition is met, the predicted frame on which the face replacement has been completed is directly used to replace the next frame; since the action of the live host is not large during the live broadcast, the difference between the adjacent two frames in the formed live video stream is not large, and most of them meet the requirement of the similarity condition, therefore, by using the method in the present application, the face replacement and the playing of the next frame can be operated in parallel, the face replacement efficiency of each frame in the entire live video stream is greatly improved, and the real-time degree and smoothness of the live video stream are improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0043] Figure 1 The flowchart of the live real-time face replacement method provided by the present application is shown in the figure.

[0044] Figure 2 The 68 face recognition key points in the Dlib face detection algorithm. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0046] Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the application claimed, but merely represents selected embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the application.

[0047] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0048] In the description of the application, it should be understood that the terms "center", "longitudinal", "transverse", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc. indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, only for the purpose of facilitating the description of the application and simplifying the description, and do not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the application.

[0049] In addition, the terms "first" and "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first" and "second" can explicitly or implicitly include one or more of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0050] As Figure 1 shown, a live real-time face replacement method provided by the first aspect of the application is a flowchart, which comprises the following steps:

[0051] S10, obtaining the previous frame and the current frame in the live video stream, and obtaining the predicted frame according to the previous frame and the current frame;

[0052] S20, face replacement of the predicted frame, i.e. replacing the target face in the predicted frame with the required face, the required face coming from the preset;

[0053] S30, obtaining the next frame in the live video stream, if the similarity of the non-face area of the next frame and the predicted frame is greater than the similarity threshold, using the predicted frame to replace the next frame; otherwise, replacing the target face in the next frame with the required face;

[0054] S40, repeating steps S10 to S30 until the live broadcast ends.

[0055] The similarity threshold is 0.95.

[0056] The required face is obtained from the preset step, and the required face can be a modeled face or a face image of another person.

[0057] In this scheme, the steps of replacing the target face in the predicted frame with the required face or replacing the target face in the next frame with the required face are as follows:

[0058] When describing, the predicted frame or the next frame is taken as the original image,

[0059] Step 1: Face detection

[0060] The face detection algorithm is used to detect two or more faces in the original image and extract the faces from the original image. The face detection can use the Adaboost cascade classifier algorithm based on Haar features, or use the model based on deep convolutional neural network, such as Faster R-CNN, SSD, etc. These algorithms can quickly and accurately detect the position of the face. Assuming that the detected face is A, and the preset face is B.

[0061] Step 2: Feature extraction

[0062] The face key point detection algorithm based on deep learning, such as MTCNN or dlib, is used to detect the key points of the face and extract rich feature information. In the MTCNN algorithm, the convolutional neural network is used to extract the face frame, key point position, face pose angle and other information. In the dlib algorithm, the traditional cascade classifier is used for face key point detection and feature extraction. These features include the core features of the face such as eyes, nose, mouth, and other features such as the overall structure of the face and skin color. Assuming that the features of A and B are fa and fb respectively.

[0063] Step 3: Face alignment

[0064] The face alignment algorithm is used to ensure that the face features of A and B are highly aligned in position and size, so as to facilitate subsequent feature fusion and optimization. Assuming that the position and size of A and B obtained by the face alignment algorithm are consistent.

[0065] Step 4: Feature fusion

[0066] The features of A and B are fused into the same space to achieve the purpose of approximate features. Here we will use the Style Mixing technology in the StyleGAN2 generator, which linearly combines the latent vectors of the mapping network in the two feature sources, so that the features of the two feature sources can be fused with each other.

[0067] The Style Mixing technique can be represented by the following equation:

[0068]

[0069] where w is a random noise vector, k is the position of the fusion point, ψ As (w) and ψ Bs (w) are style networks based on A and B features.

[0070] Step 5: Image synthesis

[0071] The fused features are used to generate a final synthesized image through the generator neural network in StyleGAN2.

[0072] StyleGAN2 can be represented by the following equation:

[0073] x = G θ (w, y);

[0074] where x is the output image, G θ is the generator network, θ is the weight parameter of the generator, w is the random noise vector, and y is the vector controlling various style parameters in the synthesized image, such as pose, expression, lighting, etc.

[0075] Step 6: Reconstructing the image

[0076] According to the two original images and the synthesized image, use Blender software or similar software to perform image re-projection operation, paste the synthesized face image onto the original face to generate the final face-swapping image. In Blender, the steps for image re-projection can be represented by the following equation:

[0077] I out (x, y) = f(I in , M, H, x, y);

[0078] where I in is the input image, I out is the output image, M is the transformation matrix from the original image to the synthesized image, H is the projection matrix from the synthesized image to the original image, and f is the synthesis function.

[0079] In the above technical solution, the steps of obtaining the previous frame and the current frame in the live video stream and obtaining the predicted frame according to the previous frame and the current frame include:

[0080] Obtain the current frame and the previous frame from the live source;

[0081] Preprocess the current frame and the previous frame, the preprocessing including denoising, deformation correction, color correction, brightness adjustment, contrast enhancement, wherein the previous frame after preprocessing is recorded as a first frame, and the current frame after preprocessing is recorded as a second frame;

[0082] Face feature extraction is performed on the first frame and the second frame to obtain a face feature vector of the first frame and a face feature vector of the second frame, wherein the face feature vector of the first frame is recorded as a first face feature vector, and the face feature vector of the second frame is recorded as a second face feature vector;

[0083] A change function from the previous frame to the current frame is calculated according to the first face feature vector and the face feature vector of the previous frame;

[0084] A predicted frame is obtained from the second frame by using the change function.

[0085] Further, in the above technical solution, the step of performing face feature extraction on the first frame and the second frame is specifically:

[0086] In the first frame and the second frame, a Dlib face detection algorithm is used to obtain face recognition key points, and a coordinate set of the face recognition key points is taken as a face feature vector, wherein the face recognition key points of the Dlib face detection algorithm are as shown in Figure 2 ;

[0087] Further, in the above technical solution, in the step of calculating the change function from the previous frame to the current frame according to the first face feature vector and the face feature vector of the previous frame, the calculation formula of the change function is:

[0088] Fx=X2-X1,

[0089] In the formula, Fx is the change function, X1 represents the first face feature vector, and X2 represents the second face feature vector.

[0090] Further, in the above technical solution, in the step of obtaining a predicted frame from the second frame by using the change function, the step is specifically:

[0091] A third face feature vector is calculated from the second face feature vector by using the change function;

[0092] The second frame is adjusted using the third face feature vector to obtain a predicted frame;

[0093] The step of calculating the third face feature vector from the second face feature vector by using the change function is represented by the following formula:

[0094] X3=X2+Fx,

[0095] In the formula, X3 represents the third feature vector.

[0096] Further, in the above technical solution, the step of adjusting the second frame using the third face feature vector to obtain the predicted frame specifically includes:

[0097] In the second frame, the Dlib face detection algorithm is used to obtain face recognition key points, and the second frame is marked as a basic key point set;

[0098] According to the basic key point set, the second frame is divided into a plurality of triangular patches;

[0099] According to the numbering of the basic key point set, the coordinates of each point in the third face feature vector are adjusted to the coordinates of each key point in the basic key point set, and the adjustment of the plurality of triangular patches is driven;

[0100] The adjusted second frame is used as the predicted frame.

[0101] Further, in the above technical solution, if the similarity of the non-face region of the next frame and the predicted frame is greater than the similarity threshold, the step of using the predicted frame to replace the next frame includes the specific steps of judging the similarity of the next frame and the predicted frame:

[0102] The next frame is preprocessed to obtain a fourth frame;

[0103] The face in the fourth frame is removed to obtain a background image of the fourth frame, denoted as a fourth background image;

[0104] The face in the predicted frame is removed to obtain a background image of the predicted frame, denoted as a third background image;

[0105] The similarity of the third background image and the fourth background image is calculated as the similarity of the next frame and the predicted frame.

[0106] Further, in the above technical solution, the face changing method is any one of DeepFake, FaceSwap and Face2Face algorithms. Deepfake (Deepfake technology) is a machine learning model called "Generative Adversarial Network" (GAN) that combines and superimposes pictures or videos onto source pictures or videos, using neural network technology for large sample learning, artificial intelligence technology that splices and synthesizes false content from a person's voice, facial expressions and body movements. The most common way of deepfake is AI face changing technology, in addition to voice simulation, face synthesis, video generation, etc. Its emergence makes it possible to tamper with or generate highly realistic and difficult to distinguish audio and video content, and the observer ultimately cannot distinguish the truth and falsehood by the naked eye. FaceSwap and Face2Face are tools that use deep learning to recognize faces in pictures and videos and change faces.

[0107] The second aspect of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores program instructions, and the program instructions are used to execute the live real-time face replacement method.

[0108] The third aspect of the present application provides a live real-time face replacement system, wherein the system comprises the computer readable storage medium.

[0109] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A live real-time face replacement method, characterized in that, The method comprises the following steps: S10, obtaining a previous frame and a current frame in a live video stream, and obtaining a predicted frame according to the previous frame and the current frame; S20, performing face replacement on the predicted frame, i.e., replacing a target face in the predicted frame with a required face, the required face being from a preset; S30, obtaining a next frame in the live video stream, if a non-face area similarity between the next frame and the predicted frame is greater than a similarity threshold, using the predicted frame to replace the next frame; otherwise, replacing a target face in the next frame with the required face; S40, repeating steps S10 to S30 until the live broadcast ends.

2. The live real-time face replacement method of claim 1, wherein, The step of obtaining a previous frame and a current frame in a live video stream, and obtaining a predicted frame according to the previous frame and the current frame, specifically comprises: obtaining the current frame and the previous frame from a live source; performing preprocessing on the current frame and the previous frame, the preprocessing comprising denoising, deformation correction, color correction, brightness adjustment, and contrast enhancement, wherein the preprocessed previous frame is denoted as a first frame, and the preprocessed current frame is denoted as a second frame; performing face feature extraction on the first frame and the second frame to obtain a face feature vector of the first frame and a face feature vector of the second frame, wherein the face feature vector of the first frame is denoted as a first face feature vector, and the face feature vector of the second frame is denoted as a second face feature vector; calculating a change function from the previous frame to the current frame according to the first face feature vector and the second face feature vector; obtaining a predicted frame according to the second frame by using the change function.

3. The live real-time face replacement method of claim 2, wherein, The step of performing face feature extraction on the first frame and the second frame specifically comprises: in the first frame and the second frame, obtaining face recognition key points by using a Dlib face detection algorithm, and collecting coordinates of the face recognition key points into a vector as a face feature vector.

4. The live real-time face replacement method of claim 3, wherein, In the step of calculating a change function from the previous frame to the current frame according to the first face feature vector and the second face feature vector, a calculation formula of the change function is as follows: Fx = X2 - X1, wherein Fx is the change function, X1 represents the first face feature vector, and X2 represents the second face feature vector.

5. The live real-time face replacement method of claim 4, wherein, The step of obtaining a predicted frame according to the second frame by using the change function specifically comprises: calculating a third face feature vector according to the second face feature vector by using the change function; adjusting the second frame by using the third face feature vector to obtain the predicted frame; wherein the step of calculating a third face feature vector according to the second face feature vector by using the change function is expressed by a formula as follows: X3 = X2 + Fx, wherein X3 represents the third feature vector.

6. The live real-time face replacement method of claim 5, wherein, The step of adjusting the second frame by using the third face feature vector to obtain the predicted frame specifically comprises: in the second frame, obtaining face recognition key points by using a Dlib face detection algorithm, and marking the second frame as a basic key point set; dividing the second frame into a plurality of triangular patches according to the basic key point set; adjusting coordinates of each point in the third face feature vector to coordinates of each key point in the basic key point set, and driving adjustment of the plurality of triangular patches according to the basic key point set; using the adjusted second frame as the predicted frame.

7. The live real-time face replacement method of claim 6, wherein, The specific steps of judging the similarity between the next frame and the predicted frame in the step of replacing the next frame with the predicted frame when the similarity between the next frame and the non-face area of the predicted frame is greater than the similarity threshold value include: performing preprocessing on the next frame to obtain a fourth frame; performing face removal on the fourth frame to obtain a background image of the fourth frame, denoted as a fourth background image; performing face removal on the predicted frame to obtain a background image of the predicted frame, denoted as a third background image; calculating the similarity between the third background image and the fourth background image as the similarity between the next frame and the predicted frame.

8. The live real-time face replacement method according to any one of claims 1-7, characterized in that, The face replacement method is any one of DeepFake, FaceSwap and Face2Face algorithms.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program instructions, and the program instructions are run by the computer to execute the live real-time face replacement method according to any one of claims 1-8.

10. A live real-time face replacement system, characterized in that, The computer readable storage medium according to claim 9. The computer readable storage medium according to claim 9.

Images