Video face replacement method and device and electronic equipment

Abstract

The invention provides a video face replacement method and device and electronic equipment. According to the method and the device, face replacement processing is carried out on the original face region to be replaced in the video frame by utilizing at least two different face replacement models existing currently so as to obtain the face replacement result of the original face region, and the optimal face replacement result is selected based on the score of each face replacement result, so that the face replacement efficiency is improved. Replacing the original face region to be replaced in the video frame by using the optimal face replacement result; according to the method and the device, face replacement is carried out on the same original face region to be replaced through different face replacement models, so that the optimal face replacement result is selected, it is ensured that various face angles of the original face of the target object in the original face region to be replaced can be subjected to optimal replacement processing, and the video quality after replacement is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a video face replacement method, device and electronic equipment. Background Art

[0002] At present, in many practical scenarios such as game entertainment and film and television production, there is an application demand to replace the face of a specified object in the original video with the target face. However, in different video frames of the original video, the face of the specified object usually appears at different angles or is even partially blocked. In this case, the effect of face replacement for the specified object is poor. Summary of the invention

[0003] In view of this, the present application provides a video face replacement method, device and electronic device to improve the quality of the video obtained by face replacement.

[0004] The technical solutions provided by this application are as follows: According to an embodiment of the first aspect of the present application, a video face replacement method is provided, the method comprising: For an original face region to be replaced in a video frame, using at least two different face replacement models currently available, face replacement processing is performed on the original face region to obtain a face replacement result of the original face region; the face replacement result represents a face region in which the original face of the target object in the original face region is replaced with the target face; Scoring the face replacement results output by each face replacement model, and selecting the optimal face replacement result based on the scores of each face replacement result; The original face region to be replaced in the video frame is replaced with the optimal face replacement result.

[0005] According to an embodiment of the second aspect of the present application, a video face replacement device is provided, the device comprising: A first replacement unit is used to perform face replacement processing on an original face region to be replaced in a video frame using at least two different face replacement models currently available, so as to obtain a face replacement result of the original face region; the face replacement result represents a face region in which an original face of a target object in the original face region is replaced with a target face; A scoring selection unit, used to score the face replacement results output by each face replacement model, and select the optimal face replacement result based on the scores of each face replacement result; The second replacement unit is used to replace the original face area to be replaced in the video frame with the optimal face replacement result.

[0006] According to an embodiment of the third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the computer program.

[0007] It can be seen from the above technical scheme that the present application targets the original face area to be replaced in the video frame, utilizes at least two different face replacement models currently available, performs face replacement processing on the original face area to obtain the face replacement result of the original face area, and selects the optimal face replacement result based on the score of each face replacement result, and utilizes the optimal face replacement result to replace the original face area to be replaced in the video frame; performs face replacement on the same original face area to be replaced by different face replacement models, and then selects the optimal face replacement result, thereby ensuring that various facial angles of the original face of the target object in the original face area to be replaced can be optimally replaced, thereby improving the quality of the replaced video. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0009] Figure 1 A flowchart of a video face replacement method provided in an embodiment of the present application; Figure 2 A schematic diagram of the overall architecture of the video face replacement method provided in the embodiment of the present application; Figure 3 A schematic diagram of the overall process of the video face replacement method provided in the embodiment of the present application; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application; Figure 5 A structural diagram of a video face replacement device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0010] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0011] Please refer to Figure 1 , Figure 1 This is a flowchart of the video face replacement method provided in an embodiment of the present application.

[0012] In this embodiment, the video face replacement method can be applied to electronic devices with video processing capabilities, such as mobile devices such as smart phones and tablet computers, or imaging devices such as smart cameras, and this application does not impose any restrictions on this.

[0013] like Figure 1 As shown, the method may include the following steps: Step 101, for an original face region to be replaced in a video frame, using at least two different face replacement models currently available, perform face replacement processing on the original face region to obtain a face replacement result of the original face region.

[0014] Among them, the original face area to be replaced is the area that includes the face image (referred to as the original face) of the designated object (referred to as the target object) that needs to undergo face replacement; the face replacement result is the face area in which the original face of the target object in the original face area is replaced with the target face.

[0015] For example, the original face region to be replaced is a rectangular frame region where the face of the target object identified by the face detection model is located, and the rectangular frame region includes the original face of the target object and a part of the background region. Then, the face replacement result obtained by performing face replacement processing on the face region to be replaced can be a new rectangular frame region obtained by replacing the original face included in the rectangular frame region with the target face, and not changing the background region. In this embodiment, the method for determining the original face region to be replaced will be described in detail below and will not be repeated here.

[0016] In this embodiment, for the original face area to be replaced in the video frame, at least two different face replacement models can be used to perform face replacement processing on the original face area to be replaced, that is, the face area to be replaced is input into each face replacement model, and a face replacement result is obtained for each face replacement model.

[0017] Exemplarily, the optimal face angles required by at least two different face replacement models may be different. The optimal face angle may be the face angle at which the model is good at face replacement processing. For example, the optimal face angles required by different face replacement models may be different angles such as the front face, the side face, and the obscured face.

[0018] As an embodiment, each face replacement model has the best face replacement effect at the optimal face angle required by the model. Each face replacement model can be a face replacement model with the best replacement effect selected from different models that are good at the same angle (such as the front face).

[0019] After performing replacement processing on the original face region using at least two different face replacement models, multiple face replacement results of the original face region are obtained.

[0020] In this embodiment, the original face region to be replaced can be determined by the following steps: Extracting features from the reference face image of the target object and the face image regions included in the video frame to obtain reference face image features of the target object and image features of the face image regions; the face image region is determined based on image size information of the reference face image, and the face image region includes at least an original face region; Determine the facial image region corresponding to the image feature that meets the specified conditions according to the similarity calculation result between the reference facial image feature of the target object and the image feature of each facial image region; wherein the specified conditions include: the similarity calculation result is the maximum value among the similarity results of each image feature, and the similarity result is greater than a first specified value; The original face region included in the face image region corresponding to the image feature satisfying the specified condition is determined as the original face region to be replaced.

[0021] The following first introduces a method for determining each face image region included in a video frame.

[0022] In this embodiment, face image detection may be performed on the video frame to obtain each original face region included in the video frame; Determine a face image region according to the image size information of the reference face image of the target object obtained and each original face region included in the video frame; the face image region refers to a region including at least the original face region; The original face region to be replaced is determined according to the reference face image of the target object and each face image region.

[0023] Specifically, the video frames can be detected using a trained face image detection model to obtain a rectangular frame used to identify the area where each original face is located, as well as the position coordinates of five key points in each original face (such as the right eye pupil, left eye pupil, nose tip, right mouth corner, and left mouth corner), and the rectangular frame of the area where the original face is located is determined as the original face area.

[0024] After determining the original face regions in the video frame, a face image region corresponding to each original face region can be determined based on the image size information of the reference face image of the target object obtained and the original face regions included in the video frame.

[0025] In this embodiment, a reference facial image of the target object can be obtained in advance. The reference facial image can be a recent photo of the target object or a facial image of the target object directly captured from any video frame. This application does not impose any restrictions on this.

[0026] In order to facilitate the subsequent comparison of facial features, the specific method for determining the facial image area corresponding to the original facial area can be as follows: the facial image area is cut out from the video frame according to the aspect ratio of the obtained reference facial image, and each facial image area includes at least one original facial area, that is, a rectangular frame including the area where the original face is located.

[0027] This concludes the introduction to the method for determining the face image area.

[0028] After the facial image area is determined, the same feature extraction method is used to extract features from the reference facial image of the target object and the facial image areas included in the video frame to obtain the reference facial image features of the target object and the image features of the facial image areas.

[0029] Furthermore, similarity calculation may be performed between the reference facial image features of the target object and the image features of each facial image region to obtain a similarity calculation result.

[0030] If there is a facial image region in the video frame whose similarity result is greater than the first specified value, it indicates that there is an original face of the target object in the video frame, that is, there is an original face region to be replaced.

[0031] At this time, the original face region included in the face image region with the largest similarity calculation result can be determined as the original face region to be replaced.

[0032] In this embodiment, the method for calculating the similarity may be to calculate the cosine similarity between the feature values, for example, by calculating the cosine similarity using the following formula:

[0033] Where similarity represents cosine similarity, A and B represent the reference face image feature vector of the target object and the image feature vector of the face image region, respectively. i represents the i-th element in the feature vector A, B i Represents the i-th element in the feature vector B.

[0034] This concludes the description of the step of determining the original face region to be replaced.

[0035] Since the solution proposed in the embodiments of the present application is to replace the original face image of the target object included in the video that needs face replacement (referred to as the original video), therefore, before performing face replacement processing on the original face area to be replaced in the video frame using at least two different face replacement models currently available, it is necessary to first determine whether there is an original face image of the target object in the current video frame.

[0036] As an embodiment, the original video may be parsed to obtain a plurality of video frames corresponding to the original video. For each video frame, the above step of determining the original face region to be replaced is performed.

[0037] After performing face image detection on the current video frame to obtain the original face regions included in the video frame, the original face regions may be numbered in a specified order, such as from left to right.

[0038] In this embodiment, each video frame also records an index number value indicating whether the video frame includes a face area.

[0039] Specifically, the index number value can be recorded by the following method.

[0040] When there is no original face region to be replaced in the current video frame, the number value corresponding to the original face region to be replaced in the current video frame may be recorded, and the number value is the index number value recorded in the video frame.

[0041] When there is no original face area to be replaced in the current video frame, or when there is no original face area in the video frame, a second designated value different from the number value corresponding to any original face area can be recorded as the index number value corresponding to the video frame.

[0042] Before performing face replacement processing on the original face area to be replaced in a video frame using at least two different face replacement models, it is possible to first determine whether the video frame includes the original face area to be replaced based on the index number value recorded in the video frame.

[0043] Specifically, for each video frame, the recorded index number value in the video frame is detected. If the recorded index number value is the number value corresponding to any original face area in the video frame, it indicates that there is an original face area to be replaced in the video frame, that is, the original face area corresponding to the recorded index number value. At this time, the step of performing face replacement processing on the original face area to be replaced in the video frame can be continued by using at least two different face replacement models currently available.

[0044] If the recorded index number value is the second specified value, it indicates that there is no original face area to be replaced in the video frame, or there is no original face area in the video frame. At this time, the video frame does not need to be processed by face replacement, and the step of performing face replacement processing on the original face area to be replaced in the video frame by using at least two different face replacement models currently available can be refused.

[0045] In this embodiment, the second specified value is different from the numbering value of any original face area. For example, the numbering value of the original face area can be 1, 2, 3,...; and the second specified value can be set to a value different from the numbering value of any original face area, such as -1, etc., and this application does not impose any restrictions on this.

[0046] At this point, the description of step 101 ends, and step 102 is executed next.

[0047] Step 102: Score the face replacement results output by each face replacement model, and select the optimal face replacement result based on the scores of each face replacement result.

[0048] In this embodiment, after the face replacement results output by each face replacement model are obtained in step 101, the face replacement results output by each face replacement model may be scored to select the optimal face replacement result based on the scores of each face replacement result.

[0049] Specifically, for each face replacement result, a convolution operation may be performed on the grayscale value of each pixel in the face replacement result based on a Laplace operator to obtain a convolution operation result corresponding to the face replacement result; The variance of the convolution operation result corresponding to each face replacement result is used as the score of each face replacement result; the score of each face replacement result is used to indicate the image blur degree of each face replacement result, and the image blur degree is inversely correlated with the variance of the convolution operation result; The face replacement result with the highest score among all face replacement results is determined as the optimal face replacement result.

[0050] In this embodiment, each face replacement result may be first converted into a grayscale image, and further, a convolution operation may be performed on the grayscale value of each pixel in the face replacement result based on a Laplace operator to obtain a convolution operation result corresponding to the face replacement result.

[0051] After determining the convolution operation results corresponding to each face replacement result, the variance corresponding to each convolution operation result may be further determined, and the variance may be used as the score of each face replacement result.

[0052] The variance corresponding to each convolution operation result can be used to indicate the image blur degree of each face replacement result. The image blur degree is inversely correlated with the variance of the convolution operation result. That is, the larger the variance result, the lower the image blur degree, the clearer the image, the better the display effect, and the higher the score of the face replacement result.

[0053] In this embodiment, the face replacement result with the highest score may be determined as the optimal face replacement result.

[0054] At this point, the description of step 102 ends, and step 103 is executed next.

[0055] Step 103: Replace the original face region to be replaced in the video frame with the optimal face replacement result.

[0056] In this embodiment, after determining the optimal face replacement result corresponding to the video frame, the optimal face replacement result can be used to replace the original face area to be replaced in the video frame, that is, the original face area to be replaced in the video frame is replaced with the optimal face replacement result to obtain the video frame after face replacement, and further, the video frames after each face replacement and the video frames in the original video where the original face area to be replaced does not exist can be video encoded to obtain the face replacement video corresponding to the original video.

[0057] As an embodiment, a specific method of replacing the original face region to be replaced in the video frame with the optimal face replacement result may include: Based on the trained face restoration model, the optimal face replacement result is optimized to obtain the optimal face replacement result after image optimization; the original face area to be replaced in the video frame is replaced by using the optimal face replacement result after image optimization; Alternatively, the original face region to be replaced in the video frame is replaced using the optimal face replacement result; and image optimization is performed on all face regions included in the video frame based on the trained face restoration model.

[0058] In this embodiment, after the optimal face replacement result is determined, the optimal face replacement result can be further optimized. For example, the optimal face replacement result is optimized through a trained facial restoration model, and the original face area to be replaced in the video frame is replaced with the optimal face replacement result after image optimization, so as to obtain a video frame after face replacement.

[0059] In this embodiment, after determining the optimal face replacement result, the original face area to be replaced in the video frame can be replaced with the optimal face replacement result to obtain a reference video frame; further, based on the trained facial restoration model, all face areas included in the reference video frame (including the original face areas that do not need to be replaced and the optimal face replacement result) are optimized to obtain a video frame after face replacement.

[0060] The above two methods can be selected based on actual needs or server resource usage, and this application does not limit this.

[0061] This concludes the description of step 103.

[0062] As an embodiment, before performing face replacement processing on the original face region to be replaced in the video frame using at least two different face replacement models currently available, the method may further include: Based on the trained neural network model, classify and predict the image features of the video frame and the text features of the text information included in the video frame to obtain a classification prediction result for the video frame; If the classification prediction results of all video frames meet the normal video frame conditions, the step of performing face replacement processing on the original face area to be replaced in the video frame using at least two different face replacement models currently available is executed.

[0063] In this embodiment, before performing face replacement on the original video, it can be detected whether the original video is a video that allows face replacement operation, that is, it can be determined whether all video frames of the original video meet the normal video frame conditions.

[0064] Specifically, for each video frame included in the original video, text recognition can be performed on the video frame to obtain text information included in the video frame; for an audio file that may be included in the original video, the audio file can also be converted into corresponding text information.

[0065] Furthermore, feature extraction may be performed on the video frame to obtain image features of the video frame; and feature extraction may be performed on text information (including text information included in the video frame and text information converted from the audio file) to obtain text features of the text information.

[0066] The image features and text features are classified and predicted through the trained neural network model to obtain the classification prediction result for the video frame.

[0067] In this embodiment, inception_v3 can be used to extract image features of picture frames. After OCR text recognition and BERT text feature extraction, the image features and text features (to ensure that images, subtitles, etc. in the material are compliant at the same time) are used for image classification prediction through a neural network MLP multi-layer perceptron to obtain the final classification prediction result.

[0068] As an embodiment, the classification prediction result may include a normal type and an abnormal type, and the abnormal type is used to indicate that face replacement is not allowed for the video frame.

[0069] In this embodiment, the classification prediction results of the video frames all meet the normal video frame conditions, which means that the classification prediction results of all video frames in the original video are of normal type, which means that the original video is a video that allows face replacement. At this time, the step of performing face replacement processing on the original face area to be replaced in the video frame can be continued using at least two different face replacement models currently available.

[0070] If the classification prediction result of any video frame in the original video is an abnormal type, the subsequent face replacement step is rejected, and a prompt message can be generated to indicate that the original video is a video that does not allow face replacement.

[0071] This concludes Figure 1 Description of the video face replacement method.

[0072] The present application targets an original face region to be replaced in a video frame, uses at least two different face replacement models currently available, performs face replacement processing on the original face region to obtain a face replacement result of the original face region, selects an optimal face replacement result based on a score for each face replacement result, and replaces the original face region to be replaced in the video frame using the optimal face replacement result; performs face replacement on the same original face region to be replaced using different face replacement models, and then selects the optimal face replacement result, thereby ensuring that various facial angles of the original face of the target object in the original face region to be replaced can be optimally replaced, thereby improving the quality of the replaced video.

[0073] Furthermore, before performing face replacement on the original video, the present application also performs a pre-security compliance check to extract the image features and text features of each video frame in the original video, and classifies the extracted image features and text features through the trained neural network model to obtain predicted classification results. Face replacement is only allowed for the original video when the classification prediction results of the video frames meet the normal video frame conditions, effectively ensuring the compliance of the input material and the rational and compliant use of the technology.

[0074] Combine the following Figure 2 as well as Figure 3 The video face replacement method proposed in this application is described.

[0075] Please refer to Figure 2 , Figure 2 A schematic diagram of the overall architecture of the video face replacement method provided in an embodiment of the present application.

[0076] like Figure 2 As shown, the overall architecture of the method includes the following parts: 1. An input part, which is used to input the original video, the reference face image of the target object (the obtained face image of the target object that needs face replacement) and the target face image (the replaced face image) into an electronic device that executes the method.

[0077] 2. The video analysis part is used to analyze the original video to obtain an audio file, a frame rate of the original video, and multiple video frames.

[0078] 3. The security detection part is used to extract features of the images and text information included in the video frames, and classify and predict the image features and text features obtained by the feature extraction to determine the category of each video frame.

[0079] 4. The index calculation part is used for extracting the image features of the original face area included in the video frame and the image features of the reference face image of the target object for each video frame of the original video whose classification prediction results of the video frame all meet the normal video frame conditions, and performing face similarity calculation between the image features of the reference face image and the image features of each original face area, and determining the index number value of the original face area that meets the specified conditions. The specified conditions and index number values ​​have been described in detail above and will not be repeated here.

[0080] 5. The model processing part is used to replace the original face area to be replaced by at least two face replacement models (such as replacement model A, replacement model B and replacement model C), and determine the replacement result with the highest score through Laplace convolution calculation, and perform face repair on the replacement result with the highest score.

[0081] 6. The video synthesis part is used to encode the video frames after face replacement, the video frames without face replacement, and the audio files of the original video according to the frame rate of the original video.

[0082] 7. The output part is used to obtain the result of video synthesis, that is, the face replacement video corresponding to the original video.

[0083] This concludes Figure 2 Description.

[0084] Please refer to Figure 3 , Figure 3 A schematic diagram of the overall process of the video face replacement method provided in an embodiment of the present application.

[0085] like Figure 3 As shown, the video face replacement method proposed in this embodiment may specifically include the following steps: 1. Analyze the original video.

[0086] The input original video is parsed to separate each video frame, audio file, frame rate and other information of the original video.

[0087] 2. Check whether the original video is compliant material.

[0088] The Inception_v3 model is used to extract the image features of each video frame, and the text information in the video frame is recognized through optical character recognition (OCR), and the text features of the text information are extracted through the trained language model BERT. The image features and text features are classified and predicted through a multi-layer perceptron (MLP) (to ensure that the images, subtitles, etc. in the material are compliant at the same time), and the video frame is divided into normal type and abnormal type. If the predicted type of all video frames in the original video is normal type, continue to execute the following step 3. If the predicted type of any video frame is abnormal type, the process is terminated directly, and the original video is given as a video that does not allow face replacement.

[0089] 3. Align the pictures.

[0090] Detect the original face area in each video frame and number all faces in order from left to right, such as 1, 2, 3...

[0091] The face image region in each video frame is cut out according to the aspect ratio of the reference face image to avoid changes in facial features caused by stretching the image, and the face image region includes the original face region.

[0092] 4. Similarity calculation.

[0093] The facial features of all facial image regions and the reference facial image of the target object are extracted, and the cosine similarity between the facial image regions in the video frame and the facial features of the reference facial image is calculated.

[0094] 5. Index output.

[0095] A judgment is made based on the similarity calculated in step 4. When the similarity is greater than a first specified value (for example, the default value is 0.6, which can be adjusted according to actual conditions and can usually be set within the range of 0.5-0.7), the number value of the original face area included in the face image area with the highest similarity is determined as the index number value corresponding to the video frame. When the similarity is less than the first specified value, it is considered that there is no face to be replaced in the video frame. At this time, a second specified value, such as -1, is determined as the index number value corresponding to the video frame.

[0096] This step can automatically lock the position of the face that needs to be replaced, ensuring that no matter how many faces appear in the video or where the face that needs to be replaced appears, it can be accurately replaced.

[0097] 6. Index judgment.

[0098] Before face replacement, the index number value corresponding to each video frame is detected. If the index number value is -1, it indicates that there is no original face area to be replaced in the video frame, that is, the video frame does not need to be replaced.

[0099] If the index number value is greater than -1, such as 1, 2, 3..., it indicates that there is an original face area in the video frame that needs to be replaced. The original face area to be replaced is the original face area corresponding to the index number value. At this time, the subsequent replacement steps can be continued.

[0100] 7. Replacement processing.

[0101] Use multiple face replacement models (such as replacement model A, replacement model B, and replacement model C) to perform parallel face replacement processing on the original face area to be replaced, and simultaneously obtain the face replacement results of the original face area corresponding to the index number value of each different model (such as multiple models that are good at different angles such as front face, side face, occlusion, etc.).

[0102] 8. Score the replacement results.

[0103] The opencv Laplace operator is used to perform convolution operation on the face replacement results of different models to obtain the convolution operation result. The variance of the convolution operation result is used as the score of the face replacement result, and the face replacement result corresponding to the highest score is selected as the optimal face replacement result.

[0104] 9. Model repair processing.

[0105] By step 8, the face replacement process has been basically completed, but the facial details of the optimal face replacement result may not be natural enough, and the stitching position may not be smooth enough. The face details can be optimized through the face restoration model according to actual needs.

[0106] When repairing faces, only the best face replacement result can be optimized to avoid repairing all the original face areas in the entire video frame to reduce server resource consumption. You can also repair all the original face areas according to actual needs.

[0107] 10. Merge the videos, and encode the video frames after face replacement, the video frames that do not need face replacement, and the audio files of the original videos according to the video frame rate to obtain the face replacement video corresponding to the original video.

[0108] In this embodiment, some or all of the above steps can be selected to meet the video face replacement requirements according to actual needs and resource performance requirements.

[0109] This concludes Figure 3 Description.

[0110] Please refer to Figure 4 , Figure 4 It is a schematic structural diagram of an electronic device proposed in an embodiment of the present application. At the hardware level, the electronic device includes a processor, an internal bus, a network interface, and a computer-readable storage medium, and of course may also include hardware required for other services. The processor reads the corresponding computer program instructions from the computer-readable storage medium and runs them to form a terminal interaction device at the logical level. Of course, in addition to the software implementation method, the present application does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc., that is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0111] Please refer to Figure 5 , Figure 5 FIG. 1 is a structural diagram of a video face replacement device proposed in an embodiment of the present application. Figure 5 As shown, the device may include a data first replacement unit 501, a score selection unit 502, and a second replacement unit. Specifically, the device includes: The first replacement unit 501 is used to perform face replacement processing on the original face region to be replaced in the video frame using at least two different face replacement models currently available to obtain a face replacement result of the original face region; the face replacement result represents a face region in which the original face of the target object in the original face region is replaced with the target face; A scoring selection unit 502, used to score the face replacement results output by each face replacement model, and select the best face replacement result based on the scores of each face replacement result; The second replacement unit 503 is used to replace the original face area to be replaced in the video frame with the optimal face replacement result.

[0112] Optionally, the first replacement unit 501 is further used for: Extracting features from the reference face image of the target object and the face image regions included in the video frame to obtain reference face image features of the target object and image features of the face image regions; the face image region is determined based on image size information of the reference face image, and the face image region includes at least an original face region; Determine the facial image region corresponding to the image feature that meets the specified conditions according to the similarity calculation result between the reference facial image feature of the target object and the image feature of each facial image region; wherein the specified conditions include: the similarity calculation result is the maximum value among the similarity results of each image feature, and the similarity result is greater than a first specified value; Determine the original face region included in the face image region corresponding to the image feature that meets the specified condition as the original face region to be replaced; and / or, the optimal face angles required by at least two different face replacement models are different; And / or, the original face region in each video frame is numbered in a specified order, and an index number value is also recorded in each video frame; before performing face replacement processing on the original face region to be replaced in the video frame using at least two different face replacement models currently available, the first replacement unit 501 is further used to: For each video frame, the recorded index number value in the video frame is detected. If the recorded index number value is the number value corresponding to any original face region in the video frame, the step of performing face replacement processing on the original face region to be replaced in the video frame by using at least two different face replacement models currently available is continued; the original face region to be replaced refers to the original face region corresponding to the recorded index number value; If the recorded index number value is the second specified value, the step of performing face replacement processing on the original face region to be replaced in the video frame by using at least two different face replacement models currently available is rejected; the second specified value is different from the number value corresponding to any original face region, and the second specified value is used to indicate that there is no original face region to be replaced in the video frame, or there is no original face region in the video frame; And / or, the scoring selection unit 502 is specifically used for: For each face replacement result, a convolution operation is performed on the grayscale value of each pixel in the face replacement result based on the Laplace operator to obtain a convolution operation result corresponding to the face replacement result; The variance of the convolution operation result corresponding to each face replacement result is used as the score of each face replacement result; the score of each face replacement result is used to indicate the image blur degree of each face replacement result, and the image blur degree is inversely correlated with the variance of the convolution operation result; The face replacement result with the highest score among all face replacement results is determined as the optimal face replacement result; And / or, before performing face replacement processing on the original face region to be replaced in the video frame using at least two different face replacement models currently available, the first replacement unit 501 is further configured to: Based on the trained neural network model, classify and predict the image features of the video frame and the text features of the text information included in the video frame to obtain a classification prediction result for the video frame; If the classification prediction results of all video frames meet the normal video frame condition, then executing the step of performing face replacement processing on the original face region to be replaced in the video frame using at least two different face replacement models currently available; And / or, the second replacement unit 503 is specifically used for: Based on the trained face restoration model, the optimal face replacement result is image optimized to obtain the optimal face replacement result after image optimization; the original face area to be replaced in the video frame is replaced with the optimal face replacement result after image optimization; Alternatively, the original face region to be replaced in the video frame is replaced with the optimal face replacement result; and image optimization is performed on all face regions included in the video frame based on the trained face restoration model.

[0113] This concludes Figure 5 Description of the face replacement device in the video.

[0114] Correspondingly, an embodiment of the present application further provides a computer-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed, the method disclosed in the above example of the present application can be implemented.

[0115] Exemplarily, the computer-readable storage medium may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the computer-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0116] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the scope of protection of the present application.

Claims

1. A video face replacement method, characterized in that: The method includes: For an original face region to be replaced in a video frame, using at least two different face replacement models currently available, face replacement processing is performed on the original face region to obtain a face replacement result of the original face region; the face replacement result represents a face region in which the original face of the target object in the original face region is replaced with the target face; Scoring the face replacement results output by each face replacement model, and selecting the optimal face replacement result based on the scores of each face replacement result; The original face region to be replaced in the video frame is replaced with the optimal face replacement result.

2. The method according to claim 1, characterized in that The original face area to be replaced is determined by the following steps: Performing feature extraction on the obtained reference face image of the target object and each face image region included in the video frame to obtain reference face image features of the target object and image features of each face image region; the face image region is determined according to image size information of the reference face image, and the face image region at least includes an original face region; Determining the facial image region corresponding to the image feature that meets a specified condition based on the similarity calculation result between the reference facial image feature of the target object and the image feature of each facial image region; wherein the specified condition includes: the similarity calculation result is the maximum value among the similarity results of each image feature, and the similarity result is greater than a first specified value; The original face region included in the face image region corresponding to the image feature that meets the specified condition is determined as the original face region to be replaced.

3. The method according to claim 1, characterized in that The at least two different face replacement models require different optimal face angles.

4. The method according to claim 2, characterized in that: The original face area in each video frame is numbered in a specified order, and an index number value is also recorded in each video frame; before performing face replacement processing on the original face area to be replaced in the video frame using at least two different face replacement models currently available, the method further includes: For each video frame, the recorded index number value in the video frame is detected. If the recorded index number value is the number value corresponding to any original face area in the video frame, the step of performing face replacement processing on the original face area to be replaced in the video frame by using at least two different face replacement models currently available is continued; the original face area to be replaced refers to the original face area corresponding to the recorded index number value; If the recorded index number value is a second specified value, the step of performing face replacement processing on the original face area to be replaced in the video frame using at least two different face replacement models currently available is refused to be executed; the second specified value is different from the number value corresponding to any original face area, and the second specified value is used to indicate that there is no original face area to be replaced in the video frame, or there is no original face area in the video frame.

5. The method according to claim 1, characterized in that The step of scoring the face replacement results output by each face replacement model and selecting the optimal face replacement result based on the scores of each face replacement result includes: For each face replacement result, a convolution operation is performed on the grayscale value of each pixel in the face replacement result based on the Laplace operator to obtain a convolution operation result corresponding to the face replacement result; The variance of the convolution operation result corresponding to each face replacement result is used as the score of each face replacement result; the score of each face replacement result is used to indicate the image blur degree of each face replacement result, and the image blur degree is inversely correlated with the variance of the convolution operation result; The face replacement result with the highest score among all face replacement results is determined as the optimal face replacement result.

6. The method according to claim 1, characterized in that Before performing face replacement processing on the original face region to be replaced in the video frame using at least two different face replacement models currently available, the method further includes: Based on the trained neural network model, classify and predict the image features of the video frame and the text features of the text information included in the video frame to obtain a classification prediction result for the video frame; If the classification prediction results of all video frames meet the normal video frame conditions, the step of performing face replacement processing on the original face area to be replaced in the video frame using at least two different face replacement models currently available is executed.

7. The method according to claim 1, characterized in that The replacing the original face region to be replaced in the video frame with the optimal face replacement result comprises: Based on the trained face restoration model, the optimal face replacement result is image optimized to obtain the optimal face replacement result after image optimization; the original face area to be replaced in the video frame is replaced with the optimal face replacement result after image optimization; Alternatively, the original face region to be replaced in the video frame is replaced with the optimal face replacement result; and image optimization is performed on all face regions included in the video frame based on the trained face restoration model.

8. A video face replacement device, characterized in that: The device includes: A first replacement unit is used to perform face replacement processing on an original face region to be replaced in a video frame using at least two different face replacement models currently available, so as to obtain a face replacement result of the original face region; the face replacement result represents a face region in which an original face of a target object in the original face region is replaced with a target face; A scoring selection unit, used to score the face replacement results output by each face replacement model, and select the optimal face replacement result based on the scores of each face replacement result; The second replacement unit is used to replace the original face area to be replaced in the video frame with the optimal face replacement result.

9. The device according to claim 8, characterized in that The first replacement unit is also used for: Performing feature extraction on the obtained reference face image of the target object and each face image region included in the video frame to obtain reference face image features of the target object and image features of each face image region; the face image region is determined according to image size information of the reference face image, and the face image region at least includes an original face region; Determining the facial image region corresponding to the image feature that meets a specified condition based on the similarity calculation result between the reference facial image feature of the target object and the image feature of each facial image region; wherein the specified condition includes: the similarity calculation result is the maximum value among the similarity results of each image feature, and the similarity result is greater than a first specified value; Determine the original face region included in the face image region corresponding to the image feature that meets the specified condition as the original face region to be replaced; And / or, the at least two different face replacement models require different optimal face angles; And / or, the original face area in each video frame is numbered in a specified order, and an index number value is also recorded in each video frame; before performing face replacement processing on the original face area to be replaced in the video frame using at least two different face replacement models currently available, the first replacement unit is further used to: For each video frame, the recorded index number value in the video frame is detected. If the recorded index number value is the number value corresponding to any original face area in the video frame, the step of performing face replacement processing on the original face area to be replaced in the video frame by using at least two different face replacement models currently available is continued; the original face area to be replaced refers to the original face area corresponding to the recorded index number value; If the recorded index number value is a second specified value, the step of performing face replacement processing on the original face region to be replaced in the video frame by using at least two different face replacement models currently available is rejected; the second specified value is different from the number value corresponding to any original face region, and the second specified value is used to indicate that there is no original face region to be replaced in the video frame, or there is no original face region in the video frame; And / or, the scoring selection unit is specifically used for: For each face replacement result, a convolution operation is performed on the grayscale value of each pixel in the face replacement result based on the Laplace operator to obtain a convolution operation result corresponding to the face replacement result; The variance of the convolution operation result corresponding to each face replacement result is used as the score of each face replacement result; the score of each face replacement result is used to indicate the image blur degree of each face replacement result, and the image blur degree is inversely correlated with the variance of the convolution operation result; The face replacement result with the highest score among all face replacement results is determined as the optimal face replacement result; And / or, before performing face replacement processing on the original face region to be replaced in the video frame using at least two different face replacement models currently available, the first replacement unit is further used to: Based on the trained neural network model, classify and predict the image features of the video frame and the text features of the text information included in the video frame to obtain a classification prediction result for the video frame; If the classification prediction results of all video frames meet the normal video frame condition, the step of performing face replacement processing on the original face region to be replaced in the video frame by using at least two different face replacement models currently available is executed; And / or, the second replacement unit is specifically used for: Based on the trained face restoration model, the optimal face replacement result is image optimized to obtain the optimal face replacement result after image optimization; the original face area to be replaced in the video frame is replaced with the optimal face replacement result after image optimization; Alternatively, the original face region to be replaced in the video frame is replaced with the optimal face replacement result; and image optimization is performed on all face regions included in the video frame based on the trained face restoration model.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images