Face replacement method, system, terminal device and computer storage medium

Abstract

The present invention discloses a face replacement method, system, terminal device and computer storage medium. The face replacement method comprises: determining a target face to be replaced during the real-time playback of a video screen; obtaining the weight level of the character action associated with the target face, and optimizing the face feature extraction model based on the weight level; extracting the first facial feature and feature combination relationship of the target face using the optimized face feature extraction model; and replacing the face to be replaced in the video screen according to the first facial feature and the feature combination relationship. Compared with the traditional method of extracting facial features based on weak supervision for real-time face replacement, the present invention can achieve a more flexible and coordinated effect of face replacement in real-time video screen.

Description

Technical Field

[0001] The present invention relates to the technical field of digital image processing, and in particular to a face replacement method, system, terminal equipment and computer storage medium. Background Art

[0002] With the rapid development of Internet technology, video technology is also developing rapidly. In order to meet user needs, the practical application of video character face swapping has already been introduced into existing video technology.

[0003] At present, face-changing applications in videos use weak supervision to extract facial features, so that the weak label data features of the face can be learned into stronger label features, thereby solving the problem of low accuracy of face replacement in some videos. However, in the actual application of existing video face-changing, there are still problems such as low flexibility of face replacement and incoordination of facial expressions and movements after replacement. Summary of the invention

[0004] The main purpose of the present invention is to provide a face replacement method, system, terminal device and computer storage medium, aiming to improve the flexibility and coordination of face replacement in real-time video images.

[0005] To achieve the above object, the present invention provides a face replacement method, the face replacement method comprising:

[0006] Determine a target face for face replacement;

[0007] Extracting the first facial feature and the feature combination relationship of the target face by using the optimized facial feature extraction model;

[0008] The face to be replaced is replaced in the video image according to the first facial feature and the feature combination relationship.

[0009] Preferably, the method further comprises:

[0010] Detecting the feature influence level of the first facial feature of the target face under each character action;

[0011] Adapting weight parameters corresponding to each of the character actions associated with the first facial feature according to the feature influence level;

[0012] The facial feature extraction model is optimized based on the weight levels.

[0013] Preferably, the step of optimizing the facial feature extraction model based on the weight level comprises:

[0014] The facial feature extraction model is optimized based on the weight level and the characteristic distance between the first facial features, wherein the characteristic distance includes: a characteristic pixel distance for representing the position difference between the facial features and a characteristic light distance for representing the light difference between the facial features.

[0015] Preferably, the step of replacing the face to be replaced in the video image according to the first facial feature and the feature combination relationship comprises:

[0016] Detecting a face to be replaced in the video image, and learning a second facial feature of the face to be replaced;

[0017] Determining target image data matching the second facial feature according to the first facial feature and the feature combination relationship;

[0018] The target image data is used to replace the face to be replaced in the video picture.

[0019] Preferably, the step of determining target image data matching the second facial feature according to the first facial feature and the feature combination relationship comprises:

[0020] Detecting whether there is a target feature in the first facial feature that matches the second facial feature;

[0021] If so, the image data corresponding to the target feature is optimized according to the feature combination relationship, and the optimized image data is determined as the target image data.

[0022] Preferably, after the step of detecting whether there is a target feature matching the second facial feature in the first facial feature, the method further includes:

[0023] If not, the target feature is generated according to the first facial feature and the feature combination relationship, and the image data corresponding to the target feature is determined as the target image data.

[0024] Preferably, the step of determining a target face for face replacement comprises:

[0025] Scanning facial features in image data collected in real time by a preset image acquisition device to determine a target face for face replacement in the video picture; or,

[0026] The facial features in the image data specified in the preset storage space are scanned to determine a target face for face replacement in the video frame.

[0027] In addition, to achieve the above-mentioned purpose, the present invention further provides a face replacement system, the face replacement system comprising:

[0028] A determination module, used to determine a target face for face replacement;

[0029] An extraction module, used to extract the first facial feature and feature combination relationship of the target face by using the optimized facial feature extraction model;

[0030] The replacement module is used to replace the face to be replaced in the video picture according to the first facial feature and the feature combination relationship.

[0031] Among them, each functional module of the face replacement system of the present invention implements the steps of the face replacement method described above when running.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also provides a terminal device, which includes: a memory, a processor, and a face replacement program stored in the memory and executable on the processor, wherein the face replacement program, when executed by the processor, implements the steps of the face replacement method described above.

[0033] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a face replacement program is stored, and when the face replacement program is executed by a processor, the steps of the face replacement method described above are implemented.

[0034] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a face replacement program is stored, and when the face replacement program is executed by a processor, the steps of the face replacement method described above are implemented.

[0035] The present invention provides a face replacement method, system, terminal equipment, computer storage medium and computer storage medium, which determine a target face for face replacement; use an optimized face feature extraction model to extract the first facial feature and feature combination relationship of the target face; and replace the face to be replaced in a video screen according to the first facial feature and the feature combination relationship.

[0036] In the process of performing face replacement on a video screen being played in real time, the present invention determines that a face replacement operation needs to be performed on the video screen to replace a person to be replaced in the video screen with a target face during the real-time playing of the video screen, then uses an optimized face feature extraction model to extract a first facial feature of the target face and a feature combination relationship between the first facial features, and finally, that is, in the video screen being played in real time, a replacement operation is performed on the face to be replaced that needs to be replaced in the video screen according to the first facial feature and the feature combination relationship to obtain a video screen after the face is replaced.

[0037] Compared with the traditional method of extracting facial features based on weak supervision for real-time face replacement, the present invention can not only realize real-time scanning and learning of facial information and quickly perform face replacement operations to replace the character to be replaced by the learned facial object while playing the video screen, but also increase the extraction of feature combination relationships between facial features in the process of scanning and learning facial information, so as to effectively solve the problems of poor flexibility of the face replacement process and incoordination between the face and the character's movements and expressions after face replacement caused by incomplete extraction of distance relationship information between facial features, thereby improving the flexibility and coordination of face replacement in real-time video screen. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A schematic diagram of the device structure of the hardware operating environment of the terminal device involved in the embodiment of the present invention;

[0039] Figure 2 A schematic diagram of a flow chart of an embodiment of a face replacement method of the present invention;

[0040] Figure 3 FIG. 1 is a schematic diagram of functional modules of an embodiment of a face replacement system of the present invention.

[0041] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0042] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0043] Reference Figure 1 , Figure 1 The present invention is a schematic diagram of the device structure of the hardware operating environment of the terminal device involved in the embodiment of the present invention.

[0044] The terminal device of the embodiment of the present invention can be a client device loaded with the functional service that needs to be tested, or, the terminal device of the embodiment of the present invention can also be a baffle server device specially used to test the functional service and thus configure relevant baffle rules. The device can be a smart phone, PC (Personal Computer), tablet computer, portable computer, etc.

[0045] like Figure 1 As shown, the terminal device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0046] Those skilled in the art will understand that Figure 1 The terminal device structure shown in the figure does not constitute a limitation on the terminal device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0047] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a face replacement program.

[0048] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and perform data communication with the backend server; the user interface 1003 is mainly used to connect to the client and perform data communication with the client; and the processor 1001 can be used to call the face replacement program stored in the memory 1005 and perform the following steps:

[0049] Determine a target face for face replacement;

[0050] Extracting the first facial feature and the feature combination relationship of the target face by using the optimized facial feature extraction model;

[0051] The face to be replaced is replaced in the video image according to the first facial feature and the feature combination relationship.

[0052] Preferably, the processor 1001 may be used to call the face replacement program stored in the memory 1005, and further perform the following steps:

[0053] Detecting the feature influence level of the first facial feature of the target face under each character action;

[0054] Adapting weight parameters corresponding to each of the character actions associated with the first facial feature according to the feature influence level;

[0055] The facial feature extraction model is optimized based on the weight levels.

[0056] Preferably, the processor 1001 may be used to call the face replacement program stored in the memory 1005, and further perform the following steps:

[0057] The facial feature extraction model is optimized based on the weight level and the characteristic distance between the first facial features, wherein the characteristic distance includes: a characteristic pixel distance for representing the position difference between the facial features and a characteristic light distance for representing the light difference between the facial features.

[0058] Preferably, the processor 1001 may be used to call the face replacement program stored in the memory 1005, and further perform the following steps:

[0059] Detecting a face to be replaced in the video image, and learning a second facial feature of the face to be replaced;

[0060] Determining target image data matching the second facial feature according to the first facial feature and the feature combination relationship;

[0061] The target image data is used to replace the face to be replaced in the video picture.

[0062] Preferably, the processor 1001 may be used to call the face replacement program stored in the memory 1005, and further perform the following steps:

[0063] Detecting whether there is a target feature in the first facial feature that matches the second facial feature;

[0064] If so, the image data corresponding to the target feature is optimized according to the feature combination relationship, and the optimized image data is determined as the target image data.

[0065] Preferably, the processor 1001 may be used to call the face replacement program stored in the memory 1005, and after executing the step of detecting whether there is a target feature in the first facial feature that matches the second facial feature, further execute the following steps:

[0066] If not, the target feature is generated according to the first facial feature and the feature combination relationship, and the image data corresponding to the target feature is determined as the target image data.

[0067] Preferably, the processor 1001 may be used to call the face replacement program stored in the memory 1005, and further perform the following steps:

[0068] Scanning facial features in image data collected in real time by a preset image acquisition device to determine a target face for face replacement in the video picture; or,

[0069] The facial features in the image data specified in the preset storage space are scanned to determine a target face for face replacement in the video frame.

[0070] Based on the above hardware structure, various embodiments of the face replacement method of the present invention are proposed.

[0071] In view of the above phenomenon, the present invention provides a face replacement method. Figure 2 , Figure 2 : is a flow chart of a first embodiment of the face replacement method of the present invention. In this embodiment, the face replacement method of the present invention is applied to the above-mentioned terminal device. The face replacement method of the present invention includes:

[0072] Step S10, determining a target face for face replacement;

[0073] During the process of real-time playback of a video screen that requires a face replacement operation, the terminal device receives a face replacement instruction triggered by a command control preset by the front end, and thereby determines, based on the face replacement instruction, a target face for performing a face replacement operation in the video screen to replace the face to be replaced.

[0074] Specifically, for example, in this embodiment, the terminal device can generate a virtual control for triggering the face replacement instruction: "Face Replacement" button on the terminal interface of the video application program for real-time video playback in advance. Therefore, the user of the terminal device can trigger the face replacement instruction for performing the face replacement operation on the video screen by clicking the "Face Replacement" button as needed during the real-time playback of the current video screen. The terminal device receives the face replacement instruction and starts to determine the target face for performing the face replacement operation to replace the face to be replaced in the video screen based on the face replacement instruction.

[0075] Furthermore, in a feasible embodiment, the above step S10 may include:

[0076] Step S101, scanning facial features in image data collected in real time by a preset image collection device to determine a target face for face replacement in the video picture;

[0077] It should be noted that, in the present embodiment, the preset image acquisition device is an image acquisition device that is built-in or external to the terminal device and is used to acquire image data of the target face in real time.

[0078] After receiving the face replacement instruction, the terminal device starts to call the image acquisition device built into or external to the terminal device to collect image data of the target face in real time to collect image data based on the method specified by the face replacement instruction. Then, the terminal device further scans and learns the collected image data to determine the target face currently used for the face replacement operation to replace the face to be replaced in the video screen.

[0079] Specifically, for example, in this embodiment, it is assumed that the preset image acquisition device is a device camera configured by the terminal device itself. If the terminal device receives a face replacement instruction triggered by the user clicking the "face change" button, and the user does not perform further operations within the preset time, the terminal device automatically turns on the device camera in a default manner, and collects image data containing the user's face in real time through the device camera to determine that the user's face is the target face currently used to perform a face replacement operation on the video screen being played to replace the face to be replaced in the video screen.

[0080] It should be noted that, in the present embodiment, when the terminal device calls the device camera to collect image data containing the user's face in real time, it can specifically align the user's face according to the grid lines to shoot to obtain image data of a close-up photo or video with clear facial features in various directions, dimensions, light, expression, and movements, and then upload the image data based on the user's click. After that, the terminal device scans and learns the facial features, contours, movements, mouth shape, and other information of the user's face in the image data in real time by calling the algorithm interface inside the application.

[0081] Furthermore, the terminal device here scans and learns the user's face in the image data specifically by: detecting and locating the image data containing the user's face collected in real time, using static or dynamic images obtained from the image data set, and detecting the local face image therefrom, extracting the local face structure from the complex external background and returning information such as the size, orientation and angle of the face, so as to avoid the influence of external lighting, facial tilt angles and various obstructions on the extracted face image structure.

[0082] Furthermore, in another feasible embodiment, the above step S10 may further include:

[0083] Step S102, scanning facial features in the image data specified in the preset storage space to determine a target face for face replacement in the video picture.

[0084] It should be noted that, in this embodiment, the preset storage space is a storage space built into or external to the terminal device for storing image data containing the user's face.

[0085] After receiving the face replacement instruction, the terminal device starts to read the image data containing the user's face from the storage space built into or external to the terminal device for storing image data containing the user's face based on the method specified by the face replacement instruction. Then, the terminal device further scans and learns the collected image data to determine the target face currently used for the face replacement operation to replace the face to be replaced in the video screen.

[0086] Specifically, for example, in this embodiment, it is assumed that the preset storage space is a local gallery configured by the terminal device itself. If the terminal device receives a face replacement instruction triggered by the user clicking the "face replacement" button, and the user performs further operations within a preset time to select a picture containing a face that the user wants to use for face replacement operation from the local gallery, the terminal device directly determines that the face in the picture is the target face currently used to perform a face replacement operation on the video screen being played to replace the face to be replaced in the video screen.

[0087] It should be noted that in the present embodiment, when the user independently selects image data in the local map library, the image data must also be a close-up photo or video with clear facial features in all directions, dimensions, light, expression, and movements. Thus, the terminal device can scan and learn the facial features, contours, movements, mouth shapes, and other information of the face in the image data in real time based on calling the internal algorithm interface of the application.

[0088] And, further, the terminal device here scans and learns the user's face in the image data also means: detecting and locating the image data containing the user's face collected in real time, using static or dynamic images obtained from the image data set, and extracting the local face structure from the complex external background by detecting the local face image therefrom, and returning information such as the size, orientation and angle of the face, so as to avoid the influence of external lighting, facial tilt angles and various obstructions on the extracted face image structure.

[0089] Step S20, extracting the first facial feature and feature combination relationship of the target face using the optimized facial feature extraction model;

[0090] The terminal device uses the optimized facial feature extraction model to learn to extract each first facial feature in the target face, and extracts the feature combination relationship between the first facial features.

[0091] It should be noted that, in the present embodiment, since the terminal device has combined the feature distance relationship between facial features in the process of optimizing the facial feature extraction model, the terminal device can directly extract the facial features of the target face and the feature combination relationship between the facial features by using the optimized facial feature extraction model. As for the specific method in which the terminal device uses the optimized facial feature extraction model to extract facial features and the feature combination relationship, it is the same as the existing method of using a facial feature extraction model to extract facial features in real-time face replacement based on weak supervision. The face replacement method of the present invention does not make specific limitations on the terminal device applying the facial feature extraction model to extract facial features and feature combination relationships.

[0092] Furthermore, in a feasible embodiment, after the terminal device utilizes the optimized facial feature extraction model to learn to extract each first facial feature in the target face and extract the feature combination relationship between the first facial features, it further processes the first facial features and the feature combination relationship to form a facial feature library, which can be used to directly find image data with suitable features in the facial feature library to perform face replacement operations on the video screen.

[0093] Step S30: replacing the face to be replaced in the video image according to the first facial feature and the feature combination relationship.

[0094] After the terminal device uses the optimized facial feature extraction model to extract the first facial features in the target face and the feature combination relationship between the first facial features, it can perform a face replacement operation on the face to be replaced in the video screen currently being played in real time, based on the first facial features and the feature combination relationship.

[0095] Furthermore, in a feasible embodiment, the above step S30 may include:

[0096] Step S301, detecting a face to be replaced in the video image, and learning a second facial feature of the face to be replaced;

[0097] When the terminal device performs a face replacement operation on the face to be replaced in the video screen according to the first facial features and the feature combination relationship, it first determines the face to be replaced in the video screen based on the selection operation detection triggered by the user, and then uses the mature machine learning method to learn the second facial features of the face to be replaced.

[0098] Specifically, for example, the user of the terminal device selects the corresponding character whose face he wants to replace in the playing video screen, and the terminal device determines, based on the selection operation input by the user, that the face corresponding to the character is the face to be replaced that the current user needs to replace in the video screen. Therefore, the terminal device further calls a mature machine learning algorithm to learn and understand the facial features, contours, movements, mouth shapes and other second facial features of the face corresponding to the task character in the current real-time video screen.

[0099] Step S302, determining target image data matching the second facial feature according to the first facial feature and the feature combination relationship;

[0100] After learning the second facial features of the face to be replaced, the terminal device begins to adapt according to the first facial features and the feature combination relationship between the first facial features to obtain target image data corresponding to the target features that are mutually adapted to the second facial features in various angles, expressions and other dimensions.

[0101] Furthermore, in a feasible embodiment, the above step S302 may include:

[0102] Step S3021, detecting whether there is a target feature in the first facial feature that matches the second facial feature;

[0103] When the terminal device adapts the target image data corresponding to the target feature that is mutually compatible with the second facial feature in various angles, expressions and other dimensions according to the feature combination relationship between the first facial features and the mutual first facial features, it first detects whether there is a target feature that is directly compatible with the second facial feature among all the first facial features.

[0104] Specifically, for example, in this embodiment, when the terminal device adapts the second facial features such as eyes and mouth when smiling of the face to be replaced among the first facial features, it first detects whether there are second facial features such as eyes and mouth when smiling that are directly matched with the second facial features such as eyes and mouth when smiling among all the first facial features, and then detects the first facial features such as eyes and mouth when smiling of the target face that are mutually adapted from various angles, expressions and other dimensions.

[0105] Step S3022: If it exists, the image data corresponding to the target feature is optimized according to the feature combination relationship, and the optimized image data is determined as the target image data.

[0106] When the terminal device detects that there is a target feature that is directly compatible with the second facial feature among all the first facial features, it further optimizes the image data corresponding to the target feature according to the feature combination relationship between the first facial features to obtain optimized image data, and determines the optimized image data as the target image data for performing a face replacement operation on the face to be replaced.

[0107] Specifically, for example, in this embodiment, after the terminal device detects that among all the first facial features, there are first facial features such as eyes and mouth of the target face when smiling that are compatible with the second facial features such as eyes and mouth when smiling in various angles, expressions and other dimensions, it further optimizes the eye feature distance and the feature distance of each pixel point of the mouth in the image data corresponding to the first facial features such as eyes and mouth when smiling through the feature combination relationship between the first facial features such as eyes and mouth when smiling, thereby obtaining optimized image data as target image data for subsequent real-time face replacement operations.

[0108] Further, in another feasible embodiment, after detecting whether there is a target feature matching the second facial feature in the first facial feature in the above step S3021, the face replacement method of the present invention may further include:

[0109] Step S3023: If it does not exist, generate the target feature according to the first facial feature and the feature combination relationship, and determine the image data corresponding to the target feature as the target image data.

[0110] When the terminal device detects that there is no target feature that is directly compatible with the second facial feature among all the first facial features, the terminal device first generates first facial features such as eyes and mouth when smiling, which are compatible with the target face in various angles, expressions and other dimensions according to the feature combination relationship between the first facial features, and then optimizes the image data corresponding to the target feature according to the feature combination relationship between the first facial features to obtain optimized image data, and determines the optimized image data as the target image data for performing a face replacement operation on the face to be replaced.

[0111] Specifically, for example, in the present embodiment, when the terminal device detects all the first facial features, there are no first facial features such as eyes and mouth of the target face when smiling that are compatible with the second facial features such as eyes and mouth when smiling from various angles, expressions and other dimensions. Then, the terminal device generates first facial features such as eyes and mouth of the target face when smiling that are compatible with the second facial features such as eyes and mouth when smiling from various angles, expressions and other dimensions according to all the first facial features of the existing target face and the feature combination relationship among all the first facial features. Finally, the eye feature distance and the feature distance of each pixel point of the mouth in the image data corresponding to the first facial features such as eyes and mouth when smiling are further optimized through the feature combination relationship among the first facial features such as eyes and mouth when smiling, thereby obtaining the optimized image data as the target image data for the subsequent real-time face replacement operation.

[0112] Step S303: replacing the face to be replaced in the video screen with the target image data.

[0113] After the terminal device adapts the target image data corresponding to the target features that are mutually adapted to the second facial features in various angles, expressions and other dimensions according to the feature combination relationship between the first facial features and the first facial features, the terminal device can directly use the target image data to perform a real-time face replacement operation on the face to be replaced in the currently playing video screen, thereby completing the real-time face replacement of the face to be replaced and obtaining a video screen that is played in real time after the face replacement.

[0114] It should be noted that, in the present embodiment, the specific manner in which the terminal device uses the target image data to perform the real-time face-changing operation on the face to be replaced is the same as the existing operation method of realizing real-time face-changing based on weak supervision. The face replacement method of the present invention does not specifically limit the manner in which the terminal device uses the target image data to perform the real-time face-changing operation on the face to be replaced.

[0115] In this embodiment, during the process of real-time playback of a video screen that needs to be replaced with a face, the terminal device receives a face replacement instruction triggered by a command control preset by the front end, thereby starting to determine the target face for performing a face replacement operation on the video screen currently being played in real time to replace the face to be replaced in the video screen according to the face replacement instruction; the terminal device uses the optimized face feature extraction model to learn to extract each first facial feature in the target face, and extract the feature combination relationship between the first facial features; after the terminal device uses the optimized face feature extraction model to extract each first facial feature in the target face, and the feature combination relationship between the first facial features, the terminal device can perform a face replacement operation on the face to be replaced that needs to be replaced in the video screen in the current real-time playback according to the first facial features and the feature combination relationship.

[0116] Compared with the traditional method of extracting facial features based on weak supervision for real-time face replacement, the present invention can not only realize real-time scanning and learning of facial information and quickly perform face replacement operations to replace the character to be replaced by the learned facial object while playing the video screen, but also increase the extraction of feature combination relationships between facial features in the process of scanning and learning facial information, so as to effectively solve the problems of poor flexibility of the face replacement process and incoordination between the face and the character's movements and expressions after face replacement caused by incomplete extraction of distance relationship information between facial features, thereby improving the flexibility and coordination of face replacement in real-time video screen, so that face replacement in real-time video screen achieves a more flexible and coordinated effect.

[0117] Furthermore, in a feasible embodiment, the face replacement method of the present invention may further include:

[0118] Step S40, obtaining the weight level of the human action associated with the target face, and optimizing the face feature extraction model based on the weight level;

[0119] After the terminal device determines the target face currently used to perform a face replacement operation on the video screen being played in real time, thereby replacing the face to be replaced in the video screen, the terminal device further obtains the weight level of the character action associated with the target face based on the target face, and after obtaining the weight parameter, further optimizes the facial feature extraction model used for the face replacement operation based on the weight parameter.

[0120] It should be noted that in this embodiment, after the terminal device determines the target face currently used for face replacement operation on the video screen being played in real time, thereby replacing the face to be replaced in the video screen, it also performs preliminary feature processing on each facial feature obtained by scanning and learning the target face. Specifically, the preliminary feature processing includes but is not limited to normalization, removal of redundant feature information, increase of relevant information between features, removal of light interference, etc. For example, the terminal device uses any mature processing method to perform preprocessing operations on the local face image extracted from the complex background (specifically including: correcting the geometric shape, blurring the image mode, equalizing the histogram and / or normalizing the pixel gray value).

[0121] Furthermore, in this embodiment, in the above step S40, the method of “obtaining the weight level of the human action associated with the target face” includes:

[0122] Step S401, detecting the feature influence level of the first facial feature of the target face under various character actions;

[0123] When the terminal device obtains the weight level of the human action associated with the target face, it first detects the facial features of the target face and the influence levels of the features that are differently affected in the action locations or directions of different human actions.

[0124] Specifically, for example, the terminal device combines the pixel points of the right eye as a single feature and the light distance, and calculates the probability of specific occlusion effects based on light intensity, etc. (the stronger the light intensity, the greater the probability of occlusion effects). In this way, the right eye as a facial feature has different effects in different action locations or directions. For example, blinking, opening eyes, closing eyes, squinting (the degree of eye closure), squinting (different directions and angles), etc. are relatively obvious character actions involving the eyes. Therefore, it can be determined that the level of feature influence on the right eye under the character action is a first-level feature influence. For facial expressions such as smiling, crying, getting angry, or raising eyebrows, frowning, etc., the degree of eye involvement in the eye movements is moderate, so the level of feature influence on the right eye under the character action can be determined to be a second-level feature influence. For example, nodding and shaking heads, which only involve the head, or opening and closing the mouth, which only involve the mouth, the degree of eye involvement is very low and is almost considered to be non-involved. Therefore, it can be determined that the level of feature influence on the right eye under the character action is a third-level feature influence.

[0125] Step S402, adapting the weight parameters corresponding to the character actions associated with the first facial feature according to the feature influence level;

[0126] Step S403: optimizing the facial feature extraction model based on the weight levels.

[0127] After the terminal device detects the facial features of the target face and the different levels of feature influence that are affected in the motion positioning or direction of different human actions, it further adapts the weight parameters corresponding to the different human actions corresponding to the facial features according to the feature influence levels, and obtains the weight parameters for subsequent optimization of the facial feature extraction model used for face replacement operations.

[0128] It should be noted that, in the present embodiment, the weight parameter is a weight value that is weighted based on the feature influence level of a certain facial feature in the target face under a certain character action. The weight parameter can be specifically set based on different design requirements of actual applications. The face replacement method of the present invention does not limit the specific size of the weight parameter.

[0129] Specifically, for example, in this embodiment, it is assumed that the right eye as a facial feature has different influences in different action locations or directions, such as blinking, opening eyes, closing eyes, squinting (the degree of eye closure), squinting (different directions and angles), etc., which are relatively obvious character actions involving the eyes. Therefore, it can be determined that the level of feature influence on the right eye under the character action is a first-level feature influence. Therefore, a first-level weight can be used for weighted processing, that is, the first-level weight is the weight parameter of the character action corresponding to the right eye, and facial expressions such as smiling, crying, getting angry, or raising eyebrows, frowning, etc., in which the eyebrows move the eyes, due to the participation of the eyes. The action level is medium, so it can be determined that the level of feature influence on the right eye under the character's action is secondary feature influence, and the secondary weight can be used for weighted processing, that is, the secondary weight is the weight parameter of the character action corresponding to the right eye. For example, nodding and shaking the head are actions in which only the head participates, or opening and closing the mouth are actions in which only the mouth participates, since the eyes participate in the action to a very low degree, it is almost considered that they do not participate. Therefore, it can be determined that the level of feature influence on the right eye under the character's action is tertiary feature influence, and the tertiary weight can be used for weighted processing, that is, the tertiary weight is the weight parameter of the character action corresponding to the right eye.

[0130] Further, in a feasible embodiment, in the above step S40, the step of "optimizing the face feature extraction model based on the weight level" may include:

[0131] Step S403: optimizing the facial feature extraction model based on the weight level and the characteristic distance between the first facial features, wherein the characteristic distance includes but is not limited to: characteristic pixel distance and characteristic light distance.

[0132] After the terminal device determines the weight parameters corresponding to the character action based on the feature influence level of the facial features of the target face under different character actions, it uses the weight parameters in combination with the feature pixel distance and feature light distance between the facial features to optimize the facial feature extraction model used for the face replacement operation to obtain the optimized facial feature extraction model.

[0133] It should be noted that, in this embodiment, the facial feature extraction model used for face replacement operation is an algorithm for calculating action probability by averaging frame-level prediction of conventional feature vectors used for real-time face replacement in a weakly supervised manner.

[0134] Specifically, when the frame-level action class annotation vector y appears in the process of video learning features t When training known frames of a video, the feature vector f and the joint model h can be trained by minimizing the frame-level binary cross entropy loss. Here, the joint model function is expressed as h(f(I t )), takes the feature vector f extracted from each frame as input and outputs its training model. Since it is assumed that there is no ground truth frame-level label available for training, the ground truth frame-level label here refers to the actual label available for training for each frame in the corresponding video, indicating whether there is a ground truth binary vector for each column of training video. If this behavior exists, its value is set to 1, otherwise it is set to 0. Therefore, here we are limited to using the video-level label vector y to train the predicted frame-level action class annotation vector y at test time t The action localization model, where y t That is, to predict the frame label vector of each frame, which is obtained by using a model trained with a video-level label vector y during testing, and this model is the action localization model. In simple terms, it is to obtain a feature vector by training each frame of video data, and then use a classification network to process the feature vector and output it as an action classification score vector. During testing, the model can be used to predict the frame label vector of each frame, and video-level predictions can be obtained from the frame sequence. A simple strategy is to average the frame-level predictions over the entire sequence, as shown below for the expression of the face feature extraction model:

[0135]

[0136] Here, T is the number of frames in the sequence and can vary across videos, σ is the softmax normalization function on the prediction score vector for each frame, θ is the learned parameter of the feature vector f, and θ cls are trainable parameters of the joint model h.

[0137] Since the above-mentioned facial feature extraction model, that is, the average frame-level prediction of the facial feature vector, does not consider the relationship information between features in terms of distance optimization, taking the relationship between the feature pixel distance and the feature light distance of facial features as an example, the feature pixel distance represents the difference in position between facial features and determines the overall directional layout of various facial features, while the feature light distance represents the light difference between facial features. The light movement layout between various facial features is determined by the feature light distance. Therefore, by continuously updating the feature pixel distance and feature light distance between facial features, the distance optimization goal between facial features can be achieved.

[0138] Specifically, for example, in this embodiment, assuming that the face is blocked during the arm movement, the light and shadow transition effect is not good; but if the difference distance between the feature pixels of the face and the light changes caused by the arm movement itself are considered, the feature pixel distance and light distance can be continuously optimized during the feature extraction process to obtain a smoother and more flexible face-changing effect. On the basis of obtaining the action classification score vector of each frame, the pixel and light representation corresponding to each feature vector are refined and accurately positioned. Therefore, the terminal device extracts and processes the learned facial features by adding the relationship extraction between the facial combination features based on the naive weak supervision method, that is, the above-mentioned conventional facial feature extraction model.

[0139] Assume that the feature pixel distance is expressed as p = p0*(1+cd ij 2 ), where p0 represents the minimum (original) pixel distance of a facial feature in each frame, that is, in d ij = 0 (the vectorized distance between frames is 0, i.e., the frames overlap), the minimum pixel distance of the facial feature, c represents the distance coefficient, and c can be updated during the distance optimization iteration. At the same time, because the trend of light changes with distance and pixels is different, assuming that the light follows the exponential update rule, the light distance is expressed as Among them, l0 represents the minimum (original) light distance of a feature in each frame, that is, in d ij = 0 (the vectorized distance between frames is 0, i.e., the frames overlap), the minimum ray distance of facial features.

[0140] The representation of feature pixel distance and feature light distance and their distance coefficients can change with the attributes of specific features. For example, in the scenario where the above-mentioned character action of "covering the right eye" is applicable to the secondary weight, the distance attributes of the two facial expression features of smiling and crying should be opposite. Similarly, in the scenario where the primary weight is applicable, the distance attributes of the two facial action features of opening eyes and closing eyes should also be opposite. Therefore, the representation of the above-mentioned facial feature extraction model should also be updated according to the specific features. Therefore, the above-mentioned facial feature extraction model can be specifically optimized to the optimized facial feature extraction model shown below:

[0141]

[0142] It should be noted that in this embodiment, in addition to considering the relationship between the characteristic pixel distance and the characteristic light distance when training the feature vector function f, it is also necessary to consider the relationship between the pixel distance and the light distance when the joint model function h calculates the action classification value vector of each frame to mark each frame vector. The most important thing is the distance optimization process, that is, optimizing the learning parameter θ of the feature vector function f through the pixel distance p and the light distance l, and optimizing the learning parameter θ of the feature vector function f through the pixel distance p cls The distance from the light cls Optimize the trainable parameters θ of the joint model function h cls Similarly, in order to train such a model, the terminal device can use the prediction probability vector Binary cross entropy loss on and its basic truth value are minimized. In the actual test process, the terminal device can use the function h(f(I t )) to mark each frame, and then use the joint model function h(f(I t )) can get each frame I t The action classification value vector.

[0143] In the above, by adding two learning and training parameters, namely, the feature pixel distance p and the feature light distance l, to optimize the calculation of the optimized face feature extraction model, an action classification value vector with a higher frame-level matching degree can be obtained, thereby obtaining an optimized representation of each feature of the face image.

[0144] Furthermore, the present invention also provides a face replacement system. Figure 3 , Figure 3 FIG. 1 is a schematic diagram of the functional modules of an embodiment of a face replacement system of the present invention. Figure 3 As shown, the face replacement system of the present invention comprises:

[0145] A determination module 10, used to determine a target face for face replacement;

[0146] An extraction module 20, configured to extract the first facial features and feature combination relationship of the target face using an optimized facial feature extraction model;

[0147] The replacement module 30 is used to replace the face to be replaced in the video picture according to the first facial feature and the feature combination relationship.

[0148] Furthermore, the face replacement system of the present invention further comprises:

[0149] A detection module, used for detecting the feature influence level of the first facial feature of the target face under various character actions;

[0150] An adaptation module, used for adapting the weight parameters corresponding to each of the character actions associated with the first facial feature according to the feature influence level;

[0151] A model optimization module is used to optimize the facial feature extraction model based on the weight level.

[0152] Furthermore, the model optimization module includes:

[0153] An optimization unit is used to optimize the facial feature extraction model based on the weight level and the characteristic distance between the first facial features, wherein the characteristic distance includes: a characteristic pixel distance for representing the position difference between the facial features and a characteristic light distance for representing the light difference between the facial features.

[0154] Further, the replacement module 30 includes:

[0155] A second detection unit is used to detect a face to be replaced in the video image and learn a second facial feature of the face to be replaced;

[0156] an adaptation unit, configured to determine target image data adapted to the second facial feature according to the first facial feature and the feature combination relationship;

[0157] A replacement unit is used to replace the to-be-replaced face in the video screen with the target image data.

[0158] Furthermore, the adaptation unit further includes:

[0159] a detection subunit, configured to detect whether there is a target feature in the first facial feature that matches the second facial feature;

[0160] The first adapter unit is used to optimize the image data corresponding to the target feature according to the feature combination relationship, if it exists, and determine the optimized image data as the target image data.

[0161] Furthermore, the adaptation unit further includes:

[0162] The second adapter unit is configured to generate the target feature according to the first facial feature and the feature combination relationship if it does not exist, and determine the image data corresponding to the target feature as the target image data.

[0163] Furthermore, the determination module 10 is also used to scan facial features in image data collected in real time by a preset image acquisition device to determine a target face for face replacement in the video screen; or, to scan facial features in image data specified in a preset storage space to determine a target face for face replacement in the video screen.

[0164] Among them, the functional implementation of each module in the above-mentioned face replacement system corresponds to the various steps in the above-mentioned face replacement method embodiment, and its functions and implementation processes will not be repeated here one by one.

[0165] The present invention also provides a computer storage medium, which includes a computer program. When the computer program is executed by a processor, the steps of the face replacement method described in any of the above embodiments are implemented.

[0166] The specific embodiments of the computer storage medium of the present invention are basically the same as the embodiments of the above-mentioned face replacement method, and will not be described in detail here.

[0167] The present invention also provides a computer storage medium on which a face replacement program is stored. When the face replacement program is executed by a processor, the steps of the face replacement method described in any of the above embodiments are implemented.

[0168] The specific embodiments of the computer storage medium of the present invention are basically the same as the embodiments of the above-mentioned face replacement method, and will not be described in detail here.

[0169] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0170] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0171] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0172] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A face replacement method, characterized in that: The face replacement method comprises: Determine a target face for face replacement; Extracting the first facial feature and the feature combination relationship of the target face by using the optimized facial feature extraction model; replacing the face to be replaced in the video image according to the first facial feature and the feature combination relationship; The method further comprises: Detecting the feature influence level of the first facial feature of the target face under each character action; Adapting weight parameters corresponding to each of the character actions associated with the first facial feature according to the feature influence level; The facial feature extraction model is optimized based on the weight level, wherein the facial feature extraction model is optimized based on the weight level and a characteristic distance between the first facial features, wherein the characteristic distance includes: a characteristic pixel distance for representing the position difference between the facial features and a characteristic light distance for representing the light difference between the facial features.

2. The face replacement method according to claim 1, characterized in that: The step of replacing the face to be replaced in the video image according to the first facial feature and the feature combination relationship comprises: Detecting a face to be replaced in the video image, and learning a second facial feature of the face to be replaced; Determining target image data matching the second facial feature according to the first facial feature and the feature combination relationship; The target image data is used to replace the face to be replaced in the video picture.

3. The face replacement method according to claim 2, characterized in that: The step of determining target image data matching the second facial feature according to the first facial feature and the feature combination relationship comprises: Detecting whether there is a target feature in the first facial feature that matches the second facial feature; If so, the image data corresponding to the target feature is optimized according to the feature combination relationship, and the optimized image data is determined as the target image data.

4. The face replacement method according to claim 3, characterized in that: After the step of detecting whether there is a target feature matching the second facial feature in the first facial feature, the method further includes: If not, the target feature is generated according to the first facial feature and the feature combination relationship, and the image data corresponding to the target feature is determined as the target image data.

5. The face replacement method according to any one of claims 1 to 4, characterized in that: The step of determining a target face for face replacement comprises: Scanning facial features in image data collected in real time by a preset image acquisition device to determine a target face for face replacement in the video picture; or, The facial features in the image data specified in the preset storage space are scanned to determine a target face for face replacement in the video frame.

6. A face replacement system, characterized in that: The face replacement system comprises: A determination module, used to determine a target face for face replacement; An extraction module, used to extract the first facial feature and feature combination relationship of the target face by using the optimized facial feature extraction model; A replacement module, used for replacing the face to be replaced in the video picture according to the first facial feature and the feature combination relationship; The face replacement system is further used to detect the feature influence level of the first facial feature of the target face under various character actions; Adapting weight parameters corresponding to each of the character actions associated with the first facial feature according to the feature influence level; The facial feature extraction model is optimized based on the weight level, wherein the facial feature extraction model is optimized based on the weight level and a characteristic distance between the first facial features, wherein the characteristic distance includes: a characteristic pixel distance for representing the position difference between the facial features and a characteristic light distance for representing the light difference between the facial features.

7. A terminal device, characterized in that: The terminal device comprises: a memory, a processor, and a face replacement program stored in the memory and executable on the processor, wherein the face replacement program, when executed by the processor, implements the steps of the face replacement method according to any one of claims 1 to 5.

8. A computer storage medium, characterized in that The computer storage medium stores a face replacement program, which, when executed by a processor, implements the steps of the face replacement method according to any one of claims 1 to 5.

Images