Face occlusion recognition, three-dimensional face processing method, device, equipment and medium

Through three-dimensional face modeling and face distortion detection methods, the convenience and accuracy of face occlusion recognition in the prior art are solved, and fast and accurate recognition and three-dimensional face processing are achieved in live broadcast scenarios.

CN114549501BActive Publication Date: 2024-11-29FOSHAN HUYA HUXIN TECH CO LTD
View PDF 7 Cites 0 Cited by

Patent Information

Application Number
CN202210191091.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-02-28
Publication Date
2024-11-29
Estimated Expiration
2042-02-28

AI Technical Summary

Technical Problem

The prior art has problems with convenience and accuracy in facial occlusion recognition, especially in live broadcast scenarios, where face occlusion is quickly and accurately recognized to achieve three-dimensional face reconstruction.

Method used

By acquiring the target face image, performing three-dimensional face modeling, the target face reconstruction parameters are obtained, and compared with the standard face parameters to detect face distortion to identify face occlusion.

Benefits of technology

It realizes simple, convenient and accurate face occlusion recognition, which can quickly respond in live broadcast scenarios and provide accurate three-dimensional face processing results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114549501B_ABST
    Figure CN114549501B_ABST
Patent Text Reader

Abstract

The embodiments of the present invention disclose a method, device, equipment and medium for recognizing face occlusion and processing three-dimensional faces. Among them, a target face image is obtained, and three-dimensional face modeling is performed on the two-dimensional face included in the target face image to obtain target face reconstruction parameters that match the two-dimensional face; based on the target face reconstruction parameters and the standard face parameters determined by the face model library, it is detected whether there is face distortion in the three-dimensional face reconstruction result; based on the face distortion detection result, it is identified whether there is face occlusion in the target face image. The technical solution of the embodiment of the present invention provides a new way to identify face occlusion through a two-dimensional face image, which can simply, conveniently and accurately determine face occlusion, and then accurately provide matching user services to users based on the face occlusion recognition result.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical fields of face recognition and three-dimensional face reconstruction, and in particular to a method, device, computer equipment and medium for recognizing face occlusion and processing three-dimensional faces. Background Art

[0002] With the continuous development of live broadcast technology, the application of face recognition and 3D face reconstruction technology is becoming more and more widespread. In the live broadcast scene, various gadgets that need to use 3D face reconstruction results have appeared, such as 3D headgear or expression masks, to enrich the interaction methods of live broadcast users.

[0003] In order to better use the results of 3D face reconstruction, various gadgets first need to identify face occlusion on the original 2D face image, and perform 3D face reconstruction on each 2D face image without face occlusion to meet the actual needs of live interactive broadcast. At present, the main face occlusion recognition technologies mainly include: pre-training a face recognition model, or comparing the similarity distance between the key points of each face in the 3D reconstructed face and the 2D face image.

[0004] In the process of implementing the present invention, the inventors found that the method of using a face recognition model requires the use of a large number of samples, a lot of time and computing power costs for pre-training, and the method of calculating the similarity distance will bring a large error rate. There is currently no convenient and accurate way to implement face occlusion recognition. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, equipment and medium for recognizing face occlusion and processing three-dimensional face, so as to provide a new way to recognize face occlusion and determine face occlusion simply, conveniently and accurately.

[0006] In a first aspect, an embodiment of the present invention provides a method for recognizing face occlusion, comprising:

[0007] Acquire a target face image, and perform three-dimensional face modeling on a two-dimensional face included in the target face image to obtain target face reconstruction parameters matching the two-dimensional face;

[0008] According to the target face reconstruction parameters and the standard face parameters determined by the face model library, detecting whether the three-dimensional face reconstruction result has face distortion;

[0009] According to the face distortion detection result, it is identified whether there is face occlusion in the target face image.

[0010] In a second aspect, an embodiment of the present invention provides a three-dimensional face processing method, comprising:

[0011] During the live video broadcast, in response to a 3D face processing request from a client, a face image frame matching the client is collected in real time;

[0012] Using the method described in any embodiment of the present invention, identifying face occlusion on each of the face image frames;

[0013] In each face image frame that is recognized as not having face occlusion, matching three-dimensional face processing is performed.

[0014] In a third aspect, an embodiment of the present invention further provides a face occlusion recognition device, the face occlusion recognition device comprising:

[0015] A target face reconstruction parameter obtaining module is used to obtain a target face image, and perform three-dimensional face modeling on a two-dimensional face included in the target face image to obtain target face reconstruction parameters matching the two-dimensional face;

[0016] A face distortion detection module, used to detect whether there is face distortion in the three-dimensional face reconstruction result according to the target face reconstruction parameters and the standard face parameters determined by the face model library;

[0017] The face occlusion recognition module is used to identify whether there is face occlusion in the target face image according to the face distortion detection result.

[0018] In a fourth aspect, an embodiment of the present invention further provides a three-dimensional face processing device, the three-dimensional face processing device comprising:

[0019] A face image frame acquisition module is used to respond to a three-dimensional face processing request from a client during a live video broadcast and to acquire face image frames matching the client in real time;

[0020] A face occlusion image frame recognition module, used to identify face occlusion on each face image frame using the method described in any embodiment of the present invention;

[0021] The three-dimensional face processing module is used to perform matching three-dimensional face processing in each face image frame that is identified as not having face occlusion.

[0022] In a fifth aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements a method for recognizing face occlusion as described in any embodiment of the present invention, or implements a three-dimensional face processing method as described in any embodiment of the present invention.

[0023] In a sixth aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the method for recognizing face occlusion as described in any embodiment of the present invention, or implements the three-dimensional face processing method as described in any embodiment of the present invention.

[0024] The technical solution provided by the embodiment of the present invention obtains a target face image and performs three-dimensional face modeling on the two-dimensional face included in the target face image to obtain target face reconstruction parameters that match the two-dimensional face; detects whether the three-dimensional face reconstruction result has face distortion based on the target face reconstruction parameters and standard face parameters determined by the face model library; and identifies whether there is face occlusion in the target face image based on the face distortion detection result. The technical solution of the embodiment of the present invention provides a new way to identify face occlusion through a two-dimensional face image, which can simply, conveniently and accurately determine face occlusion, and then accurately provide matching user services to users based on the face occlusion recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flow chart of a method for recognizing face occlusion provided by Embodiment 1 of the present invention;

[0026] Figure 2 is a flow chart of another method for recognizing face occlusion provided by Embodiment 2 of the present invention;

[0027] Figure 3 is a flow chart of a three-dimensional face processing method provided by Embodiment 3 of the present invention;

[0028] Figure 4 It is a structural schematic diagram of a face occlusion recognition device provided by a fourth embodiment of the present invention;

[0029] Figure 5 is a schematic structural diagram of a three-dimensional face processing device provided in Embodiment 5 of the present invention;

[0030] Figure 6 It is a structural diagram of a computer device provided in Embodiment 6 of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention.

[0032] It should also be noted that, for ease of description, only the part relevant to the present invention but not all content is shown in the accompanying drawings. It should be mentioned before discussing exemplary embodiments in more detail that some exemplary embodiments are described as processing or methods depicted as flow charts. Although the flow chart describes each operation (or step) as sequential processing, many operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. When its operation is completed, the processing can be terminated, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0033] The terms "first" and "second" and the like in the description, claims and drawings of the embodiments of the present invention are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may include steps or units that are not listed.

[0034] Embodiment 1

[0035] Figure 1 This is a flow chart of a method for recognizing face occlusion provided in Embodiment 1 of the present invention. This embodiment can be applied to recognize whether there is face occlusion in a two-dimensional face image. The method of this embodiment can be executed by a face occlusion recognition device, which can be implemented by software and / or hardware. The device can be configured in a computer device, which can be a server or a terminal device. The embodiment of the present invention does not limit the type of computer device.

[0036] Accordingly, the method specifically comprises the following steps:

[0037] S110, acquiring a target face image, and performing three-dimensional face modeling on a two-dimensional face included in the target face image to obtain target face reconstruction parameters matching the two-dimensional face.

[0038] The target face image may be a two-dimensional image containing a face, for example, an image captured in real time by a front camera during a live broadcast by a user. In the target face image, each pixel only includes two-dimensional position information (X, Y) but does not include depth information.

[0039] 3D face modeling may refer to reconstructing a 3D model of a face from a target face image, that is, making the reconstructed face include depth information (Z component). Target face reconstruction parameters may be multiple sets of parameters for corresponding face reconstruction when 3D face modeling is performed on a 2D face.

[0040] Optionally, performing three-dimensional face modeling on the two-dimensional face included in the target face image to obtain target face reconstruction parameters matching the two-dimensional face may include:

[0041] According to the three-dimensional deformation statistical model, a principal component analysis algorithm is used to construct a three-dimensional face reconstruction formula; the three-dimensional face reconstruction formula includes multiple face reconstruction parameters to be optimized; taking the minimum distance between the three-dimensional face reconstructed by the three-dimensional face reconstruction formula and the two-dimensional face as the optimization target, each of the face reconstruction parameters is iteratively optimized to obtain the target face reconstruction parameters.

[0042] Among them, the three-dimensional deformation statistical model (3DMM, 3D Morph Model) can be a statistical model of the three-dimensional face modeling process for the input two-dimensional face image. For the input two-dimensional face image, the three-dimensional face sample that is most similar to the input two-dimensional face image is obtained by adjusting the face reconstruction parameters of the model. 3DMM is a relatively basic three-dimensional face statistical model. It was first proposed to solve the problem of restoring three-dimensional shapes from two-dimensional face images. In the 20 years since the development of the 3DMM method, scholars have expanded its data and conducted in-depth research on it. Due to the widespread use of neural networks, the optimization of 3DMM parameters has been simplified.

[0043] The principal component analysis (PCA) algorithm is a dimensionality reduction algorithm that can convert multiple indicators into a few principal components. These principal components are linear combinations of the original variables and are independent of each other. They can reflect most of the information of the original data. Generally speaking, when the research problem involves multiple variables and there is a strong correlation between the variables, we can consider using the PCA algorithm to simplify the data.

[0044] Specifically, according to the three-dimensional deformation statistical model, when the PCA algorithm is used to construct the three-dimensional face reconstruction formula, the three-dimensional face reconstruction formula is as follows:

[0045]

[0046] Among them, X projection is the projection point of the reconstructed 3D face (i.e., the 3D face reconstruction result) projected into the 2D space; s represents the scaling parameter, which is used to scale the 3D face to align with the 2D image, and is an unknown parameter, such as 0.00125; P represents the 3*3 unit matrix, i.e., the orthogonal projection matrix; R represents the 3*3 unit orthogonal matrix, which refers to the rotation matrix. It can refer to the average face of the three-dimensional face in the PCA algorithm; m represents the shape dimension. The three-dimensional modeling of the PCA algorithm will construct the PCA basis of facial features and expressions. Because it is a high-dimensional space, it is not like the Cartesian coordinate system with only XYZ three dimensions. Here, the m value is 199, which means 199 dimensions; α i It refers to the 199-dimensional PCA basis of facial features. Its dimension is 199*65536*3. 65536 refers to the facial feature points included in the three-dimensional face, and *3 refers to the position parameter (XYZ) of each facial feature point. n refers to the expression dimension, which has a similar meaning to m. It is generally set to 29 and is a fixed parameter. i represents the shape coefficient, which is 199 unknowns. i The PCA basis representing facial expressions, and α i Correspondingly, the dimension is 29*65536*3. i Represents the expression coefficient, and s i Similar in meaning, it is 29 unknown numbers. 2d Represents the offset matrix, which is the parameter used to align the reconstructed 3D face with the 2D image.

[0047] It is understandable that the three-dimensional face reconstruction formula includes multiple face reconstruction parameters to be optimized: s, R, s i 、e i and t 2d ; Then, by calculating the distance between the 3D face reconstructed by the 3D face reconstruction formula and the 2D face, taking the minimum distance between the two as the optimization target, the face reconstruction parameters are iteratively optimized to obtain the target face reconstruction parameters, that is, the optimal face reconstruction parameters s, R, s i 、e i and t 2d .

[0048] The advantage of this arrangement is that by performing 3D face modeling on the 2D face included in the target face image, target face reconstruction parameters matching the 2D face are obtained. By taking the minimum distance between the 3D face reconstructed by the 3D face reconstruction formula and the 2D face as the optimization target, each face reconstruction parameter is iteratively optimized to obtain the target face reconstruction parameters. In this way, more accurate target face reconstruction parameters can be calculated, so that face occlusion in the 2D face image can be more accurately determined.

[0049] S120. Detect whether a three-dimensional face reconstruction result has face distortion according to the target face reconstruction parameters and standard face parameters determined by a face model library.

[0050] The face model library may be a model library composed of a plurality of standard face samples, and the standard face parameters may be face parameters determined by the face feature points of each standard face sample included in the face model library.

[0051] Optionally, the face model library may be a Surrey Face Model (SFM) library, a 4D Face Model (4DFM) library, or a Basel Face Model (BFM) library. As mentioned above, when performing 3D face reconstruction using a 3D face reconstruction formula, it is necessary to use the PCA basis of face shape and facial expression, that is, α i and e i The PCA basis is determined by combining the facial feature points of each standard face sample in the face model library. Furthermore, each PCA basis can be determined as a standard face parameter.

[0052] It is understandable that since the standard face parameters are determined using the standard face samples in the face model library, they can be used as a measure of a normal face. In other words, if the difference between the target face reconstruction parameters of the reconstructed 3D face (3D face reconstruction result) and the standard face parameters is smaller, the possibility of distortion of the 3D face is lower, and the possibility of face occlusion in the matched 2D face image is also lower; conversely, if the difference between the target face reconstruction parameters of the reconstructed 3D face and the standard face parameters is greater, the possibility of distortion of the 3D face is higher, and the possibility of face occlusion in the matched 2D face image is also higher.

[0053] Correspondingly, the parameter difference value between the target face reconstruction parameters and the standard face parameters determined by the face model library can be obtained, so that it can be determined whether the three-dimensional face reconstruction result has face distortion based on the parameter difference value.

[0054] S130. Identify whether there is face occlusion in the target face image according to the face distortion detection result.

[0055] In an optional implementation of this embodiment, when it is determined that the three-dimensional face reconstruction result has face distortion, it can be directly determined that there is face occlusion in the target face image;

[0056] In another optional implementation of this embodiment, when it is determined that the three-dimensional face reconstruction result has face distortion, in order to further reduce the misjudgment rate, other detection algorithms can be further combined to assist in determining whether there is face occlusion in the target face image.

[0057] For example, the similarity distance between each facial key point in the reconstructed three-dimensional face and each facial key point in the two-dimensional face can be further calculated. If it is determined that the similarity distance between the facial key points in the three-dimensional face and the two-dimensional face is also relatively far, it can be determined that there is face occlusion in the target face image.

[0058] Optionally, when it is determined that the three-dimensional face reconstruction result does not have facial distortion, it can be determined that the three-dimensional face obtained after the three-dimensional face reconstruction is visually a normal face, and further, it can be directly determined that there is no facial occlusion in the target face image.

[0059] The technical solution provided by the embodiment of the present invention obtains a target face image and performs three-dimensional face modeling on the two-dimensional face included in the target face image to obtain target face reconstruction parameters that match the two-dimensional face; detects whether the three-dimensional face reconstruction result has face distortion based on the target face reconstruction parameters and standard face parameters determined by the face model library; and identifies whether there is face occlusion in the target face image based on the face distortion detection result. The technical solution of the embodiment of the present invention provides a new way to identify face occlusion through a two-dimensional face image, which can simply, conveniently and accurately determine face occlusion, and then accurately provide matching user services to users based on the face occlusion recognition result.

[0060] Embodiment 2

[0061] Figure 2 This is a flow chart of another method for identifying face occlusion provided by Embodiment 2 of the present invention. This embodiment is refined based on the above embodiments. In this embodiment, the operation of detecting whether there is face distortion in the three-dimensional face reconstruction result according to the target face reconstruction parameters and the standard face parameters determined by the face model library is further refined, and the operation of identifying whether there is face occlusion in the target face image according to the face distortion detection result is further refined.

[0062] Accordingly, the method specifically comprises the following steps:

[0063] S210: Acquire a target face image, and perform three-dimensional face modeling on a two-dimensional face included in the target face image to obtain target face reconstruction parameters matching the two-dimensional face.

[0064] S220. Extracting a target deformation coefficient from the target face reconstruction parameters, wherein the target deformation coefficient includes feature fitting values ​​of the two-dimensional face in multiple target dimensions.

[0065] The target deformation coefficient may be a related deformation coefficient determined according to the target face reconstruction parameter. Specifically, the target deformation coefficient includes a feature fitting value of a two-dimensional face in multiple target dimensions.

[0066] Continuing from the previous example, the target deformation coefficient specifically refers to s among the face reconstruction parameters obtained by iteratively solving the aforementioned three-dimensional face reconstruction formula. i , that is, the feature fitting value of the 3D face reconstruction result in each of the above 199 dimensions. Each dimension corresponds to an abstract face feature.

[0067] S230. Obtain standard deformation coefficients determined by each face model in the face model library, wherein the standard deformation coefficients include dimension feature values ​​corresponding to each target dimension when constructing each target dimension.

[0068] As mentioned above, the standard face parameters determined by the face model library may include the PCA basis of the face shape, and then the eigenvalue of each dimension of the PCA basis of the face shape in 199 dimensions may be further obtained as the standard deformation coefficient.

[0069] S240. Detect whether there is facial distortion in the three-dimensional face reconstruction result according to the difference between the target deformation coefficient and the standard deformation coefficient.

[0070] Optionally, detecting whether a three-dimensional face reconstruction result has face distortion according to a difference between the target deformation coefficient and the standard deformation coefficient may include:

[0071] Respectively calculate the absolute difference value between the feature fitting value and the dimension feature value of the target deformation coefficient and the standard deformation coefficient in each target dimension; accumulate and sum the absolute difference values ​​of each target dimension to obtain an accumulated difference value; if it is determined that the accumulated difference value exceeds the feature difference threshold, it is determined that the two-dimensional face has facial distortion.

[0072] The absolute difference value may be a value used to measure the difference between the two. Specifically, the larger the absolute difference value, the larger the difference between the two; the smaller the absolute difference value, the smaller the difference between the two. The cumulative difference value may be obtained by superimposing the absolute difference values ​​of different dimensions to obtain the corresponding difference value. The feature difference threshold may be a threshold for determining whether the three-dimensional face reconstruction result has face distortion. When the cumulative difference value exceeds the feature difference threshold, it is determined that the three-dimensional face reconstruction result has face distortion; when the cumulative difference value does not exceed the feature difference threshold, it is determined that the three-dimensional face reconstruction result does not have face distortion.

[0073] For example, assuming that the target dimensions are the above 199 dimensions, the feature difference threshold is D 阈值 Calculate the 199 absolute difference values ​​(the absolute value of the difference between the two) between the characteristic fitting value and the dimension characteristic value of the target deformation coefficient and the standard deformation coefficient in each of the above 199 dimensions, and add up the above 199 absolute difference values ​​to obtain the cumulative difference value D 总 If the cumulative difference value D is determined 总 Exceeding the feature difference threshold D 阈值 , that is, D 总 >D 阈值 , it is determined that the three-dimensional face reconstruction result has face distortion; if the accumulated difference value D 总 Does not exceed the feature difference threshold D 阈值 , that is, D 总 ≤D 阈值 It is determined that there is no facial distortion in the three-dimensional face reconstruction result.

[0074] The advantage of this setting is that by calculating the absolute difference between the feature fitting value and the dimension feature value of the target deformation coefficient and the standard deformation coefficient in each target dimension, and obtaining the cumulative difference value, the cumulative difference value and the feature difference threshold are compared to determine whether there is face distortion in the three-dimensional face reconstruction result. In this way, it is possible to conveniently and accurately determine whether there is distortion in the three-dimensional face reconstruction result, so as to accurately judge whether there is face occlusion in the two-dimensional face, and only the parameters generated in the three-dimensional face modeling process can be used for judgment, without consuming a lot of computing power and time costs to specially train the face occlusion recognition model, while improving the effectiveness and reliability of face occlusion recognition.

[0075] S250: If it is determined that the face distortion detection result shows that face distortion exists, a target three-dimensional face reconstructed by using the target face reconstruction parameters is obtained.

[0076] As mentioned above, after solving and obtaining the optimal values ​​of various face reconstruction parameters in the three-dimensional face reconstruction formula, a matching target three-dimensional face can be reconstructed based on the above-mentioned face reconstruction parameters.

[0077] S260: Identify whether there is face occlusion in the target face image based on the spatial distance between the target three-dimensional face and at least one facial key point in the two-dimensional face.

[0078] Optionally, identifying whether there is face occlusion in the target face image according to the spatial distance between the target three-dimensional face and at least one face key point in the two-dimensional face may include:

[0079] A plurality of three-dimensional facial feature sparse points are extracted from the target three-dimensional face, and a plurality of two-dimensional facial feature sparse points are extracted from the two-dimensional face; the spatial distance values ​​between each of the three-dimensional facial feature sparse points and each of the two-dimensional facial feature sparse points are calculated respectively, and each of the spatial distance values ​​is accumulated and summed to obtain a summary spatial distance value; if it is determined that the summary spatial distance value exceeds a spatial distance value threshold, it is determined that there is face occlusion in the target face image.

[0080] Among them, the three-dimensional facial feature sparse points can be the facial feature points used to describe the shapes of the facial features in the target three-dimensional face. Generally speaking, after the target three-dimensional face is reconstructed, each facial feature point in the target three-dimensional face has semantic features. Then, the three-dimensional facial feature sparse points can be extracted from the target three-dimensional face based on the above semantic features.

[0081] Similarly, the two-dimensional facial features sparse points can be facial feature points used to describe the shapes of facial features in a two-dimensional face. Similarly, when a two-dimensional face is identified in a target face image, each facial feature point in the two-dimensional face also has semantic features, and then, the two-dimensional facial features sparse points can be extracted from the two-dimensional face based on the semantic features.

[0082] The spatial distance value may be a value used to describe the distance between each three-dimensional facial feature sparse point and each two-dimensional facial feature sparse point. Specifically, the spatial distance value between each three-dimensional facial feature sparse point and each two-dimensional facial feature sparse point is calculated using the Euclidean distance. The aggregated spatial distance value may be a summarized value obtained by accumulating multiple spatial distance values ​​calculated using different facial feature sparse points. The spatial distance value threshold may be a threshold for determining whether there is face occlusion in the target facial image. When the aggregated spatial distance value exceeds the spatial distance value threshold, it is determined that there is face occlusion in the target facial image; when the aggregated spatial distance value does not exceed the spatial distance value threshold, it is determined that there is no face occlusion in the target facial image.

[0083] For example, assume that five 3D facial sparse points are extracted from the target 3D face, namely {(X1, Y1, Z1); (X2, Y2, Z2); (X3, Y3, Z3); (X4, Y4, Z4); (X5, Y5, Z5)}, and five 2D facial sparse points are extracted from the 2D face, namely {(X1', Y1'); (X2', Y2'); (X3', Y3'); (X4', Y4'); (X5', Y5')}. Assume that the spatial distance value threshold is D 空间 . To calculate the spatial distance values ​​between each 3D facial feature sparse point and each 2D facial feature sparse point, we only need to consider the X-axis and Y-axis values ​​of the 3D facial feature sparse point and calculate with each 2D facial feature sparse point to obtain the corresponding spatial distance values, which are D1, D2, D3, D4 and D5. Furthermore, we accumulate and sum each spatial distance value to obtain the summary spatial distance value D汇总 =D1+D2+D3+D4+D5. If the summary spatial distance value D is determined 汇总 Exceeding the spatial distance value threshold D 空间 , that is, D 汇总 >D 空间 , it is determined that there is face occlusion in the target face image; if the summary spatial distance value D 汇总 Does not exceed the spatial distance value threshold D 空间 , that is, D 汇总 ≤D 空间 , it is determined that there is no face occlusion in the target face image.

[0084] Generally speaking, the semantics of three-dimensional facial feature sparse points and two-dimensional facial feature sparse points are generally different. Therefore, the two will inherently have a certain deviation. Therefore, when selecting the spatial distance value threshold, a larger value can be selected to avoid misjudgment to the greatest extent.

[0085] Specifically, through experiments, the spatial distance empirical value between at least one facial key point between a two-dimensional face and a matching three-dimensional face (without occlusion) can be calculated, and then a set multiple of the spatial distance empirical value, such as 1.5 times or 2 times, can be selected as the spatial distance value threshold.

[0086] The advantage of this setting is that: by extracting multiple three-dimensional facial features sparse points and multiple two-dimensional facial features sparse points, the spatial distance values ​​between the two are calculated respectively, and the summary spatial distance value is obtained by cumulative summation. By comparing the size of the summary spatial distance value and the spatial distance value threshold value, it is determined whether there is face occlusion in the target face image. In this way, the result obtained by calculation is more accurate, so that it can accurately judge whether there is face occlusion, and at the same time improve the effectiveness and reliability of face occlusion recognition.

[0087] The technical solution provided by the embodiment of the present invention obtains a target face image, and performs three-dimensional face modeling on the two-dimensional face included in the target face image to obtain target face reconstruction parameters matching the two-dimensional face; extracts the target deformation coefficient from the target face reconstruction parameters; obtains the standard deformation coefficient determined by each face model in the face model library; detects whether there is face distortion in the three-dimensional face reconstruction result according to the difference between the target deformation coefficient and the standard deformation coefficient; if it is determined that the face distortion detection result is that there is face distortion, obtains the target three-dimensional face reconstructed by the target face reconstruction parameters; and identifies whether there is face occlusion in the target face image according to the spatial distance between the target three-dimensional face and at least one face key point in the two-dimensional face. The technical solution of the embodiment of the present invention provides a new way to identify face occlusion through a two-dimensional face image, which can simply, conveniently and accurately determine face occlusion, and then accurately provide matching user services to users based on the face occlusion recognition result. In this way, it is possible to accurately determine whether there is face occlusion in a two-dimensional face by using only the parameters generated during the three-dimensional face modeling process. This eliminates the need to spend a lot of computing power and time to specifically train a face occlusion recognition model, while also improving the effectiveness and reliability of face occlusion recognition.

[0088] Embodiment 3

[0089] Figure 3 : is a flow chart of a 3D face processing method provided by Embodiment 3 of the present invention. This embodiment is applicable to the case where a user performs 3D special effects adding processing. The method of this embodiment can be executed by a 3D face processing device, which can be implemented by software and / or hardware. The device can be configured in a computer device, which can be a server. The embodiment of the present invention does not limit the type of computer device.

[0090] Accordingly, the method specifically comprises the following steps:

[0091] S310. During the live video broadcast, in response to a three-dimensional face processing request from a client, collect face image frames matching the client in real time.

[0092] The 3D face processing request may be a request from a user to add a 3D special effect (for example, to add a 3D headgear or other head decorations, such as glasses, to the current displayed image), and the request command is initiated to the server. The face image frame may be a single frame face image obtained by collecting face images through a client camera.

[0093] S320: performing face occlusion recognition on each of the face image frames.

[0094] In this embodiment, the method for identifying facial occlusion for each of the facial image frames can be: obtaining a target facial image, and performing three-dimensional facial modeling on the two-dimensional face included in the target facial image to obtain target facial reconstruction parameters that match the two-dimensional face; detecting whether there is facial distortion in the three-dimensional facial reconstruction result based on the target facial reconstruction parameters and standard facial parameters determined by a facial model library; and identifying whether there is facial occlusion in the target facial image based on the facial distortion detection result.

[0095] S330: Perform matching three-dimensional face processing in each face image frame that is identified as not having face occlusion.

[0096] In an optional implementation of this embodiment, suppose that the anchor wants to add three-dimensional special effects during the live broadcast, such as three-dimensional special effects with cute decorations such as rabbit masks. Accordingly, the anchor can send a three-dimensional face processing request to the live broadcast server through the anchor client. The server receives the three-dimensional face processing request and responds, so the server collects the face image frame that matches the client in real time, that is, the target face image of the anchor. Further, when the target face image of the anchor is collected, it is necessary to determine whether there is a problem of face occlusion in the collected target face image. If it is determined that there is no face occlusion, matching three-dimensional face processing is performed on each face image frame. That is, the addition of three-dimensional special effects such as cute decorations such as rabbit masks is performed in the live broadcast screen.

[0097] In another optional implementation of this embodiment, suppose that the anchor wants to add a three-dimensional special effect of an animal expression mask during the live broadcast process. Specifically, the anchor makes multiple groups of expressions, and the corresponding animal expression mask will also make related expressions. Correspondingly, the anchor can send a three-dimensional face processing request to the live broadcast server through the anchor client. The server receives the three-dimensional face processing request and responds, so the server collects the face image frame matching the client in real time, that is, the face image of the anchor. Further, when the target face image of the anchor is collected, it is necessary to determine whether there is a face occlusion problem in the collected target face image. If it is determined that there is no face occlusion, the matching three-dimensional face processing is performed on each face image frame. That is, the relevant three-dimensional special effects are added in the live broadcast screen, that is, the anchor makes multiple groups of expressions, and the corresponding animal expression mask will also make related expressions. If it is determined that there is face occlusion, the matching three-dimensional face processing is not performed on each face image frame. That is, the face image frame is not processed, the face image frame is skipped, and the recognition of the next face image frame is waited.

[0098] The technical solution provided by the embodiment of the present invention responds to the client's three-dimensional face processing request during the live video broadcast, collects face image frames matching the client in real time; uses the method described in any embodiment of the present invention to identify face occlusion on each face image frame; and performs matching three-dimensional face processing in each face image frame that is identified as not having face occlusion. The problem of accurately judging three-dimensional face occlusion and face boundaries when adding special effects to the user's three-dimensional face is solved, and the accurate addition of special effects to the three-dimensional face is achieved, which improves the user experience.

[0099] Embodiment 4

[0100] Figure 4 : is a schematic diagram of the structure of a face occlusion recognition device provided in the fourth embodiment of the present invention. The face occlusion recognition device provided in this embodiment can be implemented by software and / or hardware, and can be configured in a server or terminal device to implement a face occlusion recognition method in the embodiment of the present invention. Figure 4 As shown, the device may specifically include: a target face reconstruction parameter obtaining module 410, a face distortion detection module 420 and a face occlusion recognition module 430.

[0101] The target face reconstruction parameter obtaining module 410 is used to obtain a target face image, and perform three-dimensional face modeling on a two-dimensional face included in the target face image to obtain target face reconstruction parameters matching the two-dimensional face;

[0102] A face distortion detection module 420 is used to detect whether there is face distortion in the three-dimensional face reconstruction result according to the target face reconstruction parameters and the standard face parameters determined by the face model library;

[0103] The face occlusion recognition module 430 is used to identify whether there is face occlusion in the target face image according to the face distortion detection result.

[0104] The technical solution provided by the embodiment of the present invention obtains a target face image and performs three-dimensional face modeling on the two-dimensional face included in the target face image to obtain target face reconstruction parameters that match the two-dimensional face; detects whether the three-dimensional face reconstruction result has face distortion based on the target face reconstruction parameters and standard face parameters determined by the face model library; and identifies whether there is face occlusion in the target face image based on the face distortion detection result. The technical solution of the embodiment of the present invention provides a new way to identify face occlusion through a two-dimensional face image, which can simply, conveniently and accurately determine face occlusion, and then accurately provide matching user services to users based on the face occlusion recognition result.

[0105] On the basis of the above embodiments, the target face reconstruction parameter obtaining module 410 can be specifically used to: construct a three-dimensional face reconstruction formula based on a three-dimensional deformation statistical model and a principal component analysis algorithm; the three-dimensional face reconstruction formula includes multiple face reconstruction parameters to be optimized; taking the minimum distance between the three-dimensional face reconstructed by the three-dimensional face reconstruction formula and the two-dimensional face as the optimization target, iteratively optimize each of the face reconstruction parameters to obtain the target face reconstruction parameters.

[0106] On the basis of the above embodiments, the face distortion detection module 420 may specifically include: a target deformation coefficient extraction unit, used to extract the target deformation coefficient from the target face reconstruction parameters, wherein the target deformation coefficient includes the characteristic fitting values ​​of the two-dimensional face under multiple target dimensions; a standard deformation coefficient determination unit, used to obtain the standard deformation coefficient determined by each face model in the face model library, wherein the standard deformation coefficient includes the dimensional characteristic values ​​corresponding to each target dimension when constructing each target dimension; a face distortion unit, used to detect whether there is face distortion in the three-dimensional face reconstruction result according to the difference between the target deformation coefficient and the standard deformation coefficient.

[0107] Based on the above embodiments, the face distortion unit can be specifically used to: respectively calculate the absolute difference value between the feature fitting value and the dimension feature value of the target deformation coefficient and the standard deformation coefficient in each target dimension; accumulate and sum the absolute difference values ​​of each target dimension to obtain an accumulated difference value; if it is determined that the accumulated difference value exceeds the feature difference threshold, it is determined that the three-dimensional face reconstruction result has face distortion.

[0108] Based on the above embodiments, the face occlusion recognition module 430 may specifically include: a target three-dimensional face reconstruction unit, which is used to obtain a target three-dimensional face reconstructed by the target face reconstruction parameters if it is determined that the face distortion detection result is that face distortion exists; and a face occlusion recognition unit, which is used to identify whether there is face occlusion in the target face image based on the spatial distance between the target three-dimensional face and at least one face key point in the two-dimensional face.

[0109] Based on the above embodiments, the face occlusion recognition unit can be specifically used to: extract multiple three-dimensional facial feature sparse points in the target three-dimensional face, and extract multiple two-dimensional facial feature sparse points in the two-dimensional face; respectively calculate the spatial distance values ​​between each of the three-dimensional facial feature sparse points and each of the two-dimensional facial feature sparse points, and add up the spatial distance values ​​to obtain a summary spatial distance value; if it is determined that the summary spatial distance value exceeds the spatial distance value threshold, it is determined that there is face occlusion in the target face image.

[0110] The above-mentioned face occlusion recognition device can execute the face occlusion recognition method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0111] Embodiment 5

[0112] Figure 5 : is a schematic diagram of the structure of a three-dimensional face processing device provided in Embodiment 5 of the present invention. The three-dimensional face processing device provided in this embodiment can be implemented by software and / or hardware, and can be configured in a server to implement a three-dimensional face processing method in the embodiment of the present invention. Figure 5 As shown, the device may specifically include: a face image frame acquisition module 510 , a face image frame recognition module 520 and a three-dimensional face processing module 530 .

[0113] The face image frame acquisition module 510 is used to respond to the client's three-dimensional face processing request during the live video broadcast and acquire the face image frame matching the client in real time;

[0114] A face image frame recognition module 520, configured to recognize face occlusion on each face image frame using a method described in any embodiment of the present invention;

[0115] The three-dimensional face processing module 530 is used to perform matching three-dimensional face processing in each face image frame that is identified as not having face occlusion.

[0116] The technical solution provided by the embodiment of the present invention responds to the client's three-dimensional face processing request during the live video broadcast, collects face image frames matching the client in real time; uses the method described in any embodiment of the present invention to identify face occlusion on each face image frame; and performs matching three-dimensional face processing in each face image frame that is identified as not having face occlusion. The problem of accurately judging three-dimensional face occlusion and face boundaries when adding special effects to the user's three-dimensional face is solved, and the accurate addition of special effects to the three-dimensional face is achieved, which improves the user experience.

[0117] The above-mentioned three-dimensional face processing device can execute the three-dimensional face processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0118] Embodiment 6

[0119] Figure 6 Schematic diagram of the structure of a computer device provided by Embodiment 6 of the present invention. Figure 6 As shown, the device includes a processor 610, a memory 620, an input device 630, and an output device 640; the number of processors 610 in the device can be one or more. Figure 6A processor 610 is taken as an example; the processor 610, memory 620, input device 630 and output device 640 in the device can be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0120] The memory 620, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the face occlusion recognition method in the embodiment of the present invention (for example, the target face reconstruction parameter acquisition module 410, the face distortion detection module 420 and the face occlusion recognition module 430). The processor 610 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 620, that is, to implement the above-mentioned face occlusion recognition method, which includes: obtaining a target face image, and performing three-dimensional face modeling on the two-dimensional face included in the target face image to obtain a target face reconstruction parameter matching the two-dimensional face; detecting whether the three-dimensional face reconstruction result has face distortion according to the target face reconstruction parameter and the standard face parameter determined by the face model library; and identifying whether there is face occlusion in the target face image according to the face distortion detection result.

[0121] The memory 620, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, or program instructions / modules corresponding to the three-dimensional face processing method in the embodiment of the present invention (for example, the face image frame acquisition module 510, the face image frame recognition module 520 and the three-dimensional face processing module 530). The processor 610 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 620, that is, to implement the above-mentioned three-dimensional face processing method, or the method includes: in the process of live video broadcast, in response to the three-dimensional face processing request of the client, real-time acquisition of face image frames matching the client; using the method described in any embodiment of the present invention to identify face occlusion for each face image frame; in each face image frame identified as not having face occlusion, matching three-dimensional face processing is performed.

[0122] The memory 620 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 620 may further include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0123] The input device 630 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 640 may include a display device such as a display screen.

[0124] Embodiment 7

[0125] Embodiment 7 of the present invention also provides a computer-readable storage medium, wherein the computer-readable instructions are used to execute a method for identifying face occlusion when executed by a computer processor, the method comprising: obtaining a target face image, and performing three-dimensional face modeling on the two-dimensional face included in the target face image to obtain target face reconstruction parameters that match the two-dimensional face; detecting whether there is face distortion in the three-dimensional face reconstruction result based on the target face reconstruction parameters and standard face parameters determined by a face model library; and identifying whether there is face occlusion in the target face image based on the face distortion detection result.

[0126] Alternatively, the computer-readable instructions are used to execute a three-dimensional face processing method when executed by a computer processor, the method comprising: during a live video broadcast, in response to a three-dimensional face processing request from a client, real-time acquisition of face image frames matching the client; using a method as described in any embodiment of the present invention to identify face occlusion on each of the face image frames; and performing matching three-dimensional face processing on each face image frame that is identified as not having face occlusion.

[0127] Of course, the computer-readable storage medium provided in an embodiment of the present invention has computer-readable instructions that are not limited to the method operations described above, but can also execute related operations in the face occlusion recognition method or three-dimensional face processing method provided in any embodiment of the present invention.

[0128] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (FLASH), a hard disk or an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0129] It is worth noting that in the embodiments of the above-mentioned face occlusion recognition device or three-dimensional face processing device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0130] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for recognizing face occlusion, characterized in that: include: Acquire a target face image, and perform three-dimensional face modeling on a two-dimensional face included in the target face image to obtain target face reconstruction parameters matching the two-dimensional face; According to the target face reconstruction parameters and the standard face parameters determined by the face model library, detecting whether the three-dimensional face reconstruction result has face distortion; According to the face distortion detection result, identifying whether there is face occlusion in the target face image; The method of detecting whether a three-dimensional face reconstruction result has face distortion according to the target face reconstruction parameters and the standard face parameters determined by the face model library includes: Extracting a target deformation coefficient from the target face reconstruction parameters, wherein the target deformation coefficient includes feature fitting values ​​of the two-dimensional face in multiple target dimensions; Obtaining a standard deformation coefficient determined by each face model in the face model library, wherein the standard deformation coefficient includes dimension feature values ​​corresponding to each target dimension when constructing each target dimension; According to the difference between the target deformation coefficient and the standard deformation coefficient, it is detected whether there is facial distortion in the three-dimensional face reconstruction result.

2. The method according to claim 1, characterized in that: Performing three-dimensional face modeling on the two-dimensional face included in the target face image to obtain target face reconstruction parameters matching the two-dimensional face includes: According to the three-dimensional deformation statistical model, a principal component analysis algorithm is used to construct a three-dimensional face reconstruction formula; the three-dimensional face reconstruction formula includes multiple face reconstruction parameters to be optimized; Taking the minimum distance between the three-dimensional face reconstructed by the three-dimensional face reconstruction formula and the two-dimensional face as the optimization target, each of the face reconstruction parameters is iteratively optimized to obtain the target face reconstruction parameters.

3. The method according to claim 1, characterized in that Detecting whether a three-dimensional face reconstruction result has face distortion according to the difference between the target deformation coefficient and the standard deformation coefficient, including: Calculate the absolute difference between the characteristic fitting value and the dimension characteristic value of the target deformation coefficient and the standard deformation coefficient in each target dimension respectively; Accumulating and summing the absolute difference values ​​of each target dimension to obtain an accumulated difference value; If it is determined that the accumulated difference value exceeds the feature difference threshold, it is determined that the three-dimensional face reconstruction result has face distortion.

4. The method according to any one of claims 1 to 3, characterized in that: According to the face distortion detection result, identifying whether there is face occlusion in the target face image includes: If it is determined that the face distortion detection result is that face distortion exists, obtaining a target three-dimensional face reconstructed by the target face reconstruction parameters; According to the spatial distance between the target three-dimensional face and at least one facial key point in the two-dimensional face, it is identified whether there is facial occlusion in the target facial image.

5. The method according to claim 4, characterized in that Identifying whether there is face occlusion in the target face image according to the spatial distance between the target three-dimensional face and at least one face key point in the two-dimensional face includes: Extracting a plurality of sparse points of three-dimensional facial features from the target three-dimensional human face, and extracting a plurality of sparse points of two-dimensional facial features from the two-dimensional human face; Respectively calculating the spatial distance values ​​between each of the three-dimensional facial feature sparse points and each of the two-dimensional facial feature sparse points, and accumulating and summing the spatial distance values ​​to obtain a summary spatial distance value; If it is determined that the summarized spatial distance value exceeds the spatial distance value threshold, it is determined that there is face occlusion in the target face image.

6. A three-dimensional face processing method, characterized in that: include: During the live video broadcast, in response to a 3D face processing request from a client, a face image frame matching the client is collected in real time; Using the method according to any one of claims 1 to 5, identifying face occlusion on each of the face image frames; In each face image frame that is recognized as not having face occlusion, matching three-dimensional face processing is performed.

7. A face occlusion recognition device, characterized in that: include: A target face reconstruction parameter obtaining module is used to obtain a target face image, and perform three-dimensional face modeling on a two-dimensional face included in the target face image to obtain target face reconstruction parameters matching the two-dimensional face; A face distortion detection module, used to detect whether there is face distortion in the three-dimensional face reconstruction result according to the target face reconstruction parameters and the standard face parameters determined by the face model library; A face occlusion recognition module, used to identify whether there is face occlusion in the target face image according to the face distortion detection result; Among them, the face distortion detection module includes: A target deformation coefficient extraction unit, used to extract a target deformation coefficient from the target face reconstruction parameters, wherein the target deformation coefficient includes a feature fitting value of the two-dimensional face in multiple target dimensions; A standard deformation coefficient determination unit, used to obtain a standard deformation coefficient determined by each face model in the face model library, wherein the standard deformation coefficient includes dimension feature values ​​corresponding to each target dimension when constructing each target dimension; The face distortion unit is used to detect whether there is face distortion in the three-dimensional face reconstruction result according to the difference between the target deformation coefficient and the standard deformation coefficient.

8. A three-dimensional face processing device, characterized in that: include: A face image frame acquisition module is used to respond to a three-dimensional face processing request from a client during a live video broadcast and to acquire face image frames matching the client in real time; A face image frame recognition module, configured to recognize face occlusion on each face image frame using the method according to any one of claims 1 to 5; The three-dimensional face processing module is used to perform matching three-dimensional face processing in each face image frame that is identified as not having face occlusion.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements the face occlusion recognition method as described in any one of claims 1 to 5, or implements the three-dimensional face processing method as described in claim 6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the face occlusion recognition method as described in any one of claims 1 to 5, or implements the three-dimensional face processing method as described in claim 6.

Citation Information

Patent Citations

  • Three-dimensional face reconstruction method

    CN102999942A

  • Face posture correction method and face posture correction device

    CN105844276A

  • Face reconstruction method, device and equipment and storage medium

    CN109685873A

  • Three-dimensional face reconstruction method, apparatus and device, and storage medium

    CN110956691A

  • Face image restoration method and system

    CN112001859A