Face expression determination method and apparatus under face occlusion

Abstract

The present disclosure provides a method for determining facial expression under face occlusion, comprising: detecting whether a first image in a video has face occlusion; in response to a first region of the face being occluded, determining, based on the first image, a first vector including expression values related to the first region of the face, and a second vector including expression values related to regions of the face other than the first region; adjusting the first vector to obtain a third vector according to the second vector based on face symmetry; adjusting the third vector to obtain a fourth vector based on linkage relationships between facial parts; determining a fifth vector based on a second image before the first image in the video, the first region of the face in the second image being unoccluded, the fifth vector including expression values related to the first region of the face in the second image, adjusting the fourth vector to obtain a sixth vector based on the fifth vector; and adjusting a three-dimensional face model established based on the first image based on the sixth vector. The present disclosure also relates to a device for determining facial expression under face occlusion.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to a method and device for determining facial expression under facial occlusion. BACKGROUND

[0002] Facial expression recognition technology is a comprehensive technology involving image processing, machine learning, computer vision, and other fields. It can recognize the emotional state of a person by analyzing the feature points and expressions of the face. Facial expression recognition technology can be widely used in face recognition, sentiment analysis, human-computer interaction, and security monitoring. Facial expression recognition technology mainly includes face frame detection, face feature extraction, and facial expression recognition. SUMMARY

[0003] One of the purposes of one or more embodiments of the present disclosure is to provide a method and device for determining facial expression under facial occlusion, and a computer readable storage medium.

[0004] According to a first aspect of an embodiment of the present disclosure, a method for determining facial expression under facial occlusion is provided, which includes receiving a video containing a face, detecting whether there is facial occlusion in a first image of the video, in response to a first region of the face in the first image being occluded, determining a facial expression vector based on the first image, the facial expression vector including a first vector and a second vector, the first vector including one or more expression values related to the first region of the face, and the second vector including one or more expression values related to a region of the face other than the first region, each expression value being used to represent the state of a part of the face, adjusting the first vector according to the second vector based on facial symmetry to obtain a third vector, adjusting the third vector based on the linkage relationship between the parts of the face to obtain a fourth vector, determining one or more fifth vectors based on one or more second images located before the first image in the video, the first region of the face in the one or more second images being unoccluded, each of the one or more fifth vectors including one or more expression values related to the first region of the face in the corresponding second image in the one or more second images, adjusting the fourth vector based on the one or more fifth vectors to obtain a sixth vector, and adjusting a three-dimensional face model established based on the first image based on the sixth vector.

[0005] According to a second aspect of the embodiments of the present disclosure, a method for determining facial expression under face occlusion is provided, including: receiving a video containing a face, detecting whether there is face occlusion in a first image of the video; in response to a first region of the face in the first image being occluded, determining a first facial expression vector based on the first image, the first facial expression vector including a first sub-vector and a second sub-vector, the first sub-vector including one or more expression values related to the first region of the face, and the second sub-vector including one or more expression values related to a region of the face other than the first region, each expression value being used to represent a state of a part of the face; adjusting the first sub-vector according to the second sub-vector based on facial symmetry, determining a second facial expression vector based on the adjusted first sub-vector and the second sub-vector; determining one or more third sub-vectors based on one or more second images of the video located before the first image, the first region of the face in the one or more second images being unoccluded, each of the one or more third sub-vectors including one or more expression values related to the first region of the face in a corresponding second image of the one or more second images, adjusting the second facial expression vector based on the one or more third sub-vectors, and determining a third facial expression vector; adjusting the third facial expression vector based on a linkage relationship between facial parts, and determining a fourth facial expression vector; and adjusting a three-dimensional face model established based on the first image based on the fourth facial expression vector.

[0006] According to a third aspect of the embodiments of the present disclosure, a device for determining facial expression under face occlusion is provided, including: a memory; and a processor coupled to the memory, the processor being configured to execute the method of any one of the above-mentioned embodiments based on instructions stored in the memory.

[0007] According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, including computer program instructions, wherein the computer program instructions are executed by a processor to implement the method of any one of the above-mentioned embodiments.

[0008] According to a fifth aspect of the embodiments of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program is executed by a processor to implement the method of any one of the above-mentioned embodiments.

[0009] The technical solutions of the present disclosure are described in further detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0010] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 and Figure 2 These are schematic flowcharts illustrating methods for determining facial expressions under face occlusion according to some embodiments of this disclosure.

[0012] Figure 3 This is a schematic diagram of a first image, a second image, and a first region in a method for determining facial expressions under face occlusion according to some embodiments of the present disclosure.

[0013] Figure 4 and Figure 5 These are schematic flowcharts illustrating methods for determining facial expressions under face occlusion according to some embodiments of this disclosure.

[0014] Figure 6 This is a structural schematic diagram of a facial expression determination device under facial occlusion according to some embodiments of the present disclosure. Detailed Implementation

[0015] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0016] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0017] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0018] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0019] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0021] Facial expression recognition technology can obtain multiple expression values ​​for a face, each representing the state of a specific facial feature. For example, the eyebrows, eyes, and mouth can each have corresponding expression values. The eyes can have various expression values, such as their open / closed state and geometric shape. Taking the open / closed state of the eyes as an example, the expression value for a fully open eye can be set to 1, and the expression value for a closed eye can be set to 0. For simplicity, this article typically uses values ​​of 0 or 1 when illustrating expression values. However, it should be understood that the expression value corresponding to each facial feature can also have other values ​​greater than 0 and less than 1. For example, the expression value corresponding to the eyes represents the degree to which the eyes are open; the larger the value (i.e., the closer to 1), the closer the eyes are to being fully open. To obtain accurate expression values, facial expression recognition technology often requires the captured face to be clear and complete, meaning the face is not obscured, especially the facial features. However, in real-world scenarios, there is often self-occlusion caused by large head postures (such as looking down or tilting the head), or object occlusion caused by hands, mobile phones, or other objects blocking the face. Obscuring a face reduces the accuracy of facial expression recognition, potentially resulting in incorrect or abnormal expression values. Common solutions to this problem include multimodal combination schemes and multi-camera visual fusion schemes. Multimodal combination schemes combine multiple inputs, such as images and audio, to predict and correct facial expressions. Multi-camera visual fusion schemes use multiple cameras to simultaneously capture facial images and then fuse these images using computer vision algorithms, thereby improving the accuracy of recognizing and predicting occluded areas caused by changes in head posture.

[0022] The inventors noticed that these solutions all led to increased hardware costs, increased algorithm complexity, and increased computation time. To address this problem, the inventors proposed the following method.

[0023] Figure 1 and Figure 2 These are schematic flowcharts illustrating methods for determining facial expressions under face occlusion according to some embodiments of this disclosure. Figure 3 This is a schematic diagram of a first image, a second image, and a first region in a method for determining facial expressions under face occlusion according to some embodiments of the present disclosure. Figure 1 The method shown may include steps S110, S120, S130, and S140 as described below, and steps S150 and S160. The following is in conjunction with... Figure 2 and Figure 3 introduce Figure 1 The method shown.

[0024] In step S110, a video containing a face is received, and it is detected whether a face occlusion exists in a first frame of the video. For example... Figure 3 As shown, a video may include multiple frames, and one frame may be selected as the first frame. It should be understood that this is for illustrative purposes only. Figure 2 and Figure 3 Only the face in the first image is shown; the first image may include other background content in addition to the face. In some embodiments, a face detection algorithm can be used to detect whether a face is occluded in the first image.

[0025] In step S120, in response to the occlusion of a first region of the face in the first image, a facial expression vector is determined based on the first image. The facial expression vector may include a first vector and a second vector. The first vector includes one or more expression values ​​relating to the first region of the face, and the second vector includes one or more expression values ​​relating to regions of the face other than the first region. Each expression value represents the state of a part of the face. For example, as... Figure 3 As shown, the first region is the left eye area covered by the palm. Correspondingly, the first vector may include expression values ​​related to the left eye. Expression values ​​related to the left eye may include, for example, the open / closed state of the left eye and whether there are wrinkles at the outer corner of the left eye. If the expression value corresponding to an open left eye is set to 1, and the expression value corresponding to a closed left eye is set to 0; the expression value corresponding to no wrinkles at the outer corner of the left eye is set to 1, and the expression value corresponding to wrinkles at the outer corner of the left eye is set to 0, and the first vector is [1, 1], then the first vector means that the left eye is open and there are no wrinkles at the outer corner of the left eye. The second vector may include, for example, expression values ​​relating to areas of the face other than the left eye, such as expression values ​​related to the left eyebrow, the right eyebrow, the right eye, and the lips. In some embodiments, a facial expression recognition algorithm can be used to recognize the first image to obtain the facial expression vector. It should be understood that although the first region is occluded, the facial expression recognition algorithm can determine the facial expression vector of the entire face, including the first region. That is, it can determine the first vector and the second vector. However, due to the influence of occlusion, the accuracy of the first vector may not be high enough.

[0026] In some embodiments, such as Figure 2As shown, determining facial expression vectors based on a first image includes: reconstructing the face based on the first image to obtain a 3D face model, and determining facial expression vectors based on the 3D face model. Face reconstruction can obtain a 3D face model of the entire face, including a first region. The facial expression vectors determined from this model can include both a first vector corresponding to the occluded first region and a second vector corresponding to the regions of the face other than the first region. Since face reconstruction can recover missing information, the facial expression vectors determined from the 3D face model obtained from face reconstruction will more accurately reflect the expression region information, thereby further improving the accuracy of expression prediction. As some implementation methods, to fully consider the influence of head pose, facial expression, and other factors on the reconstruction results, deep learning models can be used for face reconstruction. Furthermore, a large amount of data can be used to train deep learning models, which is beneficial for improving the accuracy and stability of face reconstruction.

[0027] In step S130, based on facial symmetry, the first vector is adjusted according to the second vector to obtain a third vector. Facial expression symmetry refers to the similarity of features in facial expressions when they are symmetrical. Therefore, the expression of the occluded first region can be inferred from the expression of the unoccluded region. In some embodiments, adjusting the first vector according to the second vector includes: determining whether there are one or more symmetrical facial features in the region of the face other than the first region that are symmetrical about the facial midline with respect to one or more facial features in the first region; and in response to the existence of one or more symmetrical facial features, selecting one or more expression values ​​corresponding to the one or more symmetrical facial features from the second vector to form a seventh vector; and adjusting the first vector based on the seventh vector. For example, as... Figure 2 As shown, the first region is the left eye area, and there is a right eye area that is symmetrical to the left eye area about the midline of the face. In this case, one or more expression values ​​corresponding to the right eye area can be selected from the second vector to form the seventh vector. For example, the second vector includes expression values ​​related to the left eyebrow, expression values ​​related to the right eyebrow, and expression values ​​related to the right eye; the expression values ​​related to the right eye in the second vector can be selected to form the seventh vector. It should be understood that although there are some cases of asymmetry in left and right expressions in reality, the probability of these cases is very low. For example, in most cases, a person's two eyes open and close synchronously, and only in a few cases do the two eyes not open and close synchronously. The purpose of this disclosure is to obtain the most reasonable expression values ​​so that expression recognition does not cause confusion and errors; therefore, these cases with low probability of occurrence can be excluded. The following sections will describe how to adjust the first vector based on the seventh vector with some examples.

[0028] In step S140, the third vector is adjusted based on the linkage between facial features to obtain the fourth vector. Different facial features typically exhibit linkages; for example, wrinkles at the corners of the eyes often accompany a smile, thus the expression value of the left eye can be predicted based on the left corner of the mouth, right corner of the mouth, and / or the nose. Utilizing the linkages between different expressions helps improve the accuracy and robustness of expression prediction. In some embodiments, adjusting the third vector includes: determining an expression transfer matrix based on the linkages between facial features; determining an eighth vector based on the expression transfer matrix and a second vector; and calculating the weighted sum of the eighth vector and the third vector to obtain the fourth vector. For example, the expression transfer matrix can be pre-determined based on the linkage between the left corner of the mouth and the left eye, and then the eighth vector can be determined based on the expression value related to the left corner of the mouth in the second vector and the expression transfer matrix. By establishing an expression transfer matrix, the linkages between different features can be modeled, thereby improving the accuracy and continuity of expression prediction.

[0029] In step S150, as Figure 3 As shown, one or more fifth vectors are determined based on one or more second images preceding the first image in the video. The fourth vector is then adjusted based on these fifth vectors to obtain a sixth vector. Here, the first region of the face in one or more second images is not occluded, and each of the one or more fifth vectors includes one or more expression values ​​relating to the first region of the face in the corresponding frame of the second image. During video input, the time interval between each frame is very short, while expression changes are usually continuous. Therefore, adjacent frames are interconnected, and the expression of the occluded portion in the current frame can be predicted based on the expressions in previous frames. For ease of description, step S150 will be referred to below as adjusting the fourth vector based on the inter-frame continuity of the video to obtain the sixth vector.

[0030] like Figure 3 As shown, the video includes multiple frames. One or more frames on the video's timeline that meet the condition that the first region is not occluded can be selected as the second image. Based on each second image, a corresponding fifth vector can be determined. It should be understood that... Figure 3For illustrative purposes, the first region in the second image is not shown in a hand shape, but rather in a box shape, indicating the occluded first region. In the case of only one frame of the second image, there is only one fifth vector. Adjusting the fourth vector based on this single fifth vector involves, for example, calculating a weighted sum of the fourth and fifth vectors to obtain the sixth vector. In the case of multiple frames of the second image, there are multiple fifth vectors. Adjusting the fourth vector based on these multiple fifth vectors involves, for example, calculating a weighted sum of the fourth vector and the multiple fifth vectors to obtain the sixth vector. The following sections will describe how to calculate the weighted sum of the fourth vector and the multiple fifth vectors using some examples.

[0031] In step S160, the 3D facial model established based on the first image is adjusted based on the sixth vector. In some embodiments, the 3D facial model consists of multiple facial feature points, and expression values ​​can affect the position of these feature points. The position of the facial feature points in the first region can be adjusted based on the expression values ​​in the sixth vector. In some embodiments, the 3D facial model is established based on the first image during step S120; in this case, it is not necessary to establish the 3D facial model again. In other embodiments, the 3D facial model is not established during step S120; in this case, a facial reconstruction algorithm can be used to establish the 3D facial model.

[0032] In the above embodiments, firstly, facial expression vectors are determined based on a first image with facial occlusion, enabling the prediction of expression values ​​involving the occluded first region, providing a foundation for subsequent adjustment of expression values. Secondly, expression values ​​are adjusted based on facial symmetry, as expression values ​​determined based on facial symmetry are often highly reliable, resulting in more reasonable and credible adjusted expression values. Then, expression values ​​are adjusted based on the linkage relationship between facial parts, further improving the accuracy of expression values. Finally, adjustments are made based on video continuity, better capturing the changing trends of expressions, thereby improving the accuracy and continuity of expression prediction. In summary, this disclosure improves the accuracy and stability of expression prediction for the occluded first region through the collaboration and complementarity of multiple steps. Furthermore, this invention relies only on a single ordinary camera for acquisition, requiring no additional hardware sensors, resulting in low computational load. It can perform expression prediction for the occluded first region in real-time (e.g., single calculation time can be less than 30ms) at low cost, providing relatively reasonable expression values, thereby improving the accuracy and real-time performance of the 3D facial model. The resulting 3D facial model can be used in low-cost, high-real-time, and high-accuracy scenarios such as virtual avatar live streaming.

[0033] The following examples illustrate how to adjust the first vector based on the seventh vector.

[0034] In some embodiments, adjusting the first vector based on the seventh vector includes: determining the occlusion type of the face based on the first image; and, in response to the occlusion type being self-occlusion, calculating a weighted sum of the seventh vector and the first vector to obtain the adjusted first vector. Occlusion types include self-occlusion and object occlusion, such as... Figure 3 In the case shown, where the left eye is covered by a hand, the left eye portion of the first occluded area is invisible. Therefore, the reliability of the first vector before adjustment (i.e., the first vector obtained in step S120) is low. When the occlusion type is object occlusion, the corresponding expression value in the first vector can be replaced with the expression value of the seventh vector to obtain the adjusted first vector (i.e., the third vector). Self-occlusion is, for example, when the face rotates too much, causing part of the face to be invisible. For instance, if the face rotates a large angle to the left, the left eye portion becomes invisible. Since the eye portion of the first occluded area is not completely invisible in the case of self-occlusion, the first vector before adjustment has a certain degree of reliability. A weighted sum of the first vector and the seventh vector can be calculated to obtain the adjusted first vector, thereby improving the reliability of expression prediction. In some embodiments, calculating the weighted sum of the seventh vector and the first vector includes: calculating the weighted sum of each expression value in the seventh vector and the corresponding expression value in the first vector, and updating the corresponding expression value in the first vector with the weighted sum. For example, if the seventh vector is [1, 1] and the first vector is [0, 1], and the weights of both the seventh and first vectors are 0.5, the updated first vector can be [0.5, 1]. In the above embodiment, determining the adjustment method of the first vector based on the occlusion type is beneficial to improving the reliability and accuracy of expression prediction.

[0035] As one implementation method, calculating the weighted sum of the seventh vector and the first vector includes: determining the degree of self-occlusion based on the first image; and determining the weights of the seventh vector and the first vector respectively based on the degree of self-occlusion, with a higher weight for the seventh vector as the degree of self-occlusion increases. For example, a smaller angle of facial rotation indicates a lower degree of self-occlusion, while a larger angle indicates a higher degree of self-occlusion. A higher degree of self-occlusion results in lower reliability of the unadjusted first vector, thus allowing for a higher weight for the seventh vector to improve the accuracy of expression prediction. For example, with a low degree of self-occlusion, the weights of the first and seventh vectors can be set to 0.5 and 0.5 respectively; with a high degree of self-occlusion, the weights can be set to 0.3 and 0.7 respectively. Adjusting the weight of the seventh vector according to the degree of self-occlusion helps improve the accuracy of expression prediction results.

[0036] In some embodiments, before calculating the weighted sum of the seventh vector and the first vector, it is determined whether the difference between the seventh vector and the first vector falls within a preset range; in response to the difference falling within the preset range, the weighted sum of the seventh vector and the first vector is calculated. As some implementations, the difference between the seventh vector and the first vector can be determined based on the sum of the differences between each expression value in the seventh vector and the corresponding expression value in the first vector. For example, if the seventh vector is [0, 1, 1] and the first vector is [0, 0, 1], the difference between the seventh vector and the first vector is (0-0) + (1-0) + (1-1) = 1. Before adjusting the first vector based on the seventh vector, it is pre-determined whether the seventh vector and the first vector fall within a preset range. If they fall within the preset range, it indicates that the seventh vector has high reliability. In this case, adjusting the first vector based on the seventh vector helps ensure the reliability of the adjusted first vector. In some embodiments, if the difference does not fall within the preset range, the first vector may not be adjusted using the seventh vector; that is, the third vector is equal to the first vector.

[0037] The following examples illustrate how to calculate the weighted sum of the fourth vector and multiple fifth vectors.

[0038] In some embodiments, a plurality of corresponding fifth vectors can be determined based on multiple frames of the second image, and a fourth vector can be adjusted based on the plurality of fifth vectors. Adjusting the fourth vector based on the plurality of fifth vectors includes calculating a weighted sum of the plurality of fifth vectors and the fourth vector to obtain a sixth vector. As some implementations, a weighted sum of each expression value in the fourth vector and the corresponding expression value in each of the plurality of fifth vectors can be calculated, and the sixth vector is obtained based on the weighted sum. For example, as... Figure 3 As shown, three corresponding fifth vectors can be determined based on three second images. If the three fifth vectors from front to back are [0, 1, 1], [0, 1, 1], [1, 1, 1], and the fourth vector is [1, 1, 1], and the weights of each fifth and fourth vector are the same (i.e., the first, second, third, and fourth weights are all 0.25), then the sixth vector can be calculated as [0.5, 1, 1]. In some implementations, the weight of the fifth vector determined based on the second image that is closer to the first image is higher. For example, as... Figure 3As shown, the first, second, and third weights can be set to 0.1, 0.2, and 0.3, respectively. Since facial expression changes are continuous, the closer the expression in the second image is to the first image, the stronger the correlation between the expression in the second image and the expression in the first image. Setting its weight higher helps improve the stability and continuity of expression prediction. As one implementation method, it can be determined whether the difference between each of the multiple fifth vectors and the fourth vector falls within a preset range. If the difference between one of the multiple fifth vectors and the fourth vector does not fall within the preset range, the weight of that fifth vector is set to zero. A difference between the fifth vector and the fourth vector not falling within the preset range indicates that the fifth vector has low reliability and is likely erroneous data. In this case, setting its weight to 0 helps ensure the accuracy of expression prediction.

[0039] In some embodiments, before determining the facial expression vector, it is determined whether the area proportion of the first region in the face meets the preset conditions; in response to the area proportion meeting the preset conditions, the facial expression vector is determined. As some implementations, the preset conditions include an area proportion less than or equal to 40%. An excessively large area proportion of the first region will significantly reduce the reliability of the facial expression vector. Determining the facial expression vector when the area proportion of the first region meets the preset conditions helps ensure the accuracy of facial expression prediction.

[0040] In some embodiments, a model can be trained, and the trained model can be used to perform... Figure 1 and Figure 2 The methods shown can be used to improve model performance. One approach is to leverage facial symmetry for data augmentation. For example, face images can be transformed, such as by horizontal flipping, to generate various training samples. Training the model with these samples improves robustness and generalization ability. Another approach is to utilize the interrelationships between facial features to augment expression data, further enhancing robustness and generalization. Finally, models such as Long Short-Term Memory (LSTM) networks can be used to model continuous video frame sequences, improving the accuracy and continuity of predictions.

[0041] The following example illustrates this. Figure 1 and Figure 2 The method shown.

[0042] like Figure 2As shown, firstly, facial occlusion is detected, revealing that the left eye is obscured. Next, full-face 3D reconstruction yields a complete 3D facial model, accurately capturing facial geometry and pose information, including the geometry and open / closed state of the left eye. Based on this information, a rough prediction of expression values ​​related to the left eye can be made, thus correcting and completing missing expression information. Then, leveraging facial symmetry, expression values ​​related to the left eye are corrected using expression values ​​related to the right eye and right eyebrow. Subsequently, based on the interrelationships between facial features, such as expression values ​​related to the left corner of the mouth or nose, the expression values ​​related to the left eye are further corrected. Finally, utilizing the inter-frame continuity of the video input, the left eye expression is modeled within a continuous video frame sequence, better capturing the trend of expression changes. Through these completions and corrections, a more accurate and stable prediction result for the left eye expression can be obtained.

[0043] Figure 4 and Figure 5 These are methods for determining facial expressions under face occlusion according to some embodiments of this disclosure, since... Figure 4 and Figure 5 The illustrated embodiments and Figure 1 and Figure 2 The embodiments shown are quite similar; the following will only focus on... Figure 4 and Figure 5 The illustrated embodiments and Figure 1 and Figure 2 The differences between the embodiments shown will be described, while the similarities can be found in the preceding description.

[0044] like Figure 4 As shown, this disclosure provides a method for determining facial expressions under face occlusion, including the following steps S410, S420, S430, S440, S450 and S460.

[0045] In step S410, as Figure 5 As shown, a video containing a face is received, and the system detects whether a face is obscured in a first frame of the video.

[0046] In step S420, in response to the occlusion of a first region of the face in the first image, a first facial expression vector is determined based on the first image. Here, the first facial expression vector includes a first sub-vector and a second sub-vector. The first sub-vector includes one or more expression values ​​relating to the first region of the face, and the second sub-vector includes one or more expression values ​​relating to regions of the face other than the first region. Each expression value is used to represent the state of a part of the face.

[0047] In step S430, based on facial symmetry, the first sub-vector is adjusted according to the second sub-vector, and the second facial expression vector is determined based on the adjusted first and second sub-vectors. The second facial expression vector can be formed by concatenating the adjusted first and second sub-vectors.

[0048] In step S440, one or more third sub-vectors are determined based on one or more second images preceding the first image in the video. The second facial expression vector is then adjusted based on these one or more third sub-vectors to determine the third facial expression vector. Here, the first region of the face in the one or more second images is not occluded, and each of the one or more third sub-vectors includes one or more expression values ​​relating to the first region of the face in the corresponding frame of the one or more second images.

[0049] In step S450, based on the linkage relationship between facial features, the third facial expression vector is adjusted to determine the fourth facial expression vector.

[0050] In step S460, the three-dimensional face model established based on the first image is adjusted based on the fourth facial expression vector.

[0051] In the above embodiments, firstly, facial expression vectors are determined based on a first image with facial occlusion, enabling the prediction of expression values ​​involving the occluded first region, providing a foundation for subsequent adjustment of expression values. Secondly, expression values ​​are adjusted based on facial symmetry, as expression values ​​determined based on facial symmetry are often highly reliable, resulting in more reasonable and credible adjusted expression values. Then, adjustments are made based on video continuity, which better captures the changing trends of expressions, thereby improving the accuracy and continuity of expression prediction. Finally, adjustments are made based on the linkage relationships between facial parts, further enhancing the accuracy of expression values. In summary, the above embodiments are beneficial for improving the accuracy, stability, and continuity of expression prediction in occluded regions. Furthermore, this invention relies only on a single ordinary camera for acquisition, requiring no additional hardware sensors, resulting in low computational load. It can perform expression prediction of the occluded first region in real time at low cost, thereby improving the accuracy and real-time performance of the 3D facial model. The resulting 3D facial model can be used in low-cost, high-real-time, and high-accuracy scenarios such as virtual avatar live streaming.

[0052] In some embodiments, adjusting the first subvector according to the second subvector includes: determining whether there are one or more symmetrical face parts in the region other than the first region that are symmetrical about the midline of the face to one or more face parts in the first region; and in response to the existence of one or more symmetrical face parts, selecting one or more expression values ​​corresponding to one or more symmetrical face parts from the second subvector to form a fourth subvector; and adjusting the first subvector based on the fourth subvector.

[0053] In some embodiments, adjusting the first subvector based on the fourth subvector includes: detecting the occlusion type of the face, including self-occlusion and object occlusion; and in response to the occlusion type being self-occlusion, calculating a weighted sum of the fourth subvector and the first subvector to obtain the adjusted first subvector. As some implementations, calculating the weighted sum of the fourth subvector and the first subvector includes: calculating a weighted sum of each expression value in the fourth subvector and the corresponding expression value in the first subvector, and updating the corresponding expression value in the first subvector with the weighted sum.

[0054] In some embodiments, calculating the weighted sum of the fourth sub-vector and the first sub-vector includes: determining the degree of self-occlusion based on the first image; and determining the weights of the fourth sub-vector and the first sub-vector based on the degree of self-occlusion, wherein the higher the degree of self-occlusion, the higher the weight of the fourth sub-vector.

[0055] In some embodiments, the second facial expression vector includes an adjusted first sub-vector and a second sub-vector, the second image includes multiple frames, and the third sub-vector includes multiple sub-vectors; adjusting the second facial expression vector includes: determining the third facial expression vector based on a weighted sum of multiple third sub-vectors and the adjusted first sub-vector.

[0056] In some embodiments, the weight of a third sub-vector determined based on a second image that is closer to the first image is higher.

[0057] In some embodiments, calculating a weighted sum of a plurality of third sub-vectors and an adjusted first sub-vector includes: calculating a weighted sum of the expression values ​​in the adjusted first sub-vector and the corresponding expression values ​​in each of the plurality of third sub-vectors, and determining a third facial expression vector based on the weighted sum.

[0058] In some embodiments, determining the first facial expression vector includes: reconstructing a face based on a first image to obtain a three-dimensional facial model, and determining the first facial expression vector based on the three-dimensional facial model.

[0059] In some embodiments, before determining the first facial expression vector, it is determined whether the area ratio of the first region in the face meets a preset condition; and in response to meeting the preset condition, the first facial expression vector is determined.

[0060] In some embodiments, the preset condition includes the first region accounting for less than or equal to 40% of the area of ​​the face.

[0061] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus embodiments, since they largely correspond to the method embodiments, the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0062] Figure 6 This is a structural schematic diagram of a facial expression determination device under facial occlusion according to some embodiments of the present disclosure.

[0063] like Figure 6 As shown, the facial expression determination device 600 under facial occlusion includes a memory 610 and a processor 620 coupled to the memory 610. The processor 620 is configured to execute the method of any of the foregoing embodiments based on instructions stored in the memory 610.

[0064] The memory 610 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory may store, for example, an operating system, application programs, a boot loader, and other programs.

[0065] The facial expression determination device 600 under facial occlusion may also include an input / output interface 630, a network interface 640, and a storage interface 650. These interfaces 630, 640, and 650, as well as the memory 610 and processor 620, can be connected, for example, via a bus 660. The input / output interface 630 provides a connection interface for input / output devices such as a monitor, mouse, keyboard, and touchscreen. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0066] This disclosure also provides a computer-readable storage medium including computer program instructions that, when executed by a processor, implement the method of any of the above embodiments.

[0067] This disclosure also provides a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the method described in any of the above embodiments.

[0068] The embodiments of this disclosure have now been described in detail. To avoid obscuring the concept of this disclosure, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

[0069] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0070] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that the functions specified in one or more flowchart illustrations and / or one or more blocks in a block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate functions for implementing the functions in the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0071] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0072] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0073] In addition, embodiments of this disclosure may also include the following examples:

[0074] 1. A method for determining facial expressions when a face is obscured, comprising:

[0075] Receive a video containing a face, and detect whether a face is obscured in a first frame of the video;

[0076] In response to the occlusion of a first region of a face in the first image, a facial expression vector is determined based on the first image. The facial expression vector includes a first vector and a second vector. The first vector includes one or more expression values ​​relating to the first region of the face, and the second vector includes one or more expression values ​​relating to regions of the face other than the first region. Each expression value is used to represent the state of a part of the face.

[0077] Based on the symmetry of human face, the first vector is adjusted according to the second vector to obtain the third vector;

[0078] Based on the interaction between facial features, the third vector is adjusted to obtain the fourth vector;

[0079] Based on one or more frames of second images preceding the first image in the video, one or more corresponding fifth vectors are determined. The first region of a face in the one or more frames of second images is not occluded. Each of the one or more fifth vectors includes one or more expression values ​​relating to the first region of the face in the corresponding frame of the one or more frames of second images. The fourth vector is adjusted based on the one or more fifth vectors to obtain a sixth vector; and

[0080] The 3D face model built based on the first image is adjusted based on the sixth vector.

[0081] 2. The method according to 1, wherein adjusting the first vector according to the second vector includes:

[0082] Determine whether there exist one or more symmetrical facial features in regions other than the first region that are symmetrical about the facial midline to one or more facial features in the first region; and

[0083] In response to the presence of one or more symmetrical facial features, one or more expression values ​​corresponding to the one or more symmetrical facial features are selected from the second vector to form a seventh vector; and

[0084] The first vector is adjusted based on the seventh vector.

[0085] 3. According to the method described in 2, adjusting the first vector based on the seventh vector includes:

[0086] Based on the first image, the occlusion type of the face is determined, including self-occlusion and object occlusion; and

[0087] In response to the occlusion type being self-occlusion, the weighted sum of the seventh vector and the first vector is calculated to obtain the adjusted first vector.

[0088] 4. The method according to 3, wherein calculating the weighted sum includes:

[0089] Determine the degree of self-occlusion based on the first image; and

[0090] The weights of the seventh vector and the first vector are determined based on the degree of self-occlusion. The higher the degree of self-occlusion, the higher the weight of the seventh vector.

[0091] 5. The method according to 3, wherein calculating the weighted sum includes:

[0092] Calculate the weighted sum of each expression value in the seventh vector and the corresponding expression value in the first vector, and update the corresponding expression value in the first vector with the weighted sum.

[0093] 6. The method according to 3 further includes:

[0094] Before calculating the weighted sum, it is determined whether the difference between the seventh vector and the first vector falls within a preset range;

[0095] In response to the difference falling within a preset range, the weighted sum is calculated.

[0096] 7. According to the method described in 1, wherein,

[0097] Determine multiple corresponding fifth vectors based on multiple frames of second images; and

[0098] Adjusting the fourth vector based on multiple fifth vectors includes: calculating a weighted sum of the multiple fifth vectors and the fourth vector to obtain the sixth vector.

[0099] 8. According to the method described in 7, wherein,

[0100] The weight of the fifth vector is higher based on the second image, which is closer to the first image.

[0101] 9. The method according to 7, wherein calculating the weighted sum includes:

[0102] The sixth vector is obtained by calculating the weighted sum of each expression value in the fourth vector and the corresponding expression value in each of the plurality of fifth vectors.

[0103] 10. The method according to 7, wherein calculating the weighted sum includes:

[0104] Determine whether the difference between each of the multiple fifth vectors and the fourth vector falls within a preset range;

[0105] In response to the fact that the difference between one of the fifth vectors and the fourth vector does not fall within a preset range, the weight of the fifth vector is set to zero.

[0106] 11. The method according to 1, wherein adjusting the third vector includes:

[0107] The expression transition matrix is ​​determined based on the interaction between facial features;

[0108] The eighth vector is determined based on the expression transfer matrix and the second vector. The weighted sum of the eighth vector and the third vector is calculated to obtain the fourth vector.

[0109] 12. According to the method described in 1, determining the facial expression vector includes:

[0110] Face reconstruction is performed based on the first image to obtain the three-dimensional face model, and facial expression vectors are determined based on the three-dimensional face model.

[0111] 13. The method according to 1 further includes:

[0112] Before determining the facial expression vector, it is determined whether the area ratio of the first region in the face meets a preset condition; and

[0113] In response to the area ratio meeting preset conditions, the facial expression vector is determined.

[0114] 14. According to the method described in 13, wherein,

[0115] The preset condition includes that the area ratio is less than or equal to 40%.

[0116] 15. A method for determining facial expressions when a face is occluded, comprising:

[0117] Receive a video containing a face, and detect whether a face is obscured in a first frame of the video;

[0118] In response to the occlusion of a first region of a face in the first image, a first facial expression vector is determined based on the first image. The first facial expression vector includes a first sub-vector and a second sub-vector. The first sub-vector includes one or more expression values ​​relating to the first region of the face, and the second sub-vector includes one or more expression values ​​relating to regions of the face other than the first region. Each expression value is used to represent the state of a part of the face.

[0119] Based on facial symmetry, the first sub-vector is adjusted according to the second sub-vector, and the second facial expression vector is determined based on the adjusted first sub-vector and the second sub-vector.

[0120] Based on one or more frames of second images in the video that are located before the first image, one or more corresponding third sub-vectors are determined. The first region of the face in the one or more frames of second images is not occluded. Each of the one or more third sub-vectors includes one or more expression values ​​related to the first region of the face in the corresponding frame of the one or more frames of second images. The second facial expression vector is adjusted based on the one or more third sub-vectors to determine the third facial expression vector.

[0121] Based on the interaction between facial features, the third facial expression vector is adjusted to determine the fourth facial expression vector; and

[0122] The facial 3D model built based on the first image is adjusted based on the fourth facial expression vector.

[0123] 16. The method according to 1, wherein adjusting the first sub-vector according to the second sub-vector includes:

[0124] Determine whether there exist one or more symmetrical facial features in regions other than the first region that are symmetrical about the facial midline to one or more facial features in the first region; and

[0125] In response to the presence of one or more symmetrical facial features, one or more expression values ​​corresponding to the one or more symmetrical facial features are selected from the second sub-vector to form a fourth sub-vector;

[0126] The first sub-vector is adjusted based on the fourth sub-vector.

[0127] 17. The method according to 16, wherein adjusting the first sub-vector based on the fourth sub-vector includes:

[0128] Detect the type of occlusion on a face, including self-occlusion and object occlusion; and

[0129] In response to the occlusion type being self-occlusion, the weighted sum of the fourth sub-vector and the first sub-vector is calculated to obtain the adjusted first sub-vector.

[0130] 18. The method according to 17, wherein calculating the weighted sum includes:

[0131] Determine the degree of self-occlusion based on the first image; and

[0132] The weights of the fourth sub-vector and the first sub-vector are determined based on the degree of self-occlusion. The higher the degree of self-occlusion, the higher the weight of the fourth sub-vector.

[0133] 19. The method according to 17, wherein calculating the weighted sum includes: calculating a weighted sum of each expression value in the fourth sub-vector and the corresponding expression value in the first sub-vector, and updating the corresponding expression value in the first sub-vector with the weighted sum.

[0134] 20. According to the method described in 15, wherein the second facial expression vector includes an adjusted first sub-vector and a second sub-vector, the second image includes multiple frames, and the third sub-vector includes multiple sub-vectors;

[0135] Adjusting the second facial expression vector includes determining the third facial expression vector based on a weighted sum of multiple third sub-vectors and the adjusted first sub-vector.

[0136] 21. According to the method described in 20, wherein,

[0137] The weight of the third sub-vector is higher based on the second image, which is closer to the first image.

[0138] 22. The method according to 20, wherein calculating the weighted sum includes:

[0139] Calculate the weighted sum of the expression values ​​in the adjusted first sub-vector and the corresponding expression values ​​in each of the multiple third sub-vectors, and determine the third facial expression vector based on the weighted sum.

[0140] 23. According to the method described in 15, determining the first facial expression vector includes:

[0141] Face reconstruction is performed based on the first image to obtain the three-dimensional face model, and the first facial expression vector is determined based on the three-dimensional face model.

[0142] 24. The method according to 15 further includes:

[0143] Before determining the first facial expression vector, it is determined whether the area proportion of the first region in the face meets a preset condition; and

[0144] In response to meeting preset conditions, the first facial expression vector is determined.

[0145] 25. According to the method described in 24, wherein,

[0146] The preset conditions include that the area of ​​the first region in the face is less than or equal to 40%.

[0147] 26. A facial expression determination device under face occlusion, comprising:

[0148] Memory; and

[0149] A processor coupled to the memory is configured to execute any one of the methods described in 1-25 based on instructions stored in the memory.

[0150] 27. A computer-readable storage medium comprising computer program instructions, wherein the computer program instructions, when executed by a processor, implement the method described in any one of 1-25.

[0151] 28. A computer program product comprising a computer program, wherein the computer program, when executed by a processor, implements the method described in any one of 1-25.

[0152] While specific embodiments of this disclosure have been described in detail by way of examples, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for determining facial expressions when a face is occluded, comprising: Receive a video containing a face, and detect whether a face is obscured in a first frame of the video; In response to the occlusion of a first region of a face in the first image, a facial expression vector is determined based on the first image. The facial expression vector includes a first vector and a second vector. The first vector includes one or more expression values ​​relating to the first region of the face, and the second vector includes one or more expression values ​​relating to regions of the face other than the first region. Each expression value is used to represent the state of a part of the face. Based on the symmetry of human face, the first vector is adjusted according to the second vector to obtain the third vector; Based on the interaction between facial features, the third vector is adjusted to obtain the fourth vector; One or more fifth vectors are determined based on one or more second images located before the first image in the video. The first region of the face in the one or more second images is not occluded. Each of the one or more fifth vectors includes one or more expression values ​​related to the first region of the face in the corresponding frame of the one or more second images. The fourth vector is adjusted based on the one or more fifth vectors to obtain the sixth vector. as well as The 3D face model built from the first image is adjusted based on the sixth vector. The adjustment of the third vector includes: The expression transition matrix is ​​determined based on the interaction between facial features. The eighth vector is determined based on the expression transfer matrix and the second vector. The weighted sum of the eighth vector and the third vector is calculated to obtain the fourth vector.

2. The method according to claim 1, wherein, Adjusting the first vector based on the second vector includes: Determine whether there exist one or more symmetrical facial features in regions other than the first region that are symmetrical about the facial midline to one or more facial features in the first region; and In response to the presence of one or more symmetrical facial features, one or more expression values ​​corresponding to the one or more symmetrical facial features are selected from the second vector to form a seventh vector; and The first vector is adjusted based on the seventh vector.

3. The method according to claim 2, wherein, Adjusting the first vector based on the seventh vector includes: Based on the first image, the occlusion type of the face is determined, including self-occlusion and object occlusion; and In response to the occlusion type being self-occlusion, the weighted sum of the seventh vector and the first vector is calculated to obtain the adjusted first vector.

4. The method according to claim 3, wherein, Calculating the weighted sum includes: Determine the degree of self-occlusion based on the first image; and The weights of the seventh vector and the first vector are determined based on the degree of self-occlusion. The higher the degree of self-occlusion, the higher the weight of the seventh vector.

5. The method according to claim 3, wherein, Calculating the weighted sum includes: Calculate the weighted sum of each expression value in the seventh vector and the corresponding expression value in the first vector, and update the corresponding expression value in the first vector with the weighted sum.

6. The method according to claim 3, further comprising: Before calculating the weighted sum, it is determined whether the difference between the seventh vector and the first vector falls within a preset range; In response to the difference falling within a preset range, the weighted sum is calculated.

7. The method according to claim 1, wherein, Multiple corresponding fifth vectors are determined based on multiple frames of the second image; as well as Adjusting the fourth vector based on multiple fifth vectors includes: calculating a weighted sum of the multiple fifth vectors and the fourth vector to obtain the sixth vector.

8. The method according to claim 7, wherein, The weight of the fifth vector is higher based on the second image, which is closer to the first image.

9. The method according to claim 7, wherein, Calculating the weighted sum includes: The sixth vector is obtained by calculating the weighted sum of each expression value in the fourth vector and the corresponding expression value in each of the plurality of fifth vectors.

10. The method according to claim 7, wherein, Calculating the weighted sum includes: Determine whether the difference between each of the multiple fifth vectors and the fourth vector falls within a preset range; In response to the fact that the difference between one of the fifth vectors and the fourth vector does not fall within a preset range, the weight of the fifth vector is set to zero.

11. The method according to claim 1, wherein, Determining facial expression vectors includes: Face reconstruction is performed based on the first image to obtain the three-dimensional face model, and facial expression vectors are determined based on the three-dimensional face model.

12. The method according to claim 1, further comprising: Before determining the facial expression vector, it is determined whether the area ratio of the first region in the face meets a preset condition; as well as In response to the area ratio meeting preset conditions, the facial expression vector is determined.

13. The method according to claim 12, wherein, The preset condition includes that the area ratio is less than or equal to 40%.

14. A method for determining facial expressions when a face is occluded, comprising: Receive a video containing a face, and detect whether a face is obscured in a first frame of the video; In response to the occlusion of a first region of a face in the first image, a first facial expression vector is determined based on the first image. The first facial expression vector includes a first sub-vector and a second sub-vector. The first sub-vector includes one or more expression values ​​relating to the first region of the face, and the second sub-vector includes one or more expression values ​​relating to regions of the face other than the first region. Each expression value is used to represent the state of a part of the face. Based on facial symmetry, the first sub-vector is adjusted according to the second sub-vector, and the second facial expression vector is determined based on the adjusted first sub-vector and the second sub-vector. Based on one or more frames of second images in the video that are located before the first image, one or more corresponding third sub-vectors are determined. The first region of the face in the one or more frames of second images is not occluded. Each of the one or more third sub-vectors includes one or more expression values ​​related to the first region of the face in the corresponding frame of the one or more frames of second images. The second facial expression vector is adjusted based on the one or more third sub-vectors to determine the third facial expression vector. Based on the interaction between facial features, the third facial expression vector is adjusted to determine the fourth facial expression vector. as well as The facial 3D model built from the first image is adjusted based on the fourth facial expression vector. The adjustment of the third facial expression vector includes: The expression transition matrix is ​​determined based on the interaction between facial features. The eighth vector is determined based on the expression transition matrix and the second sub-vector. The weighted sum of the eighth vector and the third facial expression vector is calculated to determine the fourth facial expression vector.

15. The method according to claim 14, wherein, Adjusting the first subvector based on the second subvector includes: Determine whether there exist one or more symmetrical facial features in regions other than the first region that are symmetrical about the facial midline to one or more facial features in the first region; and In response to the presence of one or more symmetrical facial features, one or more expression values ​​corresponding to the one or more symmetrical facial features are selected from the second sub-vector to form a fourth sub-vector; The first sub-vector is adjusted based on the fourth sub-vector.

16. The method according to claim 15, wherein, Adjusting the first subvector based on the fourth subvector includes: Detect the type of occlusion on a face, including self-occlusion and object occlusion; and In response to the occlusion type being self-occlusion, the weighted sum of the fourth sub-vector and the first sub-vector is calculated to obtain the adjusted first sub-vector.

17. The method according to claim 16, wherein, Calculating the weighted sum includes: Determine the degree of self-occlusion based on the first image; and The weights of the fourth sub-vector and the first sub-vector are determined based on the degree of self-occlusion. The higher the degree of self-occlusion, the higher the weight of the fourth sub-vector.

18. The method according to claim 16, wherein, Calculating the weighted sum includes: calculating the weighted sum of each expression value in the fourth sub-vector and the corresponding expression value in the first sub-vector, and updating the corresponding expression value in the first sub-vector with the weighted sum.

19. The method of claim 14, wherein, The second facial expression vector includes the adjusted first sub-vector and the second sub-vector, the second image includes multiple frames, and the third sub-vector includes multiple sub-vectors; Adjusting the second facial expression vector includes determining the third facial expression vector based on a weighted sum of multiple third sub-vectors and the adjusted first sub-vector.

20. The method according to claim 19, wherein, The weight of the third sub-vector is higher based on the second image, which is closer to the first image.

21. The method according to claim 19, wherein, Calculating the weighted sum includes: Calculate the weighted sum of the expression values ​​in the adjusted first sub-vector and the corresponding expression values ​​in each of the multiple third sub-vectors, and determine the third facial expression vector based on the weighted sum.

22. The method according to claim 14, wherein, Determining the first facial expression vector includes: Face reconstruction is performed based on the first image to obtain the three-dimensional face model, and the first facial expression vector is determined based on the three-dimensional face model.

23. The method of claim 14, further comprising: Before determining the first facial expression vector, it is determined whether the area ratio of the first region in the face meets the preset conditions. as well as In response to meeting preset conditions, the first facial expression vector is determined.

24. The method according to claim 23, wherein, The preset conditions include that the area of ​​the first region in the face is less than or equal to 40%.

25. A facial expression determination device under face occlusion, comprising: Memory; as well as A processor coupled to the memory is configured to perform the method of any one of claims 1-24 based on instructions stored in the memory.

26. A computer-readable storage medium comprising computer program instructions, wherein, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-24.

27. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method described in any one of claims 1-24.

Images