A method, system, device and medium for 3D facial landmark detection

Abstract

This invention belongs to the field of facial landmark detection technology, and discloses a 3D facial landmark detection method, system, device, and medium. The method includes: acquiring an original facial image video, and extracting two adjacent video frames from the original facial image video, which are respectively used as a reference video frame and a current video frame; using a pre-configured 3D landmark extraction model, extracting a current landmark and a reference landmark from the reference video frame and the current video frame respectively; calculating the difference between the current landmark and the reference landmark to obtain a feature point motion deviation value; and constructing a loss function based on temporal motion constraints based on the feature point motion deviation value. This invention extracts 3D landmarks from the reference frame and the current frame respectively using a 3D landmark extraction model, and then calculates a loss value between the landmarks of the two frames based on temporal motion constraints as part of the loss function, thereby optimizing the model parameters.

Description

Technical Field

[0001] This invention relates to the field of facial landmark detection technology, and in particular to a 3D facial landmark detection method, system, device and medium. Background Technology

[0002] Facial landmark detection is one of the core technologies for face analysis in the field of computer vision. Essentially, it accurately locates key anatomical positions of a face (such as eyes, nose, mouth, and contours) from an image or video stream, and outputs the two-dimensional (x,y) or three-dimensional (x,y,z) coordinates of these landmarks. This technology serves as a bridge between "face detection" (determining the presence of a face) and "advanced face analysis" (such as expression recognition and 3D reconstruction), playing an irreplaceable role in numerous fields. Its main applications include: 1) Human-computer interaction: Tracking users' facial movements through key points to achieve "contactless" or "natural interaction"; 2) Identity authentication: Utilizing the "biological uniqueness" of key points (different people have different facial contours, eye distances, nose shapes, and other key point distributions) to enhance the security of identity verification; 3) Healthcare: Analyzing physiological states or assisting in disease diagnosis through subtle changes in key points; 4) Entertainment and content creation: Providing "real face-driven" virtual content generation to enhance the immersion and authenticity of the content.

[0003] Deep learning-based facial landmark detection methods typically extract global and local features using convolutional neural networks (CNNs), then constrain the features using regression or heatmap methods to output the final facial landmark coordinates.

[0004] The regression method inputs the face image into the CNN. The model automatically extracts global and local features through the convolutional layer and finally outputs the (x,y) coordinates of all key points directly through the fully connected layer (e.g., 68 key points output a 136-dimensional vector: 68×2).

[0005] Heatmap-based facial landmark detection is a mainstream high-precision detection solution in the deep learning era. Its core logic is not to directly regress the (x,y) coordinates of the landmarks, but rather to predict the "probability distribution heatmap" of each landmark in the image through a model, and then locate the probability peaks from the heatmap to indirectly obtain the landmark coordinates. This method, with its strong ability to capture local details and high robustness to pose and expression changes, has become the preferred technology for complex scenes (such as large poses, slight occlusion, and dynamic videos), and is widely used in high-precision scenarios such as virtual makeup try-on, expression transfer, and 3D face reconstruction.

[0006] However, in the physical world, faces are 3D. In scenarios where the input is a face image or video, current facial landmark detection techniques typically extract 2D landmarks and struggle to extract 3D landmarks. Existing 3D facial landmark extraction methods usually require 3D point cloud face data as input, and then extract 3D landmarks using a 3D convolutional network. Therefore, how to extract 3D facial landmarks from 2D images is currently a major problem in the practical application of 3D facial landmark detection.

[0007] Furthermore, existing facial landmark detection methods, whether regression-based, heatmap-based, or a fusion of both, all perform landmark detection on a single face image. However, in many video applications, such as video calls, video conferencing, and live streaming, the facial landmarks of the same person should also conform to specific motion relationships over time, such as head turning and translation. Therefore, leveraging the motion characteristics of faces to enhance the detection of facial landmarks in temporal videos can effectively improve the accuracy of facial landmark detection in video scenarios. Summary of the Invention

[0008] This invention provides a 3D facial landmark detection method, system, device, and medium to solve the aforementioned technical problems existing in the prior art.

[0009] According to a first aspect of the present invention, a 3D facial landmark detection method is provided.

[0010] Among them, 3D facial landmark detection methods include: The original face image video is acquired, and two adjacent video frames are obtained from the original face image video, which are respectively used as the reference video frame and the current video frame; Using a pre-configured 3D key point extraction model, current key points and reference key points are extracted from the reference video frame and the current video frame, respectively. Calculate the difference between the current key point and the reference key point to obtain the feature point motion deviation value; and based on the feature point motion deviation value, construct a loss function based on temporal motion constraints. With the goal of minimizing the loss function, a self-supervised training model for facial landmark detection is conducted, and 3D facial landmark detection is performed based on the trained model.

[0011] The process of extracting key points from the current frame and key points from the reference frame using a pre-configured 3D key point extraction model includes: Convolution and pooling processes are performed on the input reference video frame and the current video frame respectively to reduce the image resolution and perform channel expansion to obtain global semantic features while retaining intermediate feature maps. The global semantic features are transformed into 3D features through 3D matrix recombination, and the 3D features are then subjected to multiple 3D convolution processes to obtain 3D convolution features. Perform 3D deconvolution or upsampling on the 3D convolutional features to keep the depth of the 3D features unchanged, obtain local detail features, and retain the current feature map; The current feature map is element-wise added to an intermediate feature map of the same resolution to obtain the fused feature. The number of channels of the fused features is adjusted by convolution processing, and a multi-channel heatmap is generated by probability normalization. The obtained multi-channel heatmap is multiplied with the 3D coordinate system to obtain the 3D feature key points.

[0012] The formula for calculating the motion deviation value of the feature point is as follows: ； In the formula, The value represents the deviation between the keypoints in the reference frame and the feature points in the current frame; x represents the keypoint in the current frame; R represents the rotational motion, which is 3. A rotation matrix of 3; Indicates the key points of the reference frame; t represents the translational motion, which is 3. A translation matrix of 1.

[0013] The formula for the loss function is as follows: ； In the formula, This represents the final loss value calculated by the loss function; The loss represents the temporal motion consistency constraint loss; λ represents the weight parameter. represents the commonly used keypoint coordinate error loss, used to adjust the weight relationship between the two loss terms; x represents the keypoints predicted by the keypoint detection model in the current frame; R represents rotational motion, which is 3. A rotation matrix of 3; This represents the reference frame keypoints predicted by the keypoint detection model; t represents the translational motion, which is 3. Translation matrix of 1; This represents the truth value of the key point in the current frame.

[0014] According to a second aspect of the present invention, a 3D facial landmark detection system is provided.

[0015] The 3D facial landmark detection system includes: The raw data acquisition module is used to acquire raw face image videos and to acquire two adjacent video frames from the raw face image videos, which are respectively used as reference video frames and current video frames; The 3D key point extraction module is used to extract current key points and reference key points from the reference video frame and the current video frame respectively using a pre-configured 3D key point extraction model. The loss function construction module calculates the difference between the current key point and the reference key point to obtain the feature point motion deviation value; and constructs a loss function based on the feature point motion deviation value based on temporal motion constraints. The facial landmark detection module aims to minimize the loss function, trains a facial landmark detection model under self-supervision, and performs 3D facial landmark detection based on the trained facial landmark detection model.

[0016] Specifically, when the 3D keypoint extraction module extracts the current keypoint and the reference keypoint from the reference video frame and the current video frame respectively using a pre-configured 3D keypoint extraction model, Convolution and pooling processes are performed on the input reference video frame and the current video frame respectively to reduce the image resolution and perform channel expansion to obtain global semantic features while retaining intermediate feature maps. The global semantic features are transformed into 3D features through 3D matrix recombination, and the 3D features are then subjected to multiple 3D convolution processes to obtain 3D convolution features. Perform 3D deconvolution or upsampling on the 3D convolutional features to keep the depth of the 3D features unchanged, obtain local detail features, and retain the current feature map; The current feature map is element-wise added to an intermediate feature map of the same resolution to obtain the fused feature. The number of channels of the fused features is adjusted by convolution processing, and a multi-channel heatmap is generated by probability normalization. The obtained multi-channel heatmap is multiplied with the 3D coordinate system to obtain the 3D feature key points.

[0017] The formula for calculating the motion deviation value of the feature point is as follows: ； In the formula, The value represents the deviation between the keypoints in the reference frame and the feature points in the current frame; x represents the keypoint in the current frame; R represents the rotational motion, which is 3. A rotation matrix of 3; Indicates the key points of the reference frame; t represents the translational motion, which is 3. A translation matrix of 1.

[0018] The formula for the loss function is as follows: ； In the formula, This represents the final loss value calculated by the loss function; The loss represents the temporal motion consistency constraint loss; λ represents the weight parameter. represents the commonly used keypoint coordinate error loss, used to adjust the weight relationship between the two loss terms; x represents the keypoints predicted by the keypoint detection model in the current frame; R represents rotational motion, which is 3. A rotation matrix of 3; This represents the reference frame keypoints predicted by the keypoint detection model; t represents the translational motion, which is 3. Translation matrix of 1; This represents the truth value of the key point in the current frame.

[0019] According to a third aspect of the present invention, a computer device is provided.

[0020] In some embodiments, the computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the steps of the method described above.

[0021] According to a fourth aspect of the present invention, a computer-readable storage medium is provided.

[0022] In one embodiment, a computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the steps of the above method.

[0023] The technical solution provided by this invention may include the following beneficial effects: This invention extracts 3D key points from a reference frame and the current frame using a 3D key point extraction model. Then, it calculates the loss value between the key points of the two frames based on temporal motion constraints as part of the loss function, thereby optimizing the model parameters and effectively improving the accuracy of face key point detection in video scenes.

[0024] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0026] Figure 1 This is a flowchart illustrating the steps of a 3D face key point detection method according to an exemplary embodiment; Figure 2 This is a schematic diagram of a 3D face key point detection system according to an exemplary embodiment; Figure 3 This is a schematic diagram of the structure of a computer device according to an exemplary embodiment. Detailed Implementation

[0027] The following description and accompanying drawings fully illustrate specific embodiments described herein to enable those skilled in the art to practice them. Some embodiments may include or substitute parts and features of other embodiments. The scope of the embodiments herein encompasses the entire scope of the claims and all available equivalents thereof. Throughout this document, the terms “first,” “second,” etc., are used only to distinguish one element from another without requiring or implying any actual relationship or order between the elements. Indeed, a first element can also be referred to as a second element, and vice versa. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a structure, apparatus, or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a structure, apparatus, or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the structure, apparatus, or device that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments; similar or identical parts between embodiments can be referred to interchangeably.

[0028] The terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" used in this document to indicate orientations or positional relationships are based on the orientations or positional relationships shown in the accompanying drawings. They are used solely for the convenience of describing the document and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. In the description herein, unless otherwise specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two elements; they can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0029] In this document, unless otherwise stated, the term "multiple" means two or more.

[0030] In this article, the character " / " indicates that the objects before and after it are in an "or" relationship. For example, A / B means: A or B.

[0031] In this article, the term "and / or" describes an association between objects, indicating that three relationships can exist. For example, A and / or B means: A or B, or A and B.

[0032] It should be understood that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order constraint on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the diagram may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0033] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0034] Figure 1 An embodiment of a 3D face key point detection method of the present invention is shown.

[0035] In this optional embodiment, the 3D facial landmark detection method includes: Step S101: Obtain the original face image video, and obtain two adjacent video frames from the original face image video, and use them as the reference video frame and the current video frame, respectively. Step S102: Using a pre-configured 3D key point extraction model, extract the current key point and the reference key point from the reference video frame and the current video frame, respectively. Step S103: Calculate the difference between the current key point and the reference key point to obtain the feature point motion deviation value; and construct a loss function based on the feature point motion deviation value based on temporal motion constraints. Step S104: With the goal of minimizing the loss function, a self-supervised training model for facial landmark detection is performed, and 3D facial landmark detection is performed based on the trained model.

[0036] Figure 2 An embodiment of a 3D face landmark detection system according to the present invention is shown.

[0037] In this optional embodiment, the distributed security crowdsourcing evaluation system includes: The raw data acquisition module 201 is used to acquire raw face image video, and to acquire two adjacent video frames from the raw face image video, and use them as a reference video frame and the current video frame, respectively. The 3D key point extraction module 202 is used to extract current key points and reference key points from the reference video frame and the current video frame respectively using a pre-configured 3D key point extraction model. The loss function construction module 203 calculates the difference between the current key point and the reference key point to obtain the feature point motion deviation value; and constructs a loss function based on the feature point motion deviation value based on temporal motion constraints. The facial landmark detection module 204 aims to minimize the loss function, trains a facial landmark detection model under self-supervision, and performs 3D facial landmark detection based on the trained facial landmark detection model.

[0038] In this optional embodiment, when extracting current key points and reference key points from the reference video frame and the current video frame respectively using a pre-configured 3D key point extraction model, convolution and pooling processes are performed on the input reference video frame and the current video frame respectively to reduce the image resolution and perform channel expansion to obtain global semantic features, while retaining intermediate feature maps; the global semantic features are transformed into 3D features through 3D matrix recombination, and the 3D features are subjected to multiple 3D convolution processes to obtain 3D convolution features; the 3D convolution features are subjected to 3D deconvolution or upsampling processes to maintain the depth of the 3D features and obtain local detail features, while retaining the current feature map; the current feature map is element-wise added to the intermediate feature map of the same resolution to obtain fused features; the number of channels of the fused features is adjusted through convolution processing, and a multi-channel heatmap is generated through probability normalization; the obtained multi-channel heatmap is multiplied by the 3D coordinate system to obtain 3D feature key points.

[0039] Specifically, this invention proposes a channel asymmetric hourglass network for input images based on hourglass networks, adding a 3D transformation stage to the ordinary hourglass network's downsampling and upsampling stages.

[0040] In the downsampling stage of this invention, the proposed channel asymmetric hourglass network, like ordinary hourglass networks, gradually downsamples the input image. It progressively reduces the feature map resolution (e.g., 256×256→128×128→64×64→32×32) through convolutional layers (e.g., 3×3 convolution) and pooling layers (e.g., 2×2 max pooling) while increasing the number of channels (e.g., 64→128→256→512) to extract global semantic features (e.g., facial contours, organ layout). After each downsampling stage, an "intermediate feature map" is retained for feature fusion during subsequent upsampling.

[0041] In a typical hourglass network, the downsampled features are adjusted through one or more convolutional layers before entering the upsampling stage.

[0042] In our proposed channel-asymmetric hourglass network, the downsampled features undergo feature recombination, transforming from 2D features to 3D features. The downsampled 2D feature is (c, h, w), where c represents the number of feature channels, and h and w represent the height and width of the feature. The 2D feature is recombinated into a 3D feature (c / d, d, h, w) through matrix dimensional transformation. The core of matrix recombination is to split the original channel dimension c into two dimensions (c / d, d), where c / d is the number of channels in the 3D feature, and d is the newly added feature depth. For example, the 2D feature (512, 16, 15) can be recombinated into the 3D feature (32, 16, 16, 15), where c / d represents the new number of channels. d It is the depth of 3D features. h and w These are the height and width of the 3D feature, respectively. After being transformed into a 3D feature, one or more 3D convolutional layers will be used to optimize and adjust the feature. The whole process is the 3D transformation stage.

[0043] During the upsampling stage, similar to a typical hourglass network, the feature map resolution is gradually restored (e.g., 16×16×16→16×32×32→16×64×64→16×128×128→16×256×256) through 3D deconvolution or upsampling (such as bilinear interpolation), while simultaneously reducing the number of channels. Note that the depth of the 3D features remains constant throughout. After each upsampling stage, the current feature map is... "Intermediate feature maps of the same resolution" retained during the downsampling stage "Element-wise addition is performed, where c, d, h, and w represent the number of feature channels, depth, height, and width, respectively, and i and j represent the number of layers in the neural network. Element-wise addition means adding the feature values ​​at corresponding positions between two features of the same size to obtain a new feature, i.e...." Figure 3 As shown by the black arrow in the diagram, this network structure design integrates global semantic features (from downsampling) and local detail features (from upsampling) to avoid loss of details during the upsampling process.

[0044] In the final output stage, the number of channels is adjusted to the "number of key points" through 1×1×1 convolution, and then a multi-channel heatmap is generated through probability normalization, which can use the Sigmoid function.

[0045] Normalized heatmap middle, The z-axis corresponds to the 3D coordinate system. The x-axis corresponding to the 3D coordinate system The normalized heatmap corresponding to the y-axis in a 3D coordinate system. Multiplying by the 3D coordinate system yields the key points (z, x, y) corresponding to the 3D feature.

[0046] In the above optional embodiments, when the key points extracted by the 3D face key point extraction model are accurate, for two sets of key points of the same person within a short period of time, the above-mentioned rotational and translational motion relationship should be satisfied, therefore the deviation value... The value will approach zero; therefore, when training the 3D face key point extraction model, two images of the same person taken at similar times are input each time. The rotational motion R and translational motion t are fitted to the two sets of key points, and then the feature point motion deviation value is calculated. The formula for calculating the feature point motion deviation value is as follows: ； In the formula, The value represents the deviation between the keypoints in the reference frame and the feature points in the current frame; x represents the keypoint in the current frame; R represents the rotational motion, which is 3. A rotation matrix of 3; Indicates the key points of the reference frame; t represents the translational motion, which is 3. A translation matrix of 1.

[0047] In the above optional embodiments, two temporally close images of the same person are taken as the reference frame and the current frame, respectively. The 3D key point extraction model extracts 3D key points from the reference frame and the current frame. The key point x in the current frame and the key point x in the reference frame are compared. The following functional relationship should be satisfied: ； In the formula, x represents the key point of the current frame; Represents key points of the reference frame; T represents 3D matrix transformation and reconstruction; R represents rotational motion, which is 3. A rotation matrix of 3; t represents translational motion, which is 3. Translation matrix of 1; This represents the deviation between the feature points after motion transformation and the feature points in the current frame.

[0048] In the above optional embodiments, the formula for the loss function is: ； In the formula, This represents the final loss value calculated by the loss function; The loss represents the temporal motion consistency constraint loss; λ represents the weight parameter. represents the commonly used keypoint coordinate error loss, used to adjust the weight relationship between the two loss terms; x represents the keypoints predicted by the keypoint detection model in the current frame; R represents rotational motion, which is 3. A rotation matrix of 3; This represents the reference frame keypoints predicted by the keypoint detection model; t represents the translational motion, which is 3. Translation matrix of 1; This represents the truth value of the key point in the current frame.

[0049] Figure 3 An embodiment of a computer device according to the present invention is shown. The computer device may be a server, and includes a processor, memory, and a network interface connected via a system bus. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores static and dynamic information data. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the above-described method embodiment.

[0050] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0051] In addition, the present invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0052] In addition, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0053] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0054] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.

[0055] This invention is not limited to the structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this invention is limited only by the appended claims.

Claims

1. A 3D facial landmark detection method, characterized in that, include: The original face image video is acquired, and two adjacent video frames are obtained from the original face image video, which are respectively used as the reference video frame and the current video frame; Using a pre-configured 3D keypoint extraction model, current keypoints and reference keypoints are extracted from the reference video frame and the current video frame, respectively. The extraction process includes: performing convolution and pooling operations on the input reference and current video frames to reduce image resolution and expand channels, obtaining global semantic features while retaining intermediate feature maps; transforming the global semantic features into 3D features through 3D matrix reconstruction, and performing multiple 3D convolution operations on the 3D features to obtain 3D convolutional features; performing 3D deconvolution or upsampling operations on the 3D convolutional features while maintaining the depth of the 3D features to obtain local detail features while retaining the current feature map; element-wise summing the current feature map with an intermediate feature map of the same resolution to obtain fused features; adjusting the number of channels in the fused features through convolution processing, and generating a multi-channel heatmap through probability normalization; multiplying the obtained multi-channel heatmap with a 3D coordinate system to obtain 3D feature keypoints. Calculate the difference between the current key point and the reference key point to obtain the feature point motion deviation value; and based on the feature point motion deviation value, construct a loss function based on temporal motion constraints. With the goal of minimizing the loss function, a self-supervised training model for facial landmark detection is conducted, and 3D facial landmark detection is performed based on the trained model.

2. The 3D face key point detection method according to claim 1, characterized in that, The formula for calculating the motion deviation value of the feature point is: ； In the formula, This represents the deviation between the key points in the reference frame and the feature points in the current frame; x represents the key point in the current frame; R represents the rotation motion, which is a 3×3 rotation matrix. t represents the key point of the reference frame; t represents the translation motion, which is a 3×1 translation matrix.

3. The 3D face key point detection method according to claim 1, characterized in that, The formula for the loss function is: ； In the formula, This represents the final loss value calculated by the loss function; This represents the temporal motion consistency constraint loss; λ represents the weighting parameter; The loss term represents the commonly used keypoint coordinate error loss, used to adjust the weight relationship between the two loss terms; x represents the keypoints predicted by the keypoint detection model in the current frame; R represents rotational motion, which is a 3×3 rotation matrix. This represents the reference frame keypoints predicted by the keypoint detection model; t represents the translational motion, which is a 3×1 translation matrix. This represents the truth value of the key point in the current frame.

4. A 3D facial landmark detection system, characterized in that, include: The raw data acquisition module is used to acquire raw face image videos and to acquire two adjacent video frames from the raw face image videos, which are respectively used as reference video frames and current video frames; The 3D keypoint extraction module is used to extract current keypoints and reference keypoints from the reference video frame and the current video frame respectively using a pre-configured 3D keypoint extraction model. The extraction of current keypoints and reference keypoints from the reference video frame and the current video frame using the pre-configured 3D keypoint extraction model includes: performing convolution and pooling processing on the input reference video frame and the current video frame respectively, reducing the image resolution and performing channel expansion to obtain global semantic features, while retaining intermediate feature maps; and then processing the global semantic features... The 3D matrix is ​​reconstructed into 3D features, and these 3D features are then subjected to multiple 3D convolutions to obtain 3D convolutional features. These 3D convolutional features are then subjected to 3D deconvolution or upsampling while maintaining their depth to obtain local detail features, and the current feature map is retained. The current feature map is then element-wise added to an intermediate feature map of the same resolution to obtain fused features. The number of channels in the fused features is adjusted through convolution, and a multi-channel heatmap is generated through probability normalization. Finally, the multi-channel heatmap is multiplied by a 3D coordinate system to obtain the 3D feature keypoints. The loss function construction module calculates the difference between the current key point and the reference key point to obtain the feature point motion deviation value; and constructs a loss function based on the feature point motion deviation value based on temporal motion constraints. The facial landmark detection module aims to minimize the loss function, trains a facial landmark detection model under self-supervision, and performs 3D facial landmark detection based on the trained facial landmark detection model.

5. The 3D face key point detection system according to claim 4, characterized in that, The formula for calculating the motion deviation value of the feature point is: ； In the formula, This represents the deviation between the key points in the reference frame and the feature points in the current frame; x represents the key point in the current frame; R represents the rotation motion, which is a 3×3 rotation matrix. t represents the key point of the reference frame; t represents the translation motion, which is a 3×1 translation matrix.

6. The 3D face key point detection system according to claim 4, characterized in that, The formula for the loss function is: ； In the formula, This represents the final loss value calculated by the loss function; This represents the temporal motion consistency constraint loss; λ represents the weighting parameter; The loss term represents the commonly used keypoint coordinate error loss, used to adjust the weight relationship between the two loss terms; x represents the keypoints predicted by the keypoint detection model in the current frame; R represents rotational motion, which is a 3×3 rotation matrix. This represents the reference frame keypoints predicted by the keypoint detection model; t represents the translational motion, which is a 3×1 translation matrix. This represents the truth value of the key point in the current frame.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images