Real-time face animation method and device based on video camera

By performing triangle segmentation and analysis of key muscle points on video frames, the problem of insufficient accuracy in facial expression recognition in dynamic videos is solved, and real-time and accurate expression recognition effects are achieved.

CN119091011APending Publication Date: 2024-12-06SHENZHEN GAOGU INNOVATION TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411098778.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The prior art lacks accuracy and robustness in identifying changes in facial expressions in dynamic videos, making it difficult to achieve real-time and accurate expression recognition.

Method used

By triangulating the video frames, the key muscle points on the face are determined, and the expression change information is analyzed based on the number and position change information of the key muscle points in adjacent frames to realize animation display.

Benefits of technology

It breaks through the limitations of traditional static image recognition, realizes real-time and accurate recognition of facial expression changes in dynamic videos, and improves the accuracy and robustness of recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091011A_ABST
    Figure CN119091011A_ABST
Patent Text Reader

Abstract

The invention relates to a real-time face animation method and device based on a video camera, and the method comprises the steps: carrying out the triangular subdivision of each face image in each target frame video, and determining the face key muscle points of each face in the target frame video; determining expression change information of each face in the face image according to the quantity change information of the face key muscle points of the same face and the position change information of the same face key muscle points in the adjacent multi-frame target frame video; and displaying the face in the video in the two-dimensional display picture in an animation manner according to the expression change information. By performing triangular subdivision on the face image in each target frame, the positions of the key muscle points of the face are accurately determined. By analyzing the number change information and the position change information of the face key muscle points of the same face in the multiple adjacent target frames, the expression change information of the face can be accurately recognized. And real-time and accurate recognition of the facial expression change in the dynamic video is realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of recognition technology, and in particular to a real-time face animation method and device based on a video camera. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technology, face recognition technology has been widely used in various fields, such as security monitoring, human-computer interaction, virtual reality, etc. However, most of the existing face recognition technologies focus on face detection and recognition, and there are still certain challenges in the accurate recognition and processing of facial expression changes.

[0003] Traditional face recognition technology mainly relies on facial features in static images for matching and recognition, but there is relatively little recognition of facial expression changes in dynamic videos. Although some technologies try to recognize expressions through image changes in video frame sequences, the accuracy and robustness of these technologies still need to be improved due to the complexity of video frames and the diversity of expression changes. Summary of the invention

[0004] The purpose of the present invention is to provide a real-time facial animation method and device based on a video camera, aiming to solve the technical problem that it is difficult to accurately perform facial animation recognition and display due to the complexity of video frames and the diversity of expression changes in related scenes.

[0005] In order to achieve the above-mentioned purpose, a first aspect of the embodiments of the present disclosure provides a real-time face animation method based on a video camera, the method comprising:

[0006] In response to acquiring a video captured by a camera, extracting a target frame video containing a face image in the video;

[0007] Perform triangulation on each face image in each target frame video to determine the key facial muscle points of each face in the target frame video;

[0008] Determine expression change information of each face in the face image according to the quantity change information of the facial key muscle points of the same face and the position change information of the same facial key muscle points in the adjacent multiple frames of the target frame video;

[0009] The human face in the video is animatedly displayed in a two-dimensional display screen according to the expression change information.

[0010] In a possible implementation, triangulating each face image in each target frame video to determine the key facial muscle points of each face in the target frame video includes:

[0011] Performing contour recognition on each of the target frame videos to identify facial organs in each of the facial images, and determining pixel types of pixels corresponding to each of the facial images in each of the target frame videos according to the contour recognition results, wherein the pixel types include facial organ pixel types and non-facial organ pixel types;

[0012] Based on the principle that the connecting lines between two pixel points corresponding to each face image in each target frame video do not cross, any two pixel points corresponding to each face image in each target frame video are connected until no new connecting line between the two pixel points can be added in the face image;

[0013] Determine the triangle type of each minimum triangle formed by three pixels in the face image according to the pixel point types of the three pixels in the minimum triangle;

[0014] According to the triangle types of all the smallest triangles to which each pixel point in the facial image belongs, it is determined whether each pixel point is a key facial muscle point.

[0015] In a possible implementation, determining the triangle type of the minimum triangle according to the pixel point types of three pixels in each minimum triangle formed by three pixels in the face image includes:

[0016] In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are all the facial organ pixel point types, determining the triangle type of the minimum triangle to be the organ triangle type;

[0017] In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are all the non-facial organ pixel point types, determining the triangle type of the minimum triangle to be a facial triangle type;

[0018] In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are not a single pixel point type, it is determined that the triangle type of the minimum triangle is a mixed triangle type.

[0019] In a possible implementation, determining whether each pixel point in the face image is a key facial muscle point according to the triangle type of all the smallest triangles to which each pixel point belongs includes:

[0020] When the triangle type of all the smallest triangles to which the pixel point belongs in the face image is not a single triangle type, determining that the pixel point is a key facial muscle point;

[0021] When the triangle types of all the smallest triangles to which the pixel point belongs in the face image are the facial triangle types, determining that the pixel point is not the facial key muscle point;

[0022] When the triangle types of all the smallest triangles to which the pixel point in the facial image belongs are the organ triangle type or the mixed triangle type, whether the pixel point is the facial key muscle point is determined based on the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point.

[0023] In a possible implementation, determining whether the pixel point is the facial key muscle point according to the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point includes:

[0024] In the case where the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point is less than a preset ratio threshold, determining whether a pixel point on the same shortest connecting line as a target pixel point in the face image whose triangle type is the organ triangle type or the mixed triangle type and to which all the minimum triangles belong is the facial key muscle point;

[0025] When a pixel point on the same shortest connecting line as a target pixel point in the face image whose triangle type is the organ triangle type or the mixed triangle type is a key facial muscle point, the target pixel point is determined to be the key facial muscle point;

[0026] The above steps are traversed and executed. If after the shortest line traversal is completed, the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point in the face image is still less than the preset ratio threshold, the line is switched to the second shortest line, the third shortest line, and so on, until the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point is greater than or equal to the preset ratio threshold.

[0027] In a possible implementation, determining the expression change information of each face in the face image according to the quantity change information of the facial key muscle points of the same face and the position change information of the same facial key muscle points in the adjacent multiple frames of the target frame video includes:

[0028] Performing regional division on the key facial muscle points of the same face in the target frame videos of adjacent frames;

[0029] Compare the number of the key facial muscle points in the same area of ​​the same face in two adjacent target frame videos with a reference model of the key facial muscle points established under a neutral expression to determine first expression change information;

[0030] Tracking the position change information of the same facial key muscle point in the adjacent multiple frames of the target frame video by the optical flow method to determine the second expression change information;

[0031] The expression change information of each of the faces in the face image is determined according to the first expression change information and the second expression change information.

[0032] In a possible implementation, the method of tracking the position change information of the same facial key muscle point in the adjacent multiple frames of the target frame video by the optical flow method to determine the second expression change information includes:

[0033] Using the first frame of the target frame video among the adjacent multiple frames as a base frame, using Taylor series, and using partial derivatives in space and time coordinates to calculate the displacement and direction of the same facial key muscle point;

[0034] The second expression change information is determined according to the displacement and direction of each of the key facial muscle points in the target frame video of adjacent multiple frames.

[0035] A second aspect of the embodiments of the present disclosure provides a real-time face animation device based on a video camera, comprising:

[0036] An extraction module is configured to extract a target frame video containing a face image in response to acquiring a video captured by a camera;

[0037] A first determination module is configured to triangulate each face image in each target frame video to determine the key facial muscle points of each face in the target frame video;

[0038] The second determination module is configured to determine the expression change information of each face in the face image according to the quantity change information of the facial key muscle points of the same face and the position change information of the same facial key muscle points in the adjacent multiple frames of the target frame video;

[0039] The animation display module is configured to display the human face in the video in an animated manner in a two-dimensional display screen according to the expression change information.

[0040] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0041] According to a fourth aspect of the embodiments of the present disclosure, an electronic device is provided, including:

[0042] a memory having a computer program stored thereon;

[0043] A processor is used to execute the computer program in the memory to implement the steps of any one of the methods in the first aspect.

[0044] The present invention provides a real-time face animation method and device based on a video camera. Compared with the prior art, it has the following beneficial effects:

[0045] In response to obtaining the video captured by the camera, the target frame video containing the face image is extracted; each face image in each target frame video is triangulated to determine the key facial muscle points of each face in the target frame video; the expression change information of each face in the face image is determined according to the number change information of the key facial muscle points of the same face in the adjacent multiple frames of the target frame video and the position change information of the same key facial muscle points; the face in the video is animated in a two-dimensional display screen according to the expression change information. In this way, the position of the key facial muscle points is accurately determined by triangulating the face image in each target frame. These key muscle points are important features of facial expression changes. By analyzing the number change information and position change information of the key facial muscle points of the same face in the adjacent multiple frames of the target frame, the expression change information of the face can be accurately identified. The limitation of traditional technology that only recognizes expression based on static images is broken through, and the real-time and accurate recognition of facial expression changes in dynamic videos is realized.

[0046] Other features and advantages of the present disclosure will be described in detail in the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide a further understanding of the present disclosure and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present disclosure but do not constitute a limitation of the present disclosure. In the accompanying drawings:

[0048] Figure 1 The present invention is a flowchart of a real-time face animation method based on a video camera according to an embodiment of the specification.

[0049] Figure 2 It is a block diagram of a real-time face animation device based on a video camera according to an embodiment of the specification. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0052] In order to achieve the above object, the present invention provides a real-time face animation method based on a video camera. Figure 1 The flowchart of a real-time face animation method based on a video camera according to an embodiment is shown. The method comprises:

[0053] In step S11, in response to acquiring a video captured by a camera, extracting a target frame video containing a face image in the video;

[0054] The target frame video is a video clip of specific frames selected from the video, and these frames contain face images.

[0055] In the disclosed embodiment, the presence of a face is detected by analyzing the video frame by frame using image recognition technology. Once a frame with a face is found, it is extracted as a target frame video. For example, in a surveillance video, in order to specifically analyze the moment when a face appears, such a target frame video extraction can be performed so as to perform a detailed analysis of the face later.

[0056] When the camera captures the video stream, the system analyzes each frame in real time to see if it contains a face image. Each frame is scanned using a face detection algorithm (such as Haar Cascade, Dlib's HOG+SVM, or a deep learning model). If a face is detected, the frame is marked as a target frame and extracted from the video stream.

[0057] In step S12, triangulation is performed on each face image in each target frame video to determine the key facial muscle points of each face in the target frame video;

[0058] Among them, triangulation is the process of dividing a face image into multiple triangular regions. Key facial muscle points refer to specific locations that have an important impact on facial expressions.

[0059] In the disclosed embodiments, the face image can be divided into triangles based on a specific algorithm and model, and the vertices of these triangles are usually key muscle points. By determining these points, the features and changes of the face can be described more accurately. For example, in the production of face animation, the key muscle points can be determined by triangulating the face, so that the animation can be driven according to the changes of these points later, making the animation more natural and realistic.

[0060] The face image in each target frame is triangulated using computer vision and graphics algorithms. The key facial muscle points of the face are identified and located, which usually include the positions of key features such as eyebrows, eyes, nose, and mouth. The feature point detection algorithm (such as Dlib's 68-point or 81-point face key point detection) is used to accurately locate the positions of these points.

[0061] In step S13, the expression change information of each face in the face image is determined according to the quantity change information of the facial key muscle points of the same face and the position change information of the same facial key muscle points in the adjacent multiple frames of the target frame video;

[0062] In the disclosed embodiment, the quantity change information refers to the increase or decrease in the number of key muscle points; the position change information refers to the movement of the key muscle points in space.

[0063] In the disclosed embodiment, by comparing the key muscle points of the same person's face in multiple adjacent target frames, the differences in quantity and position are analyzed to infer the trend and degree of expression change. For example, in a video of a person speaking, the expression change process from calm to smiling and then laughing can be determined by observing the changes in the number and position of the key muscle points on the face in adjacent frames.

[0064] For the same face in multiple adjacent target frames, the system compares and analyzes the changes in the number and position of key facial muscle points in these frames. The change in number may occur under certain expressions (such as the increase in the number of muscle points at the brow when frowning), while the change in position directly reflects the movement of muscles and changes in expression. By analyzing these changes, the system can infer the expression change information of the face, such as happiness, anger, surprise, etc.

[0065] In step S14, the face of the person in the video is animatedly displayed in a two-dimensional display screen according to the expression change information.

[0066] The two-dimensional display screen may be an area that displays an image on a plane.

[0067] In the disclosed embodiment, based on the obtained expression change information, the expression change of the face is dynamically presented on the two-dimensional display screen using animation rendering technology, making it look like it is moving. For example, in a video call software, based on the user's actual expression change information, the expression change of the user's face is displayed in an animated form on the other party's screen.

[0068] For example, a graphics rendering engine (such as OpenGL, WebGL, or Unity) is used to create an animation scene in a two-dimensional display screen. The face image (or its key muscle points) extracted from the video is used as the input data of the animation. According to the expression change information of the face, the system dynamically adjusts the expression of the face in the animation scene, such as the curvature of the eyebrows, the degree of opening of the mouth, etc. This animation scene is rendered and displayed in real time, so that the audience can see the animation effect corresponding to the facial expression in the video.

[0069] The method in the above technical solution extracts the target frame video with the face image in the video in response to the acquisition of the video shot by the camera; triangulates each face image in each target frame video to determine the key facial muscle points of each face in the target frame video; determines the expression change information of each face in the face image according to the number change information of the key facial muscle points of the same face in the adjacent multiple target frame videos and the position change information of the same key facial muscle points; and displays the face in the video in an animated manner in a two-dimensional display screen according to the expression change information. In this way, by triangulating the face image in each target frame, the position of the key facial muscle points is accurately determined. These key muscle points are important features of facial expression changes. By analyzing the number change information and position change information of the key facial muscle points of the same face in the adjacent multiple target frames, the facial expression change information can be accurately identified. It breaks through the limitation of traditional technology that only recognizes expression based on static images, and realizes real-time and accurate recognition of facial expression changes in dynamic videos.

[0070] In a possible implementation, in step S12, triangulating each face image in each target frame video to determine the key facial muscle points of each face in the target frame video includes:

[0071] In step S121, contour recognition is performed on each of the target frame videos to identify facial organs in each of the facial images, and according to the contour recognition result, pixel types of pixels corresponding to each of the facial images in each of the target frame videos are determined, wherein the pixel types include facial organ pixel types and non-facial organ pixel types;

[0072] For face images, contour recognition can identify the contours of the face and the contours of facial organs (such as eyes, nose, and mouth). Pixel type: When processing images, pixel type is usually used to describe the type of information represented by the pixel. In this example, the pixel type includes facial organ pixel type and non-facial organ pixel type, which are used to distinguish whether the pixel belongs to a facial organ.

[0073] In the disclosed embodiment, a contour recognition algorithm (such as the Canny edge detection algorithm) is used to process each face image in the target frame video to identify the contour of the face and the contours of each facial organ. According to the result of contour recognition, it is determined whether each pixel belongs to a facial organ. If the pixel is located within or on the contour of a facial organ, it is marked as a facial organ pixel type; otherwise, it is marked as a non-facial organ pixel type.

[0074] In step S122, based on the principle that the connecting lines between two pixel points corresponding to each face image in each target frame video do not cross, any two pixel points corresponding to each face image in each target frame video are connected until no connecting line between two pixel points can be added in the face image;

[0075] In the disclosed embodiment, a triangle is constructed by connecting any two pixel points based on the non-intersection principle. At the beginning, two pixel points are randomly selected for connection, and then new lines are gradually added until no new lines between two pixel points can be added in the face image. In this way, the entire face image is divided into a series of non-overlapping triangles.

[0076] For example, starting from a corner point of the eye, connect it with a point on the nose and a point on the edge of the mouth to form a triangle. Then, based on the principle of non-intersection, continue to add new lines until the entire face is divided into multiple triangles.

[0077] In step S123, according to the pixel point types of three pixels in each minimum triangle formed by three pixels in the face image, the triangle type of the minimum triangle is determined;

[0078] For each minimum triangle formed by three pixels, the pixel types of its three vertices (i.e., three pixels) are checked. If the three vertices of the triangle are all of the facial organ pixel type, the triangle is marked as a facial organ triangle; otherwise, it is marked as a non-facial organ triangle.

[0079] In step S124, it is determined whether each pixel point in the facial image is a key facial muscle point based on the triangle type of all the smallest triangles to which each pixel point belongs.

[0080] Among them, whether the pixel is a facial key muscle point is determined according to the type of all the smallest triangles to which each pixel belongs. Generally, if a pixel belongs to multiple facial organ triangles, or it is located at a key position of a facial organ triangle (such as a boundary point or vertex), then it is considered to be a facial key muscle point.

[0081] For example, the pixels near the corners of the eyes, mouth, and nose tip are considered as key facial muscle points because they are located at key positions of multiple facial organ triangles. These points will be used for subsequent expression recognition or facial animation.

[0082] The above technical solution can use contour recognition technology, which can accurately identify various facial organs in the face image, such as eyes, nose, mouth, etc. Based on the results of contour recognition, the technology can divide the pixels in the face image into facial organ pixel type and non-facial organ pixel type. This distinction helps to more accurately identify the pixels related to the facial organs during triangulation and facial key muscle point identification. By connecting any two pixels on the principle of non-intersection, the technology can divide the face image into a series of non-overlapping triangles. This division method not only ensures the integrity and independence of the triangle, but also improves the accuracy and efficiency of subsequent processing. According to the pixel type of the three pixels in each minimum triangle, the technology can determine the type of the triangle, that is, whether it is a facial organ triangle. Based on the types of all the minimum triangles to which each pixel belongs, the technology can determine which pixels are facial key muscle points. Therefore, through the steps of contour recognition, pixel type distinction, non-intersection triangulation, minimum triangle type determination and facial key muscle point identification, accurate analysis and processing of the face image in the video is achieved.

[0083] In a possible implementation, in step S123, determining the triangle type of the minimum triangle according to the pixel point types of three pixels in each minimum triangle formed by three pixels in the face image includes:

[0084] In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are all the facial organ pixel point types, determining the triangle type of the minimum triangle to be the organ triangle type;

[0085] If all three pixels in a minimum triangle are marked as facial organ pixel types, then this triangle is considered to be directly related to the facial organs, so we classify it as an "organ triangle type". This type of triangle is usually located at the boundaries or inside facial organs such as eyes, nose, and mouth.

[0086] For example, a minimum triangle consisting of three pixels is identified, and these three pixels are located at the upper left corner, upper right corner and a point on the eyelid. Since these three pixels are located inside or on the boundary of the facial organ, the eye, they are marked as facial organ pixel type. Therefore, this triangle is classified as an organ triangle type.

[0087] In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are all the non-facial organ pixel point types, determining the triangle type of the minimum triangle to be a facial triangle type;

[0088] If all three pixels in a smallest triangle are marked as non-face organ pixel types, then the triangle is considered to represent the general area or background of the face and is classified as a "face triangle type". This type of triangle occupies areas other than facial organs in face images and is also important for understanding facial contours and overall structure.

[0089] For example, the smallest triangle formed by three pixels may be located on the side of the face, away from any obvious facial features. These three pixels are all labeled as non-facial feature pixel types because they are located in the general area of ​​the face or in the background. Therefore, this triangle is classified as a facial triangle type.

[0090] In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are not a single pixel point type, it is determined that the triangle type of the minimum triangle is a mixed triangle type.

[0091] If the pixel types of the three pixels in a minimum triangle are not all of the same type, that is, they contain both facial organ pixel types and non-facial organ pixel types, then the triangle is considered a mixed type and is classified as a "mixed triangle type". This type of triangle is usually located at the junction of facial organs and the general area of ​​the face, and is important for identifying facial edges and transition areas.

[0092] For example, at the junction of the eye and cheek, we might identify a smallest triangle that contains two facial organ pixel types (located at the edge of the eye) and one non-facial organ pixel type (located on the cheek). Since it contains different types of pixels, this triangle is classified as a mixed triangle type. This type of triangle is particularly useful in identifying facial edges and transition areas because they provide smooth transition information from facial organs to the general face.

[0093] In a possible implementation, in step S124, determining whether each pixel point in the face image is a key facial muscle point according to the triangle type of all the smallest triangles to which each pixel point belongs includes:

[0094] When the triangle type of all the smallest triangles to which the pixel point belongs in the face image is not a single triangle type, determining that the pixel point is a key facial muscle point;

[0095] If the type of all the smallest triangles to which a pixel belongs is not a single one (i.e., it contains organ triangle type, facial triangle type, and / or mixed triangle type), this usually means that the pixel is located at the edge or transition area of ​​the facial organ. These areas usually have a large deformation when the expression changes, so such a pixel is likely to be determined as a key facial muscle point.

[0096] For example, a pixel near the corner of the eye may belong to both an organ triangle (because it is inside the eye) and a mixed triangle (because it is also at the junction of the eye and the skin). Since this pixel does not belong to a single triangle type, it is likely to be a facial key muscle point because it is located at the edge of a facial organ.

[0097] When the triangle types of all the smallest triangles to which the pixel point belongs in the face image are the facial triangle types, determining that the pixel point is not the facial key muscle point;

[0098] Among them, if all the smallest triangles to which a pixel belongs are of the facial triangle type (i.e., only non-facial organ pixel types), then the pixel is likely to be located in the general area or background of the face and has no direct association with the facial organs. Therefore, such a pixel is usually not determined as a facial key muscle point. Consider a pixel located on the side of the face, away from any facial organ. This pixel may only belong to the facial triangle type because it is located in the general area or background of the face. Therefore, this pixel is not determined as a facial key muscle point.

[0099] When the triangle types of all the smallest triangles to which the pixel point in the facial image belongs are the organ triangle type or the mixed triangle type, whether the pixel point is the facial key muscle point is determined based on the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point.

[0100] Among them, when all the smallest triangles to which a pixel belongs are organ triangle types or mixed triangle types, further judgment is required. Because although these areas are related to facial organs, not all such points are facial key muscle points. At this time, we can determine whether the pixel is a facial key muscle point based on the ratio of facial key muscle points to pixels that are not facial key muscle points (for example, through a preset threshold or a machine learning-based model).

[0101] For example, suppose there is a pixel near the corner of the mouth, which only belongs to the organ triangle type (because it is inside the mouth). However, not all pixels inside the mouth are key facial muscle points. At this time, the ratio of key facial muscle points to pixels that are not key facial muscle points within a certain range around the pixel can be calculated. If the ratio exceeds a preset threshold (for example, indicating that the area deforms significantly when the expression changes), the pixel is determined to be a key facial muscle point. Conversely, if the ratio is lower, it is determined to be a non-key facial muscle point.

[0102] In a possible implementation, determining whether the pixel point is the facial key muscle point according to the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point includes:

[0103] In the case where the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point is less than a preset ratio threshold, determining whether a pixel point on the same shortest connecting line as a target pixel point in the face image whose triangle type is the organ triangle type or the mixed triangle type and to which all the minimum triangles belong is the facial key muscle point;

[0104] In the disclosed embodiment, firstly, other pixels on the same shortest line as the target pixel are found. These pixels may have direct geometric or structural associations with it. It is checked whether there are facial key muscle points among these pixels. If there are, it means that the target pixel is likely to be a facial key muscle point as well, because it is directly associated with the determined facial key muscle points.

[0105] For example, first find other pixels on the same shortest line as the target pixel (e.g., the line from the corner of the mouth to the cheek). Among these pixels, several pixels that have been marked as key facial muscle points (e.g., points at the edge of the corner of the mouth) are found.

[0106] When a pixel point on the same shortest connecting line as a target pixel point in the face image whose triangle type is the organ triangle type or the mixed triangle type is a key facial muscle point, the target pixel point is determined to be the key facial muscle point;

[0107] If after checking the pixels on the shortest line, the ratio of the target pixel to the pixel that is not a facial key muscle point is still less than the preset ratio threshold, it is necessary to further explore other possible associated pixels. At this time, switch the line to the second shortest line (i.e., the line that is the second closest to the target pixel) and repeat the above steps. If the pixels on the second shortest line still cannot make the ratio reach the preset threshold, continue to switch to the shortest line again, and so on.

[0108] For example, since the pixels on the shortest line include key facial muscle points, it is inferred that the target pixel is also likely to be a key facial muscle point. However, for rigor, the ratio of the target pixel to the pixel that is not a key facial muscle point is still calculated, but it is found that this ratio is still less than the preset ratio threshold. At this time, switch to the second shortest line (for example, the line from the corner of the mouth to the nostril) for further analysis. However, no new key facial muscle points are found on the second shortest line.

[0109] The above steps are traversed and executed. If after the shortest line traversal is completed, the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point in the face image is still less than the preset ratio threshold, the line is switched to the second shortest line, the third shortest line, and so on, until the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point is greater than or equal to the preset ratio threshold.

[0110] In the disclosed embodiment, this process will be iterated until a line is found such that the ratio of the target pixel to the pixel that is not a key facial muscle point is greater than or equal to a preset ratio threshold, or all possible lines have been traversed.

[0111] The above technical solution significantly improves the accuracy and robustness of recognition by introducing a strategy based on the shortest link and its extension. Specifically, when the ratio of the target pixel point (i.e., the pixel point whose smallest triangle type is an organ triangle or a mixed triangle) to the non-facial key muscle point is less than the preset threshold, the technology does not make a direct judgment, but further considers the situation of other pixels on the shortest link where the pixel point is located.

[0112] Furthermore, by checking whether there are facial key muscle points among the pixels on the shortest link, the technology can more accurately determine whether the target pixel is a facial key muscle point. Because facial key muscle points are often spatially adjacent to other facial key muscle points or have direct geometric connections, this strategy helps to discover key points that may be missed when analyzed separately.

[0113] Furthermore, when the analysis of the pixels on the shortest link still fails to make the ratio reach the preset threshold, the technology will expand to the second shortest link, the third shortest link, and so on, until a link that meets the conditions is found or all possible links are traversed. This method of gradually expanding the search range not only ensures the accuracy of the algorithm, but also avoids unnecessary computational overhead.

[0114] Finally, when the ratio of facial key muscle points to non-facial key muscle points is greater than or equal to the preset threshold, the algorithm will determine the target pixel as a facial key muscle point. This strategy comprehensively considers the local information and contextual information of the pixel, thereby improving the accuracy of recognition.

[0115] In general, this technology can effectively identify key facial muscle points when processing complex facial images, providing strong support for subsequent applications such as facial expression recognition and sentiment analysis. At the same time, its strategy based on the shortest link and its extension also demonstrates the wisdom of applying contextual information and gradually expanding the search scope in the field of image processing.

[0116] In a possible implementation, in step S13, determining the expression change information of each face in the face image according to the quantity change information of the facial key muscle points of the same face in the adjacent multiple frames of the target frame video and the position change information of the same facial key muscle points includes:

[0117] In step S131, the key facial muscle points of the same face in the adjacent multiple frames of the target frame video are divided into regions;

[0118] Among them, the face is divided into multiple areas, such as eyebrow area, eye area, nose area, mouth area, etc. The purpose of this is to better analyze and understand the number and position changes of key facial muscle points in each area, so as to more accurately identify expression changes.

[0119] For example, suppose there is a facial image, first divide it into four regions: eyebrows, eyes, nose and mouth. Then, count the number of key facial muscle points in each region.

[0120] In step S132, the number of the facial key muscle points in the same area of ​​the same face in two adjacent frames of the target frame video is compared with a reference model of the facial key muscle points established under a neutral expression to determine the first expression change information;

[0121] In the disclosed embodiment, a reference model of facial key muscle points under a neutral expression is pre-established. This model includes the number of facial key muscle points in each area when the expression is neutral. Then, the number of facial key muscle points in the same area of ​​the same face in two adjacent frames is compared with the reference model, and the change in expression is identified by the difference in quantity. For example, if it is detected that the number of facial key muscle points in the eyebrow area increases significantly in two adjacent frames, this may mean that the eyebrows are being raised upward, thereby suggesting an expression of surprise or anger.

[0122] In step S133, the position change information of the same facial key muscle point is tracked in the adjacent multiple frames of the target frame video by an optical flow method to determine the second expression change information;

[0123] Among them, the optical flow method is a method for estimating the movement of pixels or feature points in an image. Here, the optical flow method is used to track the position changes of the same facial key muscle points in adjacent multi-frame target frame videos. Through the trajectory of position changes, the changing trend of facial expressions can be inferred. For example, if the facial key muscle points at the corners of the mouth are tracked by the optical flow method and are moving to the sides, this may mean that the mouth is opening, suggesting a smiling or laughing expression.

[0124] In step S134, expression change information of each of the faces in the face image is determined according to the first expression change information and the second expression change information.

[0125] In the disclosed embodiment, the first expression change information and the second expression change information based on the change in the number and position of the key facial muscle points are obtained respectively. Finally, the two types of information are combined, and the final expression change information is determined using a machine learning model or a preset rule. For example, if it is detected that the number of key facial muscle points in the eyebrow area increases and the position of the key facial muscle points at the corners of the mouth moves to both sides, it can be determined that the face is showing a surprised expression by combining these information.

[0126] In a possible implementation, in step S133, the optical flow method is used to track the position change information of the same facial key muscle point in the adjacent multiple frames of the target frame video to determine the second expression change information, including:

[0127] In step S1331, the first frame of the target frame video of the adjacent multiple frames is used as the base frame, and the Taylor series is used to calculate the displacement and direction of the same facial key muscle point using partial derivatives in space and time coordinates;

[0128] In the optical flow method, in order to estimate the movement of pixels or feature points in an image, the Taylor series is often used to approximate the change of pixel values ​​over time and space. In the first frame of the adjacent multi-frame target frame video (as the base frame), for each key facial muscle point, the Taylor series expansion can be used to express its position change in subsequent frames.

[0129] Specifically, the Taylor series allows the motion of pixels to be estimated by calculating the partial derivatives of the pixel values ​​in space and time. These partial derivatives reflect the rate of change of the pixel value with spatial position and time, which can be used to infer the displacement direction and speed of the pixel.

[0130] For example, in the first frame, a key facial muscle point is identified (such as a point at the corner of the eye). In order to track the position of this point in subsequent frames, the Taylor series is used to estimate its motion. By calculating the partial derivatives of the pixel value of this point in space (x and y directions) and time, the approximate position of this point in the next frame can be predicted.

[0131] In step S1332, the second expression change information is determined based on the displacement and direction of each of the facial key muscle points in the target frame video of multiple adjacent frames.

[0132] In the disclosed embodiment, after tracking the displacement and direction of the same facial key muscle point in the adjacent multi-frame target frame video, the second expression change information can be determined based on this information. Specifically, by analyzing the movement trajectory (displacement and direction) of the facial key muscle points, the changing trend of the facial expression can be inferred. For example, if the muscle points near the corners of the mouth move upward, it may indicate a smiling or happy expression; if the muscle points near the eyebrows move downward, it may indicate an angry or dissatisfied expression.

[0133] For example, continuing the above scenario, assume that the motion trajectory of the key facial muscle point at the corner of the eye has been tracked over several frames. By analyzing this trajectory, it is found that the point gradually moves upward and outward. Based on this observation, it can be inferred that the person's eyes are widening, which is usually associated with expressions of surprise or excitement. Therefore, this motion trajectory is used as part of the second expression change information for subsequent expression recognition and analysis.

[0134] In summary, through the application of optical flow method and Taylor series, we can accurately track the position changes of key facial muscle points, and determine the second expression change information based on this, providing strong data support for facial expression recognition.

[0135] The present disclosure also provides a real-time face animation device based on a video camera, see Figure 2 As shown, including:

[0136] The extraction module 210 is configured to extract a target frame video containing a face image in response to acquiring a video captured by a camera;

[0137] The first determination module 220 is configured to triangulate each face image in each target frame video to determine the key facial muscle points of each face in the target frame video;

[0138] The second determination module 230 is configured to determine the expression change information of each face in the face image according to the quantity change information of the facial key muscle points of the same face and the position change information of the same facial key muscle points in the adjacent multiple frames of the target frame video;

[0139] The animation display module 240 is configured to display the face of the person in the video in an animated manner in a two-dimensional display screen according to the expression change information.

[0140] In a possible implementation, the first determining module 220 is configured to:

[0141] Performing contour recognition on each of the target frame videos to identify facial organs in each of the facial images, and determining pixel types of pixels corresponding to each of the facial images in each of the target frame videos according to the contour recognition results, wherein the pixel types include facial organ pixel types and non-facial organ pixel types;

[0142] Based on the principle that the connecting lines between two pixel points corresponding to each face image in each target frame video do not cross, any two pixel points corresponding to each face image in each target frame video are connected until no new connecting line between the two pixel points can be added in the face image;

[0143] Determine the triangle type of each minimum triangle formed by three pixels in the face image according to the pixel point types of the three pixels in the minimum triangle;

[0144] According to the triangle types of all the smallest triangles to which each pixel point in the facial image belongs, it is determined whether each pixel point is a key facial muscle point.

[0145] In a possible implementation, the first determining module 220 is configured to:

[0146] In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are all the facial organ pixel point types, determining the triangle type of the minimum triangle to be the organ triangle type;

[0147] In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are all the non-facial organ pixel point types, determining the triangle type of the minimum triangle to be a facial triangle type;

[0148] In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are not a single pixel point type, it is determined that the triangle type of the minimum triangle is a mixed triangle type.

[0149] In a possible implementation, the first determining module 220 is configured to:

[0150] When the triangle type of all the smallest triangles to which the pixel point belongs in the face image is not a single triangle type, determining that the pixel point is a key facial muscle point;

[0151] When the triangle types of all the smallest triangles to which the pixel point belongs in the face image are the facial triangle types, determining that the pixel point is not the facial key muscle point;

[0152] When the triangle types of all the smallest triangles to which the pixel point in the facial image belongs are the organ triangle type or the mixed triangle type, whether the pixel point is the facial key muscle point is determined based on the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point.

[0153] In a possible implementation, the first determining module 220 is configured to:

[0154] In the case where the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point is less than a preset ratio threshold, determining whether a pixel point on the same shortest connecting line as a target pixel point in the face image whose triangle type is the organ triangle type or the mixed triangle type and to which all the minimum triangles belong is the facial key muscle point;

[0155] When a pixel point on the same shortest connecting line as a target pixel point in the face image whose triangle type is the organ triangle type or the mixed triangle type is a key facial muscle point, the target pixel point is determined to be the key facial muscle point;

[0156] The above steps are traversed and executed. If after the shortest line traversal is completed, the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point in the face image is still less than the preset ratio threshold, the line is switched to the second shortest line, the third shortest line, and so on, until the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point is greater than or equal to the preset ratio threshold.

[0157] In a possible implementation, the second determining module is configured as 230:

[0158] Performing regional division on the key facial muscle points of the same face in the target frame videos of adjacent frames;

[0159] Compare the number of the key facial muscle points in the same area of ​​the same face in two adjacent target frame videos with a reference model of the key facial muscle points established under a neutral expression to determine first expression change information;

[0160] Tracking the position change information of the same facial key muscle point in the adjacent multiple frames of the target frame video by the optical flow method to determine the second expression change information;

[0161] The expression change information of each of the faces in the face image is determined according to the first expression change information and the second expression change information.

[0162] In a possible implementation, the second determining module is configured as 230:

[0163] Using the first frame of the target frame video among the adjacent multiple frames as a base frame, using Taylor series, and using partial derivatives in space and time coordinates to calculate the displacement and direction of the same facial key muscle point;

[0164] The second expression change information is determined according to the displacement and direction of each of the key facial muscle points in the target frame video of adjacent multiple frames.

[0165] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the method described in any one of the aforementioned embodiments are implemented.

[0166] The present disclosure also provides an electronic device, including:

[0167] a memory having a computer program stored thereon;

[0168] A processor is used to execute the computer program in the memory to implement the steps of the method in any one of the aforementioned embodiments.

[0169] The preferred embodiments of the present disclosure are described in detail above in conjunction with the accompanying drawings; however, the present disclosure is not limited to the specific details in the above embodiments; within the technical concept of the present disclosure, various changes, modifications, substitutions and variations may be made to these embodiments, and these changes, modifications, substitutions and variations all fall within the protection scope of the present disclosure.

[0170] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction, and they should also be regarded as the contents disclosed in this disclosure. In order to avoid unnecessary repetition, this disclosure will not further describe various possible combinations. The technical scope of this application is not limited to the contents in the specification, and its technical scope must be determined according to the scope of the claims.

Claims

1. A real-time face animation method based on a video camera, characterized in that: The method comprises: In response to acquiring a video captured by a camera, extracting a target frame video containing a face image in the video; Perform triangulation on each face image in each target frame video to determine the key facial muscle points of each face in the target frame video; Determine expression change information of each face in the face image according to the quantity change information of the facial key muscle points of the same face and the position change information of the same facial key muscle points in the adjacent multiple frames of the target frame video; The human face in the video is animatedly displayed in a two-dimensional display screen according to the expression change information.

2. The real-time face animation method based on a video camera as claimed in claim 1, characterized in that: The triangulating each face image in each target frame video to determine the key facial muscle points of each face in the target frame video includes: Performing contour recognition on each of the target frame videos to identify facial organs in each of the facial images, and determining pixel types of pixels corresponding to each of the facial images in each of the target frame videos according to the contour recognition results, wherein the pixel types include facial organ pixel types and non-facial organ pixel types; Based on the principle that the connecting lines between two pixel points corresponding to each face image in each target frame video do not cross, any two pixel points corresponding to each face image in each target frame video are connected until no new connecting line between the two pixel points can be added in the face image; Determine the triangle type of each minimum triangle formed by three pixels in the face image according to the pixel point types of the three pixels in the minimum triangle; According to the triangle types of all the smallest triangles to which each pixel point in the facial image belongs, it is determined whether each pixel point is a key facial muscle point.

3. The real-time face animation method based on a video camera according to claim 2, characterized in that: The determining of the triangle type of the minimum triangle according to the pixel point types of three pixels in each minimum triangle formed by three pixels in the face image comprises: In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are all the facial organ pixel point types, determining the triangle type of the minimum triangle to be the organ triangle type; In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are all the non-facial organ pixel point types, determining the triangle type of the minimum triangle to be a facial triangle type; In a case where the pixel point types of the three pixels in each minimum triangle formed by three pixels in the face image are not a single pixel point type, it is determined that the triangle type of the minimum triangle is a mixed triangle type.

4. The real-time face animation method based on a video camera according to claim 3, characterized in that: The step of determining whether each pixel point is a key facial muscle point according to the triangle type of all the smallest triangles to which each pixel point in the face image belongs comprises: When the triangle type of all the smallest triangles to which the pixel point belongs in the face image is not a single triangle type, determining that the pixel point is a key facial muscle point; When the triangle types of all the smallest triangles to which the pixel point belongs in the face image are the facial triangle types, determining that the pixel point is not the facial key muscle point; When the triangle types of all the smallest triangles to which the pixel point in the facial image belongs are the organ triangle type or the mixed triangle type, whether the pixel point is the facial key muscle point is determined based on the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point.

5. The real-time face animation method based on a video camera according to claim 4, characterized in that: The determining whether the pixel point is the facial key muscle point according to the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point comprises: In the case where the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point is less than a preset ratio threshold, determining whether a pixel point on the same shortest connecting line as a target pixel point in the face image whose triangle type is the organ triangle type or the mixed triangle type and to which all the minimum triangles belong is the facial key muscle point; When a pixel point on the same shortest connecting line as a target pixel point in the face image whose triangle type is the organ triangle type or the mixed triangle type is a key facial muscle point, the target pixel point is determined to be the key facial muscle point; The above steps are traversed and executed. If after the shortest line traversal is completed, the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point in the face image is still less than the preset ratio threshold, the line is switched to the second shortest line, the third shortest line, and so on, until the ratio of the facial key muscle point to the pixel point that is not the facial key muscle point is greater than or equal to the preset ratio threshold.

6. The real-time face animation method based on a video camera according to any one of claims 1 to 5, characterized in that: Determining expression change information of each face in the face image according to the quantity change information of the facial key muscle points of the same face and the position change information of the same facial key muscle points in the adjacent multiple frames of the target frame video includes: Performing regional division on the key facial muscle points of the same face in the target frame videos of adjacent frames; Compare the number of the key facial muscle points in the same area of ​​the same face in two adjacent target frame videos with a reference model of the key facial muscle points established under a neutral expression to determine first expression change information; Tracking the position change information of the same facial key muscle point in the adjacent multiple frames of the target frame video by the optical flow method to determine the second expression change information; The expression change information of each of the faces in the face image is determined according to the first expression change information and the second expression change information.

7. The real-time face animation method based on a video camera according to claim 6, characterized in that: The method of tracking the position change information of the same facial key muscle point in the adjacent multiple frames of the target frame video by the optical flow method to determine the second expression change information includes: Using the first frame of the target frame video among the adjacent multiple frames as a base frame, using Taylor series, and using partial derivatives on the space and time coordinates to calculate the displacement and direction of the same facial key muscle point; The second expression change information is determined according to the displacement and direction of each of the key facial muscle points in the target frame video of adjacent multiple frames.

8. A real-time face animation device based on a video camera, characterized in that: include: An extraction module is configured to extract a target frame video containing a face image in response to acquiring a video captured by a camera; A first determination module is configured to triangulate each face image in each target frame video to determine the key facial muscle points of each face in the target frame video; A second determination module is configured to determine expression change information of each face in the face image according to the quantity change information of the facial key muscle points of the same face and the position change information of the same facial key muscle points in the adjacent multiple frames of the target frame video; The animation display module is configured to display the human face in the video in an animated manner in a two-dimensional display screen according to the expression change information.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the method according to any one of claims 1 to 7.