Method, device, equipment and storage medium for processing face posture in video

Abstract

The present application provides a method, device, equipment and computer-readable storage medium for processing facial posture in a video; the method comprises: performing facial feature point detection on multiple video frames containing a target object in a target video to obtain a first feature point set including two-dimensional coordinates of facial feature points; based on the first feature point set, performing three-dimensional reconstruction on the target object to obtain the facial posture of the target object and the three-dimensional coordinates of the facial feature points of the target object; projecting the three-dimensional coordinates of the facial feature points to the camera imaging surface to obtain a second feature point set including the two-dimensional coordinates of the facial feature points; comparing the second feature point set with the first feature point set, and when the error between the first feature point set and the second feature point set meets the error condition, the facial posture obtained by three-dimensional reconstruction is used as the target facial posture of the target object in the target video. Through the present application, the stability and accuracy of determining the target facial posture can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment and computer-readable storage medium for processing facial posture in a video. Background Art

[0002] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0003] Computer vision technology (CV) is one of the artificial intelligence technologies. It uses cameras and computers to replace the human eye to identify and measure targets, and further performs graphic processing to make the computer processing into an image that is more suitable for human eye observation or transmission to instrument detection. Computer vision technology includes video processing.

[0004] In video processing, recognizing the facial posture in the video frames contained in the video is an important technology. In related technologies, the facial posture in the video frames is usually estimated by training a deep learning model. Although this method can recognize the position and direction of the face in the video frame, the accuracy and stability are poor. Summary of the invention

[0005] The embodiments of the present application provide a method, device, equipment and computer-readable storage medium for processing facial posture in a video, which can improve the stability and accuracy of determining the target facial posture.

[0006] The technical solution of the embodiment of the present application is implemented as follows:

[0007] The present application provides a method for processing facial posture in a video, including:

[0008] Performing facial feature point detection on a plurality of video frames containing a target object in a target video to obtain a first feature point set including two-dimensional coordinates of the facial feature points;

[0009] Based on the first feature point set, three-dimensionally reconstruct the target object to obtain the face posture of the target object and the three-dimensional coordinates of the face feature points of the target object;

[0010] Projecting the three-dimensional coordinates of the facial feature points to obtain a second feature point set including the two-dimensional coordinates of the facial feature points;

[0011] The second feature point set is compared with the first feature point set, and when an error between the first feature point set and the second feature point set satisfies an error condition, the facial posture obtained by the three-dimensional reconstruction is used as the target facial posture of the target object in the target video.

[0012] The present application provides a device for processing facial posture in a video, including:

[0013] A detection module, configured to detect facial feature points on a plurality of video frames containing a target object in a target video, and obtain a first feature point set including two-dimensional coordinates of the facial feature points;

[0014] A reconstruction module, configured to perform three-dimensional reconstruction on the target object based on the first feature point set to obtain a facial posture of the target object and three-dimensional coordinates of facial feature points of the target object;

[0015] A projection module, used for projecting the three-dimensional coordinates of the facial feature points to obtain a second feature point set including the two-dimensional coordinates of the facial feature points;

[0016] A comparison module is used to compare the second feature point set with the first feature point set, and when the error between the first feature point set and the second feature point set meets the error condition, the face posture obtained by the three-dimensional reconstruction is used as the target face posture of the target object in the target video.

[0017] In the above solution, the detection module is further used to capture the face region of the target object from multiple video frames containing the target object in the target video to obtain multiple face region images;

[0018] Performing facial feature point detection on each face region image respectively to obtain the two-dimensional coordinates of the facial feature points in each face region image;

[0019] Performing coordinate transformation on the two-dimensional coordinates of the facial feature points in the facial region image to obtain the two-dimensional coordinates of the facial feature points in each video frame;

[0020] The two-dimensional coordinates of the facial feature points in each video frame are combined into a first feature point set.

[0021] In the above solution, the detection module is also used to perform face re-recognition on the target video to obtain the face trajectory of the target object;

[0022] Determining, according to the face trajectory of the target object, a plurality of video frames in the target video containing the target object and the face position of the target object in the video frames;

[0023] Perform facial feature point detection on the face located at the face position in the video frame.

[0024] In the above scheme, the detection module is further used to perform face re-recognition on the target video, determine the video frame in which the target object is recognized as the video frame containing the target object, and determine the face position of the target object in the video frame;

[0025] For a first video frame and a second video frame containing a target object, when the target object is not recognized in at least one third video frame between the first video frame and the second video frame, and the number of the third video frames is less than a first number threshold, obtaining a first face position of the target object in the first video frame and a second face position of the target object in the second video frame;

[0026] When the distance between the first face position and the second face position is less than a distance threshold, determining that the third video frame contains a target object, and

[0027] An interpolation process is performed according to the first face position and the second face position to obtain a face position of the target object in at least one of the third video frames to generate a face trajectory of the target object.

[0028] In the above scheme, the detection module is further used to perform face re-recognition on multiple video frames in the target video, determine the video frames that recognize the target object and the video frames that do not contain the target object, and determine the face position of the target object in the video frames;

[0029] For a first video frame and a second video frame containing a target object, when the target object is not recognized in at least one third video frame between the first video frame and the second video frame, and the number of the third video frames reaches a second number threshold, based on the first face position of the target object in the first video frame and the second face position of the target object in the second video frame, generate at least two segments of two face trajectories of the target object.

[0030] In the above solution, the reconstruction module is further used to estimate the face posture based on the first feature point set to obtain the face posture of the target object;

[0031] Based on the facial posture of the target object, the target object is three-dimensionally reconstructed through triangulation processing to determine the three-dimensional coordinates of the facial feature points of the target object.

[0032] In the above solution, the reconstruction module is further used to obtain a basic matrix based on the positional relationship between each facial feature point in the first feature point set in the plurality of video frames;

[0033] Normalizing the basic matrix to obtain an essential matrix;

[0034] Performing singular value decomposition on the essential matrix to obtain the facial posture of the target object.

[0035] In the above solution, the reconstruction module is also used to obtain the target camera intrinsic parameters of the target video;

[0036] Based on the target camera intrinsic parameters and the first feature point set, three-dimensionally reconstruct the target object;

[0037] In the process of performing three-dimensional reconstruction on the target object, obtaining the camera intrinsic parameters of the target video;

[0038] When the error between the first feature point set and the second feature point set does not meet the error condition, the target camera intrinsic parameters are updated based on the acquired camera intrinsic parameters to

[0039] Based on the updated target camera intrinsic parameters and the first feature point set, the target object is three-dimensionally reconstructed.

[0040] In the above solution, the reconstruction module is further used to obtain the basic matrix and the essential matrix during the process of three-dimensional reconstruction of the target object;

[0041] According to the conversion relationship between the basic matrix and the essential matrix, the camera intrinsic parameters of the target video are obtained.

[0042] In the above scheme, the reconstruction module is further used for, when the error between the first feature point set and the second feature point set does not meet the error condition and the number of facial feature points in each of the video frames is multiple, for each target facial feature point in the multiple facial feature points, removing the two-dimensional coordinates of at least one facial feature point from the first feature point set to obtain a third feature point set;

[0043] Based on the third feature point set, the target object is three-dimensionally reconstructed to obtain the three-dimensional coordinates of the facial feature points of the target object;

[0044] Projecting the three-dimensional coordinates obtained by three-dimensional reconstruction based on the third feature point set onto the camera imaging plane to obtain a fourth feature point set including the two-dimensional coordinates of the facial feature points;

[0045] The first feature point set is updated based on the third feature point set and the fourth feature point set.

[0046] In the above scheme, the projection module is further used to rotate and translate the three-dimensional coordinates of the facial feature points with respect to the camera imaging plane according to the facial posture, so as to obtain the coordinates of the facial feature points in the camera coordinate system;

[0047] Obtaining target camera intrinsic parameters corresponding to the target video;

[0048] According to the target camera intrinsic parameters, the coordinates in the camera coordinate system are converted into coordinates in the image coordinate system to obtain a second feature point set including two-dimensional coordinates of facial feature points.

[0049] In the above solution, the device further includes: a processing module, which is used to generate a picture corresponding to the information to be displayed based on the information to be displayed;

[0050] Obtaining original two-dimensional coordinates of multiple vertices of the image and three-dimensional coordinates of multiple vertices of the image;

[0051] Based on the target face posture, projecting the three-dimensional coordinates of multiple vertices of the image onto a camera imaging surface to obtain the two-dimensional coordinates of multiple vertices of the image;

[0052] Determine a perspective transformation matrix according to a mapping relationship between the original two-dimensional coordinates and the two-dimensional coordinates obtained by projection;

[0053] Performing an affine transformation on the image through a perspective transformation matrix to obtain a transformed target image;

[0054] The target image is superimposed on the video frame to obtain a video frame with the information to be displayed added thereto.

[0055] An embodiment of the present application provides a computer device, including:

[0056] A memory for storing executable instructions;

[0057] The processor is used to implement the method for processing facial posture in a video provided by an embodiment of the present application when executing the executable instructions stored in the memory.

[0058] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute instructions to implement a method for processing facial posture in a video provided by an embodiment of the present application.

[0059] The embodiments of the present application have the following beneficial effects:

[0060] By applying the above embodiment, facial feature point detection is performed on multiple video frames containing a target object in a target video to obtain a first feature point set including two-dimensional coordinates of facial feature points; based on the first feature point set, three-dimensional reconstruction is performed on the target object to obtain the facial posture of the target object and the three-dimensional coordinates of the facial feature points of the target object; the three-dimensional coordinates of the facial feature points are projected to obtain a second feature point set including the two-dimensional coordinates of the facial feature points; the second feature point set is compared with the first feature point set, and when the error between the first feature point set and the second feature point set meets the error condition, the facial posture obtained by the three-dimensional reconstruction is used as the target facial posture of the target object in the target video; in this way, the second feature point set is obtained by projection, and the second feature point set is compared with the first feature point set, so as to realize the verification of the three-dimensional reconstruction result and improve the stability and accuracy of the determination of the target facial posture. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is an optional architecture diagram of a system for processing facial posture in a video provided by an embodiment of the present application;

[0062] Figure 2 It is a flowchart of a method for processing facial posture in a video provided by an embodiment of the present application;

[0063] Figure 3 is a schematic diagram of the screening process of facial feature points provided in an embodiment of the present application;

[0064] Figure 4 is a schematic diagram of a text image provided in an embodiment of the present application in a face coordinate system;

[0065] Figure 5A-5B is a schematic diagram of a playback interface provided in an embodiment of the present application;

[0066] Figure 6 It is a flowchart of a method for processing facial posture in a video provided by an embodiment of the present application;

[0067] Figure 7 is a schematic diagram of a three-dimensional reconstruction process provided by an embodiment of the present application;

[0068] Figure 8 is a schematic diagram of a process of adding information to be displayed in a video frame provided by an embodiment of the present application;

[0069] Fig. 9 It is a structural schematic diagram of a device for processing facial posture in a video provided by an embodiment of the present application;

[0070] Fig.10 It is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0072] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0073] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0074] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0075] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0076] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0077] 1) Face pose: refers to the position (3D coordinates) and direction of the face relative to the camera coordinate system in the image captured by the camera.

[0078] 2) Camera internal parameters: The camera internal parameters recorded in matrix form include the camera focal length in the horizontal and vertical directions, the position of the optical center on the sensor plane, and other parameters, which can be expressed as:

[0079]

[0080] Among them, f x is the focal length parameter in the horizontal direction, f i is the vertical focal length parameter, (c x, c y ) is the position coordinate of the optical center on the camera sensor plane.

[0081] 3) Camera extrinsic parameters: Parameters used to record the rotation and translation of the camera relative to the world coordinate system. Generally speaking, a matrix is ​​used to record the rotation of the camera, and a vector is used to record the translation of the camera. When we use the face coordinate system as the world coordinate system, the camera extrinsic parameters are the face posture.

[0082] 4) Fundamental matrix: F is a 3×3 matrix that expresses the correspondence between the image points of the stereo pair.

[0083] 5) Essential Matrix: It is a matrix relationship used to connect two corresponding points in a three-dimensional coordinate system.

[0084] See also Figure 1 , Figure 1 It is an optional architecture diagram of a system for processing facial gestures in a video provided by an embodiment of the present application. To support an exemplary application, a terminal (exemplarily showing terminal 400-1 and terminal 400-2) is connected to server 200 via network 300. Network 300 can be a wide area network or a local area network, or a combination of the two. There is no restriction on the number of servers and terminals. In actual implementation, a client, such as a video client, a browser client, an information stream client, an education client, etc., is provided on the terminal for uploading and playing videos.

[0085] The terminal is used to receive an upload instruction for a target video and send the target video to a server;

[0086] The server 200 is used to detect facial feature points on a plurality of video frames containing a target object in a target video to obtain a first feature point set including two-dimensional coordinates of facial feature points; based on the first feature point set, perform three-dimensional reconstruction on the target object to obtain a facial posture of the target object and the three-dimensional coordinates of the facial feature points of the target object; project the three-dimensional coordinates of the facial feature points onto a camera imaging surface to obtain a second feature point set including the two-dimensional coordinates of the facial feature points; compare the second feature point set with the first feature point set, and when the error between the first feature point set and the second feature point set meets an error condition, use the facial posture obtained by the three-dimensional reconstruction as a target facial posture of the target object in the target video;

[0087] The terminal is further used to receive a play instruction for a target video and send a request to the server to obtain the target video, information to be displayed, and a target facial posture of a target object;

[0088] The server 200 is used to send the target video, the information to be displayed and the target face posture of the target object to the terminal;

[0089] A terminal is used to include a target video in a playback interface; during the playback of the target video, when the target video contains a target object and there is information to be displayed associated with the target object, the information to be displayed is displayed in a display area associated with the face of the target object; during the display of the information to be displayed, when it is determined that the target face posture of the target object has changed, the display posture of the information to be displayed is adjusted along with the change of the target face posture.

[0090] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a TV terminal, a car terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0091] Based on the above description of the system for processing face gestures in videos of the embodiment of the present application, the following describes the method for processing face gestures in videos provided by the embodiment of the present application. Figure 2 , Figure 2 : is a flow chart of a method for processing facial gestures in a video provided by an embodiment of the present application; in some embodiments, the method for processing facial gestures in a video can be implemented by a terminal or a server alone, or by a terminal and a server in collaboration. Taking the server alone as an example, the method for processing facial gestures in a video provided by an embodiment of the present application includes:

[0092] Step 201: The server performs facial feature point detection on multiple video frames containing a target object in a target video to obtain a first feature point set including two-dimensional coordinates of the facial feature points.

[0093] In actual implementation, the server obtains the target video and decodes the target video to obtain multiple video frames contained in the target video, and selects multiple video frames containing the target object from the multiple video frames contained in the target video to perform facial feature point detection to obtain two-dimensional coordinates including facial feature points. Among them, the first feature point set consists of multiple facial feature points, and the facial feature points are represented by the two-dimensional coordinates of the facial feature points.

[0094] In practical applications, for each video frame containing the target object, facial feature point detection is performed on the video frame to obtain the two-dimensional coordinates of the facial feature points in the video frame. In this way, the two-dimensional coordinates of the facial feature points in multiple video frames can be obtained, and these two-dimensional coordinates constitute the first feature point set.

[0095] For example, a 68-point feature point detection algorithm can be used to detect feature points on a face. Then, for each video frame, two-dimensional coordinates of 68 facial feature points can be obtained, such as using a face attention mechanism network (FAN, FaceAttention Network) to detect feature points on a face.

[0096] In some embodiments, the terminal can perform facial feature point detection on multiple video frames containing the target object in the target video in the following manner: from the multiple video frames containing the target object in the target video, capture the face area of ​​the target object to obtain multiple face area images; perform facial feature point detection on each face area image respectively to obtain the two-dimensional coordinates of the facial feature points in each face area image; perform coordinate transformation on the two-dimensional coordinates of the facial feature points in the face area image to obtain the two-dimensional coordinates of the facial feature points in each video frame; and form the two-dimensional coordinates of the facial feature points in each video frame into a first feature point set.

[0097] In practical applications, in order to improve the detection accuracy of facial feature points, the facial area image corresponding to the facial area of ​​the target object can be first captured from the video frame, and then facial feature point detection can be performed on the facial area image; here, the two-dimensional coordinates of the facial feature points obtained when performing facial feature point detection on the facial area image are the two-dimensional coordinates in the coordinate system of the corresponding facial area image, based on this, they need to be transformed into the coordinate system of the corresponding video frame.

[0098] In actual implementation, the position of the face area image in the video frame is obtained to determine the conversion relationship between the coordinate system of the corresponding face area image and the coordinate system of the corresponding video frame. For example, when the position of the upper left vertex of the face area image in the video frame is (20, 20), then when the two-dimensional coordinates of the facial feature points in the face area image are transformed, the horizontal and vertical coordinates of the two-dimensional coordinates of the facial feature points in the face area image are added by 20 to obtain the two-dimensional coordinates of the facial feature points in the video frame.

[0099] In some embodiments, before performing facial feature point detection on multiple video frames of the target video containing the target object, face re-identification can also be performed on the target video to obtain the face trajectory of the target object; accordingly, the terminal can perform facial feature point detection on multiple video frames of the target video containing the target object in the following manner: according to the face trajectory of the target object, determine the multiple video frames of the target video containing the target object, and the face position of the target object in the video frame; perform facial feature point detection on the face located at the face position in the video frame.

[0100] Here, since a video frame may include multiple objects, that is, a video frame including a target object may also include other objects, it is necessary to determine which object is the target object so as to detect the facial feature points of the target object.

[0101] In actual implementation, the face trajectory of the target object can be obtained through face re-recognition. The face trajectory here can identify the face position of the target object in each video frame. In this way, after obtaining the face trajectory, it is possible to determine which video frames contain the target object and the face position of the target object in the corresponding video frame. According to the face position of the target object, facial feature points are detected for the face at the corresponding face position, thereby avoiding erroneous judgment of other objects as target objects and detecting facial feature points of other objects.

[0102] In practical applications, the server can perform face detection on each video frame of the target video to obtain all faces in each video frame; then, based on the detected faces, the face region is intercepted to obtain a face region image, and each face region image is re-identified to obtain the object corresponding to each face region image; then, according to the position of the face region image corresponding to the target object in the corresponding video frame, the face trajectory of the target object is determined. Among them, the position of the face region image in the corresponding video frame is the face position of the target object in the video frame.

[0103] In some embodiments, the target video may be subjected to face re-identification to obtain the face trajectory of the target object in the following manner: face re-identification is performed on the target video, and the video frame in which the target object is identified is determined as the video frame containing the target object, and the face position of the target object in the video frame is determined; for the first video frame and the second video frame containing the target object, when the target object is not identified in at least one third video frame between the first video frame and the second video frame, and the number of third video frames is less than a first number threshold, the first face position of the target object in the first video frame and the second face position of the target object in the second video frame are obtained; when the distance between the first face position and the second face position is less than a distance threshold, it is determined that the third video frame contains the target object, and interpolation processing is performed based on the first face position and the second face position to obtain the face position of the target object in at least one third video frame to generate the face trajectory of the target object.

[0104] In actual implementation, the server can perform face detection on each video frame contained in the target video to obtain all faces in each video frame; then, based on the detected faces, the face region is intercepted to obtain a face region image, and each face region image is re-identified to obtain the object corresponding to each face region image; and then the face trajectory of the target object is determined according to the position of the face region image corresponding to the target object in the corresponding video frame. The position of the face region image in the corresponding video frame is the face position of the target object in the video frame.

[0105] In practical applications, when determining the face position of the target object according to the position of the face area image corresponding to the target object in the corresponding video frame, if the target object is not recognized in a certain M video frame (the third video frame), but the target object is recognized in the two video frames before and after the M video frames (i.e., the first video frame and the second video frame), and the distance between the face positions of the target object in the two video frames before and after is less than the distance threshold, it means that the face positions of the target object in the two video frames before and after are close enough, and interpolation can be used here to supplement the face position of the middle frame (the third video frame). Wherein, M is a natural number, and M is less than the first quantity threshold, such as the first quantity threshold can be 3.

[0106] Exemplarily, a linear interpolation method can be used here to obtain the face position of the target object in the third video frame, that is, a linear function is constructed according to the first face position and the second face position, and then the face position of the target object in the third video frame is determined based on the linear function. For example, the first face position is (200, 200), and the second face position is (260, 260). When two third video frames are included between the first video frame and the second video frame, the face position of the target object in the first third video frame (one video frame after the first video frame) is (220, 220), and the face position of the target object in the second third video frame (two video frames after the first video frame) is (240, 240).

[0107] Here, since the target object is not detected in the third video frame, when determining the facial posture of the target object in the third face frame, interpolation can also be used, that is, the facial posture of the target object in the first video frame and the facial posture of the target object in the second video frame are obtained, and then the facial posture of the target object in the third video frame is determined by interpolation based on the facial posture of the target object in the first video frame and the facial posture of the target object in the second video frame.

[0108] In practical applications, if the target object is not detected in M ​​video frames (the third video frame), but the target object is recognized in the two video frames before and after the M video frames (i.e., the first video frame and the second video frame), and the distance between the face positions of the target object in the two video frames before and after reaches the distance threshold, it means that the face positions of the target object in the two video frames before and after are not close enough, then, it is considered that the video screen is switched, and the face trajectory is divided into two segments according to the moment of video screen switching, that is, the face position in the first video frame is used as the end position of the previous face trajectory, and the face position in the second video frame is used as the starting position of the next face trajectory.

[0109] In some embodiments, the target video can be subjected to face re-identification to obtain the face trajectory of the target object in the following manner: face re-identification is performed on multiple video frames in the target video, the video frames in which the target object is identified are determined as video frames containing the target object, and the face positions of the target objects in the video frames are determined; for a first video frame and a second video frame containing the target object, when the target object is not identified in at least one third video frame between the first video frame and the second video frame, and the number of third video frames reaches a second number threshold, at least two face trajectories of the target object are generated based on the first face position of the target object in the first video frame, and the second face position of the target object in the second video frame.

[0110] In practical applications, if the target object is not recognized in a certain N video frames (the third video frame), but the target object is recognized in the two video frames before and after the N video frames (i.e., the first video frame and the second video frame), N is compared with the second quantity threshold. If N reaches the second quantity threshold, it is considered that the video screen is switched, and the face track is divided into two segments according to the moment of video screen switching, that is, the face position in the first video frame is used as the end position of the previous face track, and the face position in the second video frame is used as the start position of the next face track. Wherein, N is a positive integer, and the second quantity threshold is greater than or equal to the first quantity threshold.

[0111] In some embodiments, the target object is identified in the first video frame, the second video frame and the third video frame, wherein the first video frame is the previous video frame adjacent to the third video frame, and the second video frame is the subsequent video frame adjacent to the third video frame. When the face position of the target object in the third video frame jumps relative to the first video frame and the second video frame, that is, the distance between the face position of the target object in the first video frame and the face position of the target object in the third video frame exceeds the distance threshold, and the distance between the face position of the target object in the second video frame and the face position of the target object in the third video frame exceeds the distance threshold. For example, the distance threshold can be set to 5% of the image height. In this case, it is considered that the detection of the target object in the third video frame has failed, and the correct face data in the video frame is obtained by interpolation through the face position in the first video frame and the face position in the second video frame.

[0112] Step 202: Based on the first feature point set, the target object is three-dimensionally reconstructed to obtain the face posture of the target object and the three-dimensional coordinates of the face feature points of the target object.

[0113] In actual implementation, the first feature point set includes the two-dimensional coordinates of facial feature points in multiple video frames. According to the positional relationship of each facial feature point in different video frames, the target object is reconstructed in three dimensions. For example, when the facial feature points include the left corner of the left eye, the right corner of the right eye, the left corner of the mouth, the right corner of the mouth, the tip of the nose, and the chin, the two-dimensional coordinates of the left corner of the left eye in each video frame are obtained, and the positional relationship of the left corner of the left eye in each video frame is determined. Based on the positional relationship, the target object is reconstructed in three dimensions. Here, the above-mentioned operation for the left corner of the left eye is performed on each facial feature point to obtain the facial posture of the target object and the three-dimensional coordinates of the facial feature points of the target object.

[0114] In some embodiments, the target object can be reconstructed in three dimensions based on the first feature point set to obtain the facial posture of the target object and the three-dimensional coordinates of the facial feature points of the target object in the following manner: facial posture estimation is performed based on the first feature point set to obtain the facial posture of the target object; based on the facial posture of the target object, the target object is reconstructed in three dimensions through triangulation processing to determine the three-dimensional coordinates of the facial feature points of the target object.

[0115] In actual implementation, the face posture of the target object is first estimated according to the positional relationship of each facial feature point in the first feature point set in multiple video frames to obtain the facial posture of the target object; based on the facial posture of the target object, the target object is three-dimensionally reconstructed through triangulation processing to obtain the three-dimensional coordinates of the facial feature points of the target object.

[0116] Here, the triangulation processing is explained. When the face coordinate system of the target object is used as the world coordinate system, the change in the facial posture of the target object is equivalent to observing the face of the target object at different positions. When the two-dimensional coordinates of the face observed at different positions are known, the triangulation relationship in the projection perspective model is used to reconstruct the target object in three dimensions.

[0117] In some embodiments, the server can perform facial posture estimation on the first feature point set to obtain the facial posture of the target object in the following manner: obtain a basic matrix based on the positional relationship of each facial feature point in the first feature point set in multiple video frames; normalize the basic matrix to obtain an essential matrix; perform singular value decomposition on the essential matrix to obtain the facial posture of the target object.

[0118] In actual implementation, the base matrix is ​​a 3x3 matrix of rank 2 with 7 degrees of freedom, which can be calculated based on Calculate the basic matrix, where F is the basic matrix, and is the two-dimensional coordinates of a pair of matching points in a video frame in the corresponding video frame. For example, the two-dimensional coordinates of the same facial feature point in two adjacent video frames, such as the two-dimensional coordinates of the chin in the first video frame is The 2D coordinates of the chin in the second video frame are

[0119] Here, when calculating the basic matrix, using the two-dimensional coordinates of multiple facial feature points can make the determined basic matrix more accurate.

[0120] After obtaining the basic matrix, the relationship between the basic matrix and the essential matrix is: F = K -TEK′, where F is the basic matrix, K is the camera intrinsic parameter, and E is the essential matrix. After normalization, the camera intrinsic parameter becomes the unit matrix, and the basic matrix is ​​the essential matrix. Based on this, the basic matrix is ​​normalized to obtain the essential matrix.

[0121] After obtaining the essential matrix, the essential matrix is ​​subjected to singular value decomposition. Here, four groups of translation vectors and rotation angles can be obtained by performing singular value decomposition on the essential matrix, of which only one group is correct. Based on this, the three-dimensional coordinates corresponding to the facial feature points are determined according to the triangulation processing, and the three-dimensional coordinates are substituted into each group of translation vectors and rotation angles to obtain the three-dimensional coordinates in the corresponding camera coordinate system. Only one group of resulting depth values ​​is positive, thereby screening out the correct translation vector and rotation angle to obtain the facial posture of the target object.

[0122] Step 203: Project the three-dimensional coordinates of the facial feature points to obtain a second feature point set including the two-dimensional coordinates of the facial feature points.

[0123] In actual implementation, the facial feature points are projected onto the camera presentation surface according to the facial posture of the target object. Here, the facial posture of the target object in different video frames is different. When the three-dimensional coordinates of the facial feature points are projected, the two-dimensional coordinates of the facial feature points corresponding to each video frame are obtained. For example, when each face is represented by 68 facial feature points, the two-dimensional coordinates of the facial feature points corresponding to each video frame are the two-dimensional coordinates of the 68 facial feature points obtained by projection.

[0124] In some embodiments, the three-dimensional coordinates of facial feature points can be projected in the following manner to obtain a second feature point set including the two-dimensional coordinates of the facial feature points: according to the facial posture, the three-dimensional coordinates of the facial feature points are rotated and translated with respect to the camera imaging plane to obtain the coordinates of the facial feature points in the camera coordinate system; the target camera internal parameters corresponding to the target video are obtained; according to the target camera internal parameters, the coordinates in the camera coordinate system are converted to coordinates in the image coordinate system to obtain a second feature point set including the two-dimensional coordinates of the facial feature points.

[0125] Here, the facial posture is represented by the translation vector and three rotation angles (pitch angle, yaw angle, roll angle) of the face relative to the camera. In actual implementation, the three-dimensional coordinates of the facial feature points are translated according to the translation vector, and the three-dimensional coordinates of the facial feature points are rotated according to the three rotation angles to obtain the coordinates of the facial feature points in the camera coordinate system; then, the product of the target camera intrinsic parameters and the coordinates of the facial feature points in the camera coordinate system is obtained, and the product is used as the coordinates of the facial feature points in the image coordinate system to realize the projection of the three-dimensional coordinates of the facial feature points.

[0126] Step 204: compare the second feature point set with the first feature point set, and when the error between the first feature point set and the second feature point set meets the error condition, use the face posture obtained by three-dimensional reconstruction as the target face posture of the target object in the target video.

[0127] In actual implementation, the mean square error between the second feature point set and the first feature point set can be obtained. When the mean square error between the second feature point set and the first feature point set is less than the error threshold, it is determined that the error between the first feature point set and the second feature point set meets the error condition; otherwise, it is determined that the error between the first feature point set and the second feature point set does not meet the error condition.

[0128] In some embodiments, the target object can be three-dimensionally reconstructed based on the first feature point set in the following manner: obtaining target camera intrinsic parameters of the target video; performing three-dimensional reconstruction of the target object based on the target camera intrinsic parameters and the first feature point set; the method also includes: in the process of three-dimensional reconstruction of the target object, obtaining the camera intrinsic parameters of the target video; when the error between the first feature point set and the second feature point set does not meet the error condition, updating the target camera intrinsic parameters based on the obtained camera intrinsic parameters, so as to perform three-dimensional reconstruction of the target object based on the updated target camera intrinsic parameters and the first feature point set.

[0129] Here, the target camera intrinsic parameters of the target video obtained for the first time are the initial camera intrinsic parameters, which can be given manually or obtained based on the camera intrinsic parameter estimation algorithm. Here, the initial camera intrinsic parameters are <focal length set to 50mm><optical center position set to the center position of the imaging surface>. When the initial camera intrinsic parameters are given manually, they can be set according to the height of the image. When the error between the first feature point set and the second feature point set does not meet the error condition, such as when the mean square error between the second feature point set and the first feature point set reaches the error threshold, it means that the target camera intrinsic parameters may not be accurate enough. Here, the target camera intrinsic parameters are updated according to the camera intrinsic parameters obtained during the three-dimensional reconstruction process.

[0130] In actual implementation, after obtaining the intrinsic parameters of the target camera, the three-dimensional coordinates of the facial feature points in each video frame in the corresponding camera coordinate system can be determined according to the intrinsic parameters of the target camera; then, based on p T Ep′=0, obtain the calculated essential matrix, where E is the basic matrix, p and p′ are the three-dimensional coordinates of a pair of matching points in the video frame in the camera coordinate system, for example, the three-dimensional coordinates of the same facial feature point in two adjacent video frames in the corresponding camera coordinate system, such as the three-dimensional coordinate of the chin in the camera coordinate system in the first video frame is p, and the two-dimensional coordinate of the chin in the second video frame is p′; after obtaining the essential matrix, perform singular value decomposition on the essential matrix to obtain the facial posture of the target object.

[0131] Here, the intrinsic parameters of the target camera may be updated in an iterative manner until the error between the first feature point set and the second feature point set satisfies the error condition.

[0132] In some embodiments, the camera intrinsic parameters of the target video can be obtained during the three-dimensional reconstruction of the target object in the following manner: during the three-dimensional reconstruction of the target object, a basic matrix and an intrinsic matrix are obtained; based on the conversion relationship between the basic matrix and the intrinsic matrix, the camera intrinsic parameters of the target video are obtained.

[0133] In actual implementation, during the 3D reconstruction process, the basic matrix and the essential matrix are obtained, and F = K - T EK′, where F is the basic matrix, K is the camera intrinsic parameter, and E is the essential matrix. In this way, the camera intrinsic parameter K can be calculated.

[0134] In some embodiments, the terminal can also eliminate the two-dimensional coordinates of at least one facial feature point from the first feature point set to obtain a third feature point set when the error between the first feature point set and the second feature point set does not meet the error condition and the number of facial feature points in each video frame is multiple; based on the third feature point set, perform three-dimensional reconstruction on the target object to obtain the three-dimensional coordinates of the facial feature points of the target object; project the three-dimensional coordinates obtained by the three-dimensional reconstruction based on the third feature point set to the camera imaging surface to obtain a fourth feature point set including the two-dimensional coordinates of the facial feature points; and update the first feature point set based on the third feature point set and the fourth feature point set.

[0135] In actual implementation, when the three-dimensional reconstruction is performed for the first time, the two-dimensional coordinates of all detected facial feature points are used as the first feature point set. If the error between the first feature point set and the second feature point set does not meet the error condition, the facial feature points can be screened to remove the facial feature points that reduce the accuracy of the three-dimensional reconstruction. For example, due to the behavior of people speaking, the position of the facial feature points corresponding to the upper lip relative to other facial feature points is not fixed. Based on this, the facial feature points corresponding to the upper lip in the facial feature points can be removed, and after the removal, the removal result is verified, that is, based on the third feature point set, the target object is reconstructed in three dimensions to obtain the three-dimensional coordinates of the facial feature points of the target object; the three-dimensional coordinates obtained by the three-dimensional reconstruction based on the third feature point set are projected to the camera imaging surface to obtain the fourth feature point set; the error between the third feature point set and the fourth feature point set is determined, and when the error meets a certain error condition, the first feature point set is updated with the third feature point set.

[0136] For example, feature point detection is performed on the target object to obtain the two-dimensional coordinates of 68 facial feature points, then the first feature point set contains the two-dimensional coordinates of the 68 facial feature points; the two-dimensional coordinates of the facial feature points corresponding to the lips are eliminated from the first feature point set, since the number of facial feature points corresponding to the lips is 16, then the two-dimensional coordinates of the remaining 52 facial feature points are used to form a third feature point set; then based on the third feature point set, the target object is three-dimensionally reconstructed to obtain the three-dimensional coordinates of the 52 facial feature points; the three-dimensional coordinates of the 52 facial feature points are projected onto the camera imaging surface to obtain a fourth feature point set, where the fourth feature point set contains the two-dimensional coordinates of the 52 facial feature points obtained by projection; the two-dimensional coordinates in the third feature point set are compared with the two-dimensional coordinates in the fourth feature point set to obtain the error between the third feature point set and the fourth feature point set, and when the error meets the error condition, the first feature point set is updated with the third feature point set.

[0137] In some embodiments, multiple third feature point sets and corresponding fourth feature point sets may be obtained, and the first feature point set may be updated by comparing the errors between the multiple third feature point sets and the corresponding fourth feature point sets. Figure 3 is a schematic diagram of the screening process of facial feature points provided in the embodiment of the present application, see Figure 3 , for each facial feature point, remove the two-dimensional coordinates of the facial feature point from the first feature point set to obtain the corresponding third feature point set, such as removing the two-dimensional coordinates of facial feature point 1 from the first feature point set to obtain the corresponding third feature point set, and obtain the third feature point set 1; when the number of facial feature points is N, N third feature point sets can be obtained. After obtaining the N third feature point sets, for each third feature point set, based on the third feature point set, the target object is reconstructed in three dimensions to obtain the three-dimensional coordinates of the facial feature points of the target object; the three-dimensional coordinates obtained by the three-dimensional reconstruction based on the third feature point set are projected to the camera imaging plane to obtain the fourth feature point set including the two-dimensional coordinates of the facial feature points; after obtaining the fourth feature point set corresponding to the N third feature point sets, based on the errors between each third feature point set and the corresponding fourth feature point set, the third feature point set with the smallest corresponding error is selected to update the first feature point set.

[0138] Here, the facial feature points may be screened in an iterative manner until the error obtained after the first feature point set is updated satisfies the error condition, or the number of facial feature points is lower than a number threshold.

[0139] In some embodiments, after obtaining the target face posture, the server may further perform filtering processing on the target face posture based on the face trajectory to obtain the target face posture without jitter.

[0140] In actual implementation, for each segment of face trajectory, a target face posture sequence corresponding to the face trajectory is obtained; here, the face posture in the target face posture sequence includes data of 6 dimensions, and the data of each dimension in the target face posture sequence is filtered to filter out jitter. Among them, an extended Kalman filter can be used to filter the data of each dimension, and the value at the start time point of the face trajectory is used as the initial value of the extended Kalman filter to ensure that the filtering result is normal and available at the start time of each segment of the trajectory.

[0141] In some embodiments, the terminal can also generate a picture corresponding to the information to be displayed based on the information to be displayed; obtain the original two-dimensional coordinates of multiple vertices of the picture and the three-dimensional coordinates of multiple vertices of the picture; based on the target face posture, project the three-dimensional coordinates of multiple vertices of the picture onto the camera imaging surface to obtain the two-dimensional coordinates of multiple vertices of the picture; determine the perspective transformation matrix according to the mapping relationship between the original two-dimensional coordinates and the projected two-dimensional coordinates; perform an affine transformation on the picture through the perspective transformation matrix to obtain the transformed target picture; superimpose the target picture on the video frame to obtain a video frame with the information to be displayed added.

[0142] In actual implementation, the information to be displayed can be displayed based on the determined target face posture, so that when the target face posture of the target object changes during the display of the information to be displayed, the display posture of the information to be displayed is adjusted along with the change of the target face posture. Here, the information to be displayed can be text information, image information, etc., and the specific form of the information to be displayed is not limited here.

[0143] In actual applications, when the information to be displayed is image information, the information to be displayed is directly used as a picture; when the information to be displayed is text information, a text picture is drawn to obtain a picture corresponding to the information to be displayed. Here, when drawing a text picture, automatic line break can be selected, that is, when the character length reaches a given threshold, a line break symbol is automatically added, the text has a transparent channel, and the non-text area is transparent.

[0144] In practical applications, multiple vertices refer to at least two vertices. Here, the original two-dimensional coordinates of the image can be determined according to the shape, size and display position of the image. For example, when the image is a rectangle with a length of w and a width of H, and the height of the text image from the nose of the target object is D, the four vertices of the text image are the upper left corner (0, 0), the lower left corner (0, h-1), the lower right corner (w-1, h-1), and the upper right corner (w-1, 0).

[0145] Also, the three-dimensional coordinates of multiple vertices of the image here refer to the three-dimensional coordinates of multiple vertices of the image in the face coordinate system. The original coordinates of multiple vertices can be determined according to the size of the image and the positional relationship between the original coordinates and the face of the target object. For example, the length and width of the image can be obtained, and the three-dimensional coordinates of multiple vertices of the image in the face coordinate system can be determined based on the length and width of the image and the positional relationship between the display area of ​​the image and the face of the target object.

[0146] As an example, let's display the image above the target object's head. Figure 4 is a schematic diagram of a text image provided in an embodiment of the present application in a face coordinate system, see Figure 4 When the image is a rectangle with a length of w and a width of H, and a height of D from the nose of the target object, the coordinates of the four vertices of the image in the face coordinate system are determined as the upper left corner (-W / 2, -DH, 0), the lower left corner (-W / 2, -D, 0), the lower right corner (W / 2, -D, 0), and the upper right corner (W / 2, -DH, 0).

[0147] After obtaining the original two-dimensional coordinates of multiple vertices of the image and the three-dimensional coordinates of multiple vertices of the image, project the three-dimensional coordinates of multiple vertices of the image onto the camera imaging surface based on the target face posture to obtain the two-dimensional coordinates of multiple vertices of the image; mark the two-dimensional coordinates of multiple vertices as point combination A, mark the original two-dimensional coordinates of multiple vertices as point combination B, and calculate the perspective transformation matrix M of point combination B to point combination A; then perform affine transformation on the image through the perspective transformation matrix, that is, multiply the coordinates of each point in the image by the perspective transformation matrix to obtain the coordinates of the point in the target image, and then obtain the target image. After obtaining the target image, superimpose the target image on the video frame. When the terminal receives the playback instruction for the target video, it renders based on the video frame with the information to be displayed, so that the display posture of the information to be displayed obtained by rendering matches the facial posture of the target object.

[0148] See also Figure 5A-5B , Figure 5A-5B It is a schematic diagram of the playback interface provided in an embodiment of the present application, in which the display posture of the information to be displayed 501 is consistent with the facial posture of the target object 502, both of which are rotated to the left; and the display posture of the information to be displayed 503 is consistent with the facial posture of the target object 504, both of which are rotated to the left; and, since the face of the target object 504 is rotated to the left at a larger angle relative to the face of the target object 502, correspondingly, the information to be displayed 503 is rotated to the left at a larger angle relative to the information to be displayed 501.

[0149] By applying the above embodiment, facial feature point detection is performed on multiple video frames containing a target object in a target video to obtain a first feature point set including two-dimensional coordinates of facial feature points; based on the first feature point set, three-dimensional reconstruction is performed on the target object to obtain the facial posture of the target object and the three-dimensional coordinates of the facial feature points of the target object; the three-dimensional coordinates of the facial feature points are projected to obtain a second feature point set including the two-dimensional coordinates of the facial feature points; the second feature point set is compared with the first feature point set, and when the error between the first feature point set and the second feature point set meets the error condition, the facial posture obtained by three-dimensional reconstruction is used as the target facial posture of the target object in the target video; in this way, the second feature point set is obtained by projection, and the second feature point set is compared with the first feature point set, so as to realize the verification of the three-dimensional reconstruction result and improve the stability and accuracy of the determination of the target facial posture.

[0150] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0151] Figure 6 is a flow chart of a method for processing facial posture in a video provided by an embodiment of the present application, see Figure 6 In some embodiments, the method for processing facial gestures in the video can be implemented by the terminal or the server alone, or by the terminal and the server in collaboration. Taking the server alone as an example, the method for processing facial gestures in the video provided in the embodiment of the present application includes:

[0152] Step 601: The server receives a target video uploaded by a user.

[0153] Step 602: The server performs face detection on each video frame in the target video using a face detection algorithm.

[0154] For each video frame, face detection is performed on the video frame to detect all faces in the video frame. Here, a multi-task convolutional neural network (MTCNN) can be used to perform face detection on the video frame.

[0155] Step 603: The server re-identifies the faces in each video frame.

[0156] Here, a celebrity recognition algorithm can be used to re-recognize the face and obtain the face track. It can be understood that the face track here corresponds to the object, that is, when the target video contains multiple objects, the face track of each object can be obtained.

[0157] In actual implementation, face re-recognition is performed on the faces in each video frame to determine the object to which each face belongs, and the face trajectory is obtained according to the position of the same object in consecutive video frames.

[0158] In practical applications, if the face of a certain object is not detected in M ​​video frames, but the face of the object is detected in the two video frames before and after the M video frames, and the face position (face frame position) of the object in the two video frames before and after is close enough, the face data (face position, face posture, etc.) of the middle frame are supplemented by interpolation. Among them, M is a natural number, such as M can be 2.

[0159] If a video frame detects the face of an object, but the face position in the two video frames before and after the video frame jumps (the distance between the face positions in two adjacent video frames exceeds the distance threshold, such as 5% of the image height), then the face detection in the video frame is considered to have failed, and the correct face data in the video frame is obtained by interpolation using the face data of the previous and next frames.

[0160] If there is no face of a certain object in N consecutive video frames, but the face of the object is detected in the two video frames before and after N video frames, it is considered that the video screen is switched, and the face trajectory is divided into two segments according to the switching time. Here N is a natural number, and N is greater than M, such as M is 2 and N is 3.

[0161] Step 604: The server detects facial feature points.

[0162] Here, based on the face trajectory, feature point detection is performed on the face. Here, a 68-point feature point detection algorithm can be used to detect feature points on the face, such as using a face attention mechanism network (FAN, Face Attention Network) to detect face feature points.

[0163] In actual implementation, based on the detected face, the face area image corresponding to each video frame is intercepted, and according to the face trajectory, a face area image sequence matching each face trajectory is obtained. Here, for each face area image in the face area image sequence, face feature point detection is performed on the face area image to determine the coordinates of each face feature point in the face area image; then, based on the position of the face area image in the video frame, the coordinates of the face feature points are converted to determine the coordinates of each face feature point in the video frame.

[0164] Step 605: Obtain initial camera intrinsic parameters.

[0165] Here, an initial camera intrinsic parameter is obtained for initializing the subsequent algorithm. The initial camera intrinsic parameter can be given manually or obtained based on the camera intrinsic parameter estimation algorithm. Here, the initial camera intrinsic parameter is <focal length is set to 50mm><optical center position is set to the center position of the imaging surface>. When the initial camera intrinsic parameter is given manually, it can be set according to the height of the image.

[0166] Step 606: Perform 3D reconstruction based on the initial camera intrinsic parameters and facial feature points.

[0167] Here, a monocular camera 3D reconstruction algorithm is used for 3D reconstruction, such as the Structure From Motion (SFM) algorithm, to obtain the face posture, accurate camera internal parameters and 3D coordinates of face feature points.

[0168] The three-dimensional reconstruction process is described in detail below. Figure 7 is a schematic diagram of the three-dimensional reconstruction process provided in the embodiment of the present application, see Figure 7 , the three-dimensional reconstruction process provided in the embodiment of the present application includes:

[0169] Step 701: Obtain the face coordinates in each video frame.

[0170] Step 702: Screen the facial feature points to obtain a first feature point set.

[0171] Here, the facial feature points can be screened in an iterative manner until the error obtained after updating the first feature point set meets the error condition, or the number of facial feature points is lower than the number threshold. All facial feature points are retained for the first time, and in the iterative process, the following is adopted: Figure 3 The method shown is used to filter facial feature points, see Figure 3 , for each facial feature point, the two-dimensional coordinates of the facial feature point are removed from the first feature point set to obtain the corresponding third feature point set, such as removing the two-dimensional coordinates of facial feature point 1 from the first feature point set to obtain the corresponding third feature point set, i.e., third feature point set 1; when the number of facial feature points is N, N third feature point sets can be obtained. After obtaining N third feature point sets, for each third feature point set, the third feature point set is input into the SMF module, and the target object is reconstructed in three dimensions to obtain the three-dimensional coordinates of the facial feature points of the target object; the three-dimensional coordinates obtained by the three-dimensional reconstruction based on the third feature point set are projected onto the camera imaging plane to obtain a fourth feature point set including the two-dimensional coordinates of the facial feature points; after obtaining the fourth feature point set corresponding to the N third feature point sets, based on the errors between each third feature point set and the corresponding fourth feature point set, the third feature point set with the smallest corresponding error is selected to update the first feature point set.

[0172] Step 703: Obtain the target camera intrinsic parameters.

[0173] Here, the target camera intrinsic parameters acquired for the first time are initial camera intrinsic parameters, which may be manually set. After the camera intrinsic parameter update operation is performed, the target camera intrinsic parameters are updated camera intrinsic parameters.

[0174] Step 704: Input the target camera intrinsic parameters and the first feature point set into the SMF module, and output the face posture, camera intrinsic parameters and three-dimensional coordinates of the face feature points.

[0175] Here, through the SMF module, based on the positional relationship of each facial feature point in the first feature point set in multiple video frames, the target object is three-dimensionally reconstructed to obtain the facial posture, camera internal parameters and three-dimensional coordinates of the facial feature points.

[0176] Step 705: According to the facial posture and the camera internal parameters, the three-dimensional coordinates of the facial feature points are projected onto the camera imaging plane to obtain a second feature point set.

[0177] Step 706: Obtain the mean square error between the first feature point set and the second feature point set.

[0178] Step 707: Determine whether the mean square error is less than the error threshold. If so, execute step 708; otherwise, execute step 709 and step 702.

[0179] Step 708: Output the obtained facial posture, camera internal parameters and three-dimensional coordinates of facial feature points.

[0180] Step 709: Use the obtained camera intrinsic parameters to update the target camera intrinsic parameters.

[0181] Here, after the obtained face posture is output, the face posture can also be filtered based on the face trajectory to obtain a face posture without jitter.

[0182] For each face track, a face posture sequence corresponding to the face track is obtained; here, the face posture in the face posture sequence includes data of 6 dimensions, and the data of each dimension in the face posture sequence is filtered to filter out jitter. Among them, an extended Kalman filter can be used to filter the data of each dimension, and the value of the start time point of the face track is used as the initial value of the extended Kalman filter to ensure that the filtering result is normal and available at the start time of each track.

[0183] After obtaining the jitter-free face posture, the face position, face posture, face ID (identification object), video frame number, video ID, lens focal length and other information detected in each video frame are stored in the face posture database in the order of video frames for subsequent use by the terminal.

[0184] Here, after obtaining the facial posture, the information to be displayed can be bound to the face, so that when the information to be displayed is displayed, the display posture of the information to be displayed changes synchronously with the change of the facial posture of the object.

[0185] Here, the information to be displayed is the bullet message information as an example, see Figure 5A The barrage information (information to be displayed) 501 associated with the target object is displayed in the top area of ​​the target object 502, wherein the facial posture of the target object matches the display posture of the barrage. During the playback of the video, as the facial posture of the target object changes, the display posture of the barrage changes accordingly.

[0186] The association relationship between the target object and the barrage can be determined by matching the barrage with the object in the video screen, or can be associated by the user when posting the barrage.

[0187] Take the subtitles as an example, see Figure 5B , subtitles 503 corresponding to the current conversation content of the target object are displayed in the top area of ​​the target object 504, wherein the facial posture of the target object matches the display posture of the subtitles. During the playback of the video, as the facial posture of the target object changes, the display posture of the barrage changes accordingly.

[0188] The specific process of adding information to be displayed in a video frame is described below. The specific process of adding information to be displayed in a video frame can be executed by a terminal, or by a server, or by both a server and a terminal. Taking the execution by a terminal as an example, Figure 8 This is a schematic diagram of the process of adding information to be displayed in a video frame provided by an embodiment of the present application, see Figure 8 When the information to be displayed is text, the process of the terminal adding the information to be displayed in the video frame includes:

[0189] Step 801: Draw a text image.

[0190] Draw a text image according to the text content of the information to be displayed; automatic line break can be selected during drawing, that is, when the character length reaches a given threshold, a line break symbol is automatically added, the text has a transparent channel, and the non-text area is transparent.

[0191] Step 802: Determine the coordinates of the text image in the face coordinate system.

[0192] Here, taking drawing on the top of the target object's head as an example, in the face coordinate system, the height D (y-axis direction) from the text image to the nose is fixed, the height H of the text image is fixed, and the width W of the text image in the face coordinate system is calculated according to the aspect ratio r of the text image, where W = Hr. For example, D can be set to 125 and H can be set to 40.

[0193] Correspondingly, the coordinates of the four vertices of the text image in the face coordinate system can be determined, namely, the upper left corner (-W / 2, -DH, 0), the lower left corner (-W / 2, -D, 0), the lower right corner (W / 2, -D, 0), and the upper right corner (W / 2, -DH, 0).

[0194] Step 803: Based on the facial posture, the coordinates of the text image in the facial coordinate system are projected onto the camera imaging surface using a point cloud projection method.

[0195] In actual implementation, the point cloud projection method is adopted to project the coordinates of the four vertices of the text image in the face coordinate system onto the camera imaging surface, and the coordinates of the four vertices of the text image in the video frame are obtained. The coordinates of these four vertices in the video frame constitute a point combination A.

[0196] In practical applications, the three-dimensional coordinates of the text image in the camera coordinate system are calculated based on the face posture; then the three-dimensional coordinates of the text image in the camera coordinate system are projected onto the camera imaging surface.

[0197] Step 804: Calculate the transformation matrix for transforming the text image onto the camera imaging plane.

[0198] In actual implementation, the original two-dimensional coordinates of the four vertices of the text image are the upper left corner (0, 0), the lower left corner (0, h-1), the lower right corner (w-1, h-1), and the upper right corner (w-1, 0). The original two-dimensional coordinates of these four vertices form a point combination B. Here, the transformation matrix M of the point combination B to the point combination A is calculated.

[0199] Step 805: Use the transformation matrix to perform affine transformation on the text image to obtain a transformed text image.

[0200] Step 806: superimpose the transformed text image onto the video frame to obtain a video frame with information to be displayed added thereto.

[0201] By applying the above embodiment, more accurate in-camera parameters can be obtained, ensuring the effect when the subsequent information to be displayed is added, and improving the accuracy of determining the facial posture; and while determining the facial posture, the three-dimensional coordinates of the facial feature points can also be obtained to obtain the three-dimensional shape of the face.

[0202] The following is a description of the apparatus for processing facial posture in a video provided by an embodiment of the present application. Fig. 9 , Fig. 9 : is a structural diagram of a device for processing facial posture in a video provided by an embodiment of the present application. The device for processing facial posture in a video provided by an embodiment of the present application includes:

[0203] A detection module 901 is used to detect facial feature points on a plurality of video frames containing a target object in a target video, and obtain a first feature point set including two-dimensional coordinates of facial feature points;

[0204] A reconstruction module 902 is used to perform three-dimensional reconstruction on the target object based on the first feature point set to obtain the face posture of the target object and the three-dimensional coordinates of the face feature points of the target object;

[0205] A projection module 903 is used to project the three-dimensional coordinates of the facial feature points to obtain a second feature point set including the two-dimensional coordinates of the facial feature points;

[0206] The comparison module 904 is used to compare the second feature point set with the first feature point set. When the error between the first feature point set and the second feature point set does not meet the error condition, the facial posture obtained by the three-dimensional reconstruction is used as the target facial posture of the target object in the target video.

[0207] In some embodiments, the detection module 901 is further used to capture the face region of the target object from multiple video frames containing the target object in the target video to obtain multiple face region images;

[0208] Performing facial feature point detection on each face region image respectively to obtain the two-dimensional coordinates of the facial feature points in each face region image;

[0209] Performing coordinate transformation on the two-dimensional coordinates of the facial feature points in the facial region image to obtain the two-dimensional coordinates of the facial feature points in each video frame;

[0210] The two-dimensional coordinates of the facial feature points in each video frame are combined into a first feature point set.

[0211] In some embodiments, the detection module 901 is further used to perform face re-recognition on the target video to obtain the face track of the target object;

[0212] The detecting of facial feature points on a plurality of video frames containing a target object in a target video comprises:

[0213] Determining, according to the face trajectory of the target object, a plurality of video frames in the target video containing the target object and the face position of the target object in the video frames;

[0214] Perform facial feature point detection on the face located at the face position in the video frame.

[0215] In some embodiments, the detection module 901 is further used to perform face re-recognition on the target video, determine the video frame in which the target object is recognized as the video frame containing the target object, and determine the face position of the target object in the video frame;

[0216] For a first video frame and a second video frame containing a target object, when the target object is not recognized in at least one third video frame between the first video frame and the second video frame, and the number of the third video frames is less than a first number threshold, obtaining a first face position of the target object in the first video frame and a second face position of the target object in the second video frame;

[0217] When the distance between the first face position and the second face position is less than a distance threshold, determining that the third video frame contains a target object, and

[0218] An interpolation process is performed according to the first face position and the second face position to obtain a face position of the target object in at least one of the third video frames to generate a face trajectory of the target object.

[0219] In some embodiments, the detection module 901 is further used to perform face re-recognition on multiple video frames in the target video, determine the video frames that recognize the target object and the video frames that do not contain the target object, and determine the face position of the target object in the video frames;

[0220] For a first video frame and a second video frame containing a target object, when the target object is not recognized in at least one third video frame between the first video frame and the second video frame, and the number of the third video frames reaches a second number threshold, based on the first face position of the target object in the first video frame and the second face position of the target object in the second video frame, generate at least two segments of two face trajectories of the target object.

[0221] In some embodiments, the reconstruction module 902 is further used to estimate the face pose based on the first feature point set to obtain the face pose of the target object;

[0222] Based on the facial posture of the target object, the target object is three-dimensionally reconstructed through triangulation processing to determine the three-dimensional coordinates of the facial feature points of the target object.

[0223] In some embodiments, the reconstruction module 902 is further used to obtain a basic matrix based on the position relationship of each facial feature point in the first feature point set in the plurality of video frames;

[0224] Normalizing the basic matrix to obtain an essential matrix;

[0225] Performing singular value decomposition on the essential matrix to obtain the facial posture of the target object.

[0226] In some embodiments, the reconstruction module 902 is further used to obtain the target camera intrinsic parameters of the target video;

[0227] Based on the target camera intrinsic parameters and the first feature point set, three-dimensionally reconstruct the target object;

[0228] In the process of performing three-dimensional reconstruction on the target object, obtaining the camera intrinsic parameters of the target video;

[0229] When the error between the first feature point set and the second feature point set satisfies the error condition, the target camera intrinsic parameter is updated based on the acquired camera intrinsic parameter, so that

[0230] Based on the updated target camera intrinsic parameters and the first feature point set, the target object is three-dimensionally reconstructed.

[0231] In some embodiments, the reconstruction module 902 is further used to obtain a basic matrix and an essential matrix during the three-dimensional reconstruction of the target object;

[0232] According to the conversion relationship between the basic matrix and the essential matrix, the camera intrinsic parameters of the target video are obtained.

[0233] In some embodiments, the reconstruction module 902 is further configured to, when the error between the first feature point set and the second feature point set does not satisfy the error condition and the number of facial feature points in each of the video frames is multiple, for each target facial feature point in the multiple facial feature points, remove the two-dimensional coordinates of at least one facial feature point from the first feature point set to obtain a third feature point set;

[0234] Based on the third feature point set, the target object is three-dimensionally reconstructed to obtain the three-dimensional coordinates of the facial feature points of the target object;

[0235] Projecting the three-dimensional coordinates obtained by three-dimensional reconstruction based on the third feature point set onto the camera imaging surface to obtain a fourth feature point set corresponding to the third feature point set and including the two-dimensional coordinates of the facial feature points;

[0236] The first feature point set is updated based on the third feature point set and the fourth feature point set.

[0237] In some embodiments, the projection module 903 is further used to rotate and translate the three-dimensional coordinates of the facial feature points with respect to the camera imaging plane according to the facial posture, so as to obtain the coordinates of the facial feature points in the camera coordinate system;

[0238] Obtaining target camera intrinsic parameters corresponding to the target video;

[0239] According to the target camera intrinsic parameters, the coordinates in the camera coordinate system are converted into coordinates in the image coordinate system to obtain a second feature point set including two-dimensional coordinates of facial feature points.

[0240] In some embodiments, the device further comprises: a processing module, configured to generate a picture corresponding to the information to be displayed based on the information to be displayed;

[0241] Obtaining original two-dimensional coordinates of multiple vertices of the image and three-dimensional coordinates of multiple vertices of the image;

[0242] Based on the target face posture, projecting the three-dimensional coordinates of multiple vertices of the image onto a camera imaging surface to obtain the two-dimensional coordinates of multiple vertices of the image;

[0243] Determine a perspective transformation matrix according to a mapping relationship between the original two-dimensional coordinates and the two-dimensional coordinates obtained by projection;

[0244] Performing an affine transformation on the image through a perspective transformation matrix to obtain a transformed target image;

[0245] The target image is superimposed on the video frame to obtain a video frame with the information to be displayed added thereto.

[0246] By applying the above embodiment, a second feature point set is obtained by projection, and the second feature point set is compared with the first feature point set, thereby realizing the verification of the three-dimensional reconstruction result and improving the stability and accuracy of the target face posture determination.

[0247] The present application also provides a computer device, which may be a terminal or a server. Fig.10 , Fig.10 : is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device provided in an embodiment of the present application includes:

[0248] Memory 550, used to store executable instructions;

[0249] The processor 510 is used to implement the information display method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0250] Here, the processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0251] The memory 550 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 510.

[0252] The memory 550 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0253] In some embodiments, at least one network interface 520 and a user interface 530 may also be included. The various components in the computer device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 540 is not described in detail. Fig.10 Various buses are labeled as bus system 540 .

[0254] The embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for processing facial gestures in the video described in the embodiment of the present application.

[0255] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the method for processing facial posture in a video provided by the embodiment of the present application.

[0256] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EP ROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0257] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0258] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (for example, files storing one or more modules, subroutines, or code portions).

[0259] By way of example, executable instructions may be deployed to be executed on one computer device, or on multiple computer devices located at one site, or on multiple computer devices distributed across multiple sites and interconnected by a communication network.

[0260] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for processing facial posture in a video, characterized in that: include: Performing facial feature point detection on a plurality of video frames containing a target object in a target video to obtain a first feature point set including two-dimensional coordinates of the facial feature points; Obtaining target camera intrinsic parameters of the target video; Based on the target camera intrinsic parameters and the first feature point set, the target object is three-dimensionally reconstructed to obtain the face posture of the target object and the three-dimensional coordinates of the face feature points of the target object; Projecting the three-dimensional coordinates of the facial feature points to obtain a second feature point set including the two-dimensional coordinates of the facial feature points, and obtaining the camera intrinsic parameters of the target video during the three-dimensional reconstruction of the target object; comparing the second feature point set with the first feature point set, and when an error between the first feature point set and the second feature point set does not satisfy an error condition, updating the target camera intrinsic parameters based on the acquired camera intrinsic parameters, and performing three-dimensional reconstruction of the target object based on the updated target camera intrinsic parameters and the first feature point set; When the error between the first feature point set and the second feature point set satisfies an error condition, the facial posture obtained by the three-dimensional reconstruction is used as the target facial posture of the target object in the target video.

2. The method according to claim 1, characterized in that The method of detecting facial feature points on a plurality of video frames containing a target object in a target video to obtain a first feature point set including two-dimensional coordinates of facial feature points comprises: From a plurality of video frames containing a target object in a target video, a face region of the target object is intercepted to obtain a plurality of face region images; Performing facial feature point detection on each face region image respectively to obtain the two-dimensional coordinates of the facial feature points in each face region image; Performing coordinate transformation on the two-dimensional coordinates of the facial feature points in the facial region image to obtain the two-dimensional coordinates of the facial feature points in each video frame; The two-dimensional coordinates of the facial feature points in each video frame are combined into a first feature point set.

3. The method according to claim 1, characterized in that Before detecting facial feature points of a plurality of video frames containing a target object in a target video, the method further includes: Performing face re-recognition on the target video to obtain a face trajectory of the target object; The detecting of facial feature points on a plurality of video frames containing a target object in a target video comprises: Determining, according to the face trajectory of the target object, a plurality of video frames in the target video containing the target object and the face position of the target object in the video frames; Perform facial feature point detection on the face located at the face position in the video frame.

4. The method according to claim 3, characterized in that The step of performing face re-recognition on the target video to obtain a face track of the target object includes: Performing face re-recognition on the target video, determining the video frame in which the target object is recognized as the video frame containing the target object, and determining the face position of the target object in the video frame; For a first video frame and a second video frame containing a target object, when the target object is not recognized in at least one third video frame between the first video frame and the second video frame, and the number of the third video frames is less than a first number threshold, obtaining a first face position of the target object in the first video frame and a second face position of the target object in the second video frame; When the distance between the first face position and the second face position is less than a distance threshold, determining that the third video frame contains a target object, and An interpolation process is performed according to the first face position and the second face position to obtain a face position of the target object in at least one of the third video frames to generate a face trajectory of the target object.

5. The method according to claim 3, characterized in that The step of performing face re-recognition on the target video to obtain a face track of the target object includes: Performing face re-recognition on multiple video frames in the target video, determining a video frame in which a target object is recognized as a video frame containing the target object, and determining a face position of the target object in the video frame; For a first video frame and a second video frame containing a target object, when the target object is not recognized in at least one third video frame between the first video frame and the second video frame, and the number of the third video frames reaches a second number threshold, at least two face trajectories of the target object are generated based on the first face position of the target object in the first video frame and the second face position of the target object in the second video frame.

6. The method according to claim 1, characterized in that The method further comprises: Performing facial pose estimation based on the first feature point set to obtain the facial pose of the target object; Based on the facial posture of the target object, the target object is three-dimensionally reconstructed through triangulation processing to determine the three-dimensional coordinates of the facial feature points of the target object.

7. The method according to claim 6, characterized in that Performing facial pose estimation based on the first feature point set to obtain the facial pose of the target object includes: Based on the positional relationship of each facial feature point in the first feature point set in the plurality of video frames, obtaining a basic matrix; Normalizing the basic matrix to obtain an essential matrix; Performing singular value decomposition on the essential matrix to obtain the facial posture of the target object.

8. The method according to claim 7, characterized in that In the process of performing three-dimensional reconstruction on the target object, obtaining the camera internal parameters of the target video includes: In the process of three-dimensional reconstruction of the target object, a basic matrix and an essential matrix are obtained; According to the conversion relationship between the basic matrix and the essential matrix, the camera intrinsic parameters of the target video are obtained.

9. The method according to claim 1, characterized in that The method further comprises: When the error between the first feature point set and the second feature point set does not satisfy the error condition, and the number of facial feature points in each of the video frames is multiple, removing the two-dimensional coordinates of at least one facial feature point from the first feature point set to obtain a third feature point set; Based on the third feature point set, the target object is three-dimensionally reconstructed to obtain the three-dimensional coordinates of the facial feature points of the target object; Projecting the three-dimensional coordinates obtained by three-dimensional reconstruction based on the third feature point set onto the camera imaging plane to obtain a fourth feature point set including the two-dimensional coordinates of the facial feature points; The first feature point set is updated based on the error between the third feature point set and the fourth feature point set.

10. The method according to claim 1, characterized in that The projecting of the three-dimensional coordinates of the facial feature points to obtain a second feature point set including the two-dimensional coordinates of the facial feature points includes: According to the facial posture, the three-dimensional coordinates of the facial feature points are rotated and translated with respect to the camera imaging plane to obtain the coordinates of the facial feature points in the camera coordinate system; Obtaining target camera intrinsic parameters corresponding to the target video; According to the target camera intrinsic parameters, the coordinates in the camera coordinate system are converted into coordinates in the image coordinate system to obtain a second feature point set including two-dimensional coordinates of facial feature points.

11. The method according to claim 1, characterized in that Also includes: Based on the information to be displayed, generating a picture corresponding to the information to be displayed; Obtaining original two-dimensional coordinates of multiple vertices of the image and three-dimensional coordinates of multiple vertices of the image; Based on the target face posture, projecting the three-dimensional coordinates of multiple vertices of the image onto a camera imaging surface to obtain the two-dimensional coordinates of multiple vertices of the image; Determine a perspective transformation matrix according to a mapping relationship between the original two-dimensional coordinates and the two-dimensional coordinates obtained by projection; Performing an affine transformation on the image using the perspective transformation matrix to obtain a transformed target image; The target image is superimposed on the video frame to obtain a video frame with the information to be displayed added thereto.

12. The method according to claim 1, characterized in that The method further comprises: Obtaining a mean square error between the second feature point set and the first feature point set; When a mean square error between the second feature point set and the first feature point set is less than an error threshold, determining that the error between the first feature point set and the second feature point set meets an error condition; When the mean square error between the second feature point set and the first feature point set is greater than or equal to the error threshold, it is determined that the error between the first feature point set and the second feature point set does not satisfy the error condition.

13. A device for processing facial posture in a video, characterized in that: include: A detection module, configured to detect facial feature points on a plurality of video frames containing a target object in a target video, and obtain a first feature point set including two-dimensional coordinates of the facial feature points; A reconstruction module, used to obtain the target camera intrinsic parameters of the target video; based on the target camera intrinsic parameters and the first feature point set, perform three-dimensional reconstruction on the target object to obtain the face posture of the target object and the three-dimensional coordinates of the face feature points of the target object, and obtain the camera intrinsic parameters of the target video during the process of performing three-dimensional reconstruction on the target object; A projection module, used for projecting the three-dimensional coordinates of the facial feature points to obtain a second feature point set including the two-dimensional coordinates of the facial feature points; A comparison module is used to compare the second feature point set with the first feature point set, and when the error between the first feature point set and the second feature point set does not meet the error condition, update the target camera intrinsic parameters based on the acquired camera intrinsic parameters, and perform three-dimensional reconstruction on the target object based on the updated target camera intrinsic parameters and the first feature point set; when the error between the first feature point set and the second feature point set meets the error condition, use the facial posture obtained by the three-dimensional reconstruction as the target facial posture of the target object in the target video.

14. A computer device, characterized in that: include: A memory for storing executable instructions; A processor, configured to implement the method for processing facial posture in a video as described in any one of claims 1 to 12 when executing the executable instructions stored in the memory.

15. A computer-readable storage medium, characterized in that: Executable instructions are stored, which are used to implement the method for processing facial posture in a video as described in any one of claims 1 to 12 when executed by a processor.

16. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method for processing facial posture in a video as described in any one of claims 1 to 12 is implemented.

Images