Head posture method and device based on video stream

Abstract

The present invention discloses a head posture method based on video stream, comprising: S1: obtaining the original image of the current frame to be tested containing a face in an input video file; S2: detecting and intercepting the face area in the original image to obtain an initial face image; S3: inputting the initial face image into an ultra-lightweight face key point model to obtain an aligned face image; S4: inputting the aligned face image into a face judgment model to determine whether the aligned face image contains a face; S5: inputting the aligned face image into a face key point contour model to obtain the position coordinates of a high-definition face detection frame and a head posture result; S6: obtaining the original image of the next adjacent frame, and intercepting the initial face image of the next adjacent frame image based on the position coordinates of the high-definition face detection frame of the current frame image. The head posture method of the present application performs a face detection on the original image, calculates the similarity transformation matrix and its rotation angle, and improves the speed of the head posture algorithm.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a head posture method and device based on video stream. Background Art

[0002] With the rapid development of deep learning and artificial intelligence technologies, the demand for digital and intelligent information is growing. Head pose estimation, as a form of human feature analysis, lies at the intersection of face tracking and pose recognition, and holds great research value and significance. While humans are capable of directly observing and inferring a person's head pose, computers, which acquire three-dimensional spatial information from two-dimensional images, are not so easy to predict. Head pose estimation typically involves several steps: face detection, face alignment, facial landmark detection, and head pose estimation. Each step is crucial and determines the final result. The higher the accuracy of each module, the better the final pose estimation. While deep learning technology is becoming increasingly mature, the models commonly used in academic research often have large parameter counts, high computational load, and slow inference speed.

[0003] In the existing head posture algorithm, face tracking requires face detection for each frame of the video. The face position in the image is obtained through face detection. The face image is cropped from the original image and then fed into the face contour key point model to obtain the key points of the face, such as Figure 1 As shown in the figure. Obtaining facial key points using this method requires repeated fine-tuning of the face detection and facial contour key point models, resulting in poor real-time performance and poor results. Because the detected face moves in real-time, the face position also changes in real time. Typically, the face position detected by the face detector varies from frame to frame, making it difficult to obtain an accurate and stable face position. In the past, improving face detection accuracy could not avoid errors between previous and subsequent frames, and weighted smoothing of the face positions between previous and subsequent frames resulted in tracking delays, resulting in unsatisfactory results.

[0004] Therefore, finding a method that can stabilize face detection while improving its accuracy is a technical problem that needs to be solved urgently by those skilled in the art. This paper designs a tracking algorithm to solve the problem of head pose estimation stability, considering both accuracy and speed. Summary of the Invention

[0005] The purpose of the present invention is to provide a high-precision, low-latency head posture method based on video stream, which can solve the stability and robustness of face tracking in real-time scenarios with limited computing power and greatly reduce the jitter of face tracking, thereby obtaining a stable and accurate head posture estimation value.

[0006] To achieve the above object, the present invention provides a head posture method based on video stream, comprising the following steps:

[0007] S1. Obtain the original image of the current frame to be tested containing the face in the input video file;

[0008] S2 detects and captures the face area in the original image to obtain an initial face image;

[0009] S3 inputs the initial face image into the ultra-lightweight face key point model, aligns the initial face image to obtain an aligned face image;

[0010] S4. Sending the aligned face image to the face recognition model to determine whether the aligned face image contains a face, if so, proceed to the next step, if not, return to step S1;

[0011] S5. Input the aligned face image into the face key point contour model, perform high-definition processing on the aligned face image, and obtain the position coordinates of the high-definition face detection frame and the head posture result;

[0012] S6 obtains the original image of the next adjacent frame in the video file, and captures the initial face image of the next adjacent frame according to the position coordinates of the high-definition face detection frame of the current frame image;

[0013] Repeat steps S3-S6 to output the high-definition face detection frame and head posture results of the subsequent consecutive frame images in sequence.

[0014] In one embodiment of the present invention, step S2 specifically includes the following contents:

[0015] S21. Detect the original image and obtain the position coordinates [l, t, r, b] of a first face detection frame, where (l, t) are the horizontal and vertical coordinates of the upper left corner of the first face detection frame on the original image, and (r, b) are the horizontal and vertical coordinates of the lower right corner of the first face detection frame on the original image;

[0016] S22. Proportionally enlarge the original image to obtain an enlarged image, detect the face region in the enlarged image, and obtain the position coordinates of a second face detection frame [l', t', r', b'], where (l', t') are the horizontal and vertical coordinates of the upper left corner of the second face detection frame on the enlarged image, and (r', b') are the horizontal and vertical coordinates of the lower right corner of the second face detection frame on the enlarged image;

[0017] S23. Use the second face detection frame to capture the face portion in the enlarged image to obtain an initial face image, where the height of the initial face image is b'-t' and the width is r'-l'.

[0018] In one embodiment of the present invention, step S3 specifically includes the following contents:

[0019] S31. Input the initial face image into the ultra-lightweight face key point model, and output the first low-precision face key point coordinate set [x′0, y′0, x′1, y′1, ... x′ n-1 , y′ n-1 ], where n represents the total number of facial key points, (x′0, y′0) represents the horizontal and vertical coordinates of the first facial key point, (x′ n-1 , y′ n-1 ) represents the horizontal and vertical coordinates of the nth facial key point;

[0020] S32. Obtain a second low-precision facial key point coordinate set [x″0, y″0, x″1, y″1, ...x″] of the face on the original image by adding the horizontal coordinate and vertical coordinate of each key point in the first low-precision facial key point coordinate set to the horizontal coordinate and vertical coordinate of the upper left corner of the second face detection frame on the original image. n-1 , y″ n-1 ];

[0021] S33. By operating the second low-precision facial key point coordinate set with the preset standard facial key point coordinate set, a similarity transformation matrix M1 is obtained. where R 00 =R 11 , R 01 =-R 10 , R 00 、R 01 、R 10 、R 11 Represents the rotation and scaling coefficients of the similarity transformation matrix, T x 、T y Represents the offset coefficient of the similarity transformation matrix;

[0022] S34. Based on the similarity transformation matrix M1 and the original image, align the initial face image to obtain an aligned face image.

[0023] In one embodiment of the present invention, step S5 specifically includes the following contents:

[0024] S51. Inputting the aligned face image into the face key point contour model, outputting a first high-precision face key point coordinate set and a head posture estimation result;

[0025] S52. Based on the inverse matrix of the similarity transformation matrix M1, the coordinates of each key point in the first high-precision facial key point coordinate set and the head posture estimation result are inversely transformed to obtain a second high-precision facial key point coordinate set and a rotation angle of the similarity transformation matrix of the face on the original image;

[0026] S53. Determine the position coordinates of the third face detection frame on the original image by calculating the maximum and minimum values ​​of the horizontal and vertical coordinates in the second high-precision facial key point coordinate set;

[0027] S54. Enlarge the original image proportionally, and calculate the position coordinates of the high-definition face detection frame and the head posture result through the position coordinates of the third face detection frame.

[0028] In one embodiment of the present invention, step S6 specifically includes the following contents:

[0029] Obtain the original image of the adjacent next frame, use the position coordinates of the high-definition face detection frame in the current frame image as the position coordinates of the second detection frame of the adjacent next frame image, and intercept the initial face image of the adjacent next frame image based on the position coordinates of the second detection frame of the adjacent next frame image.

[0030] The present application also provides a head posture device based on video stream, comprising:

[0031] An acquisition module is used to obtain the original image of the current frame to be tested containing the face;

[0032] A detection module, configured to detect a face region in the original image;

[0033] An alignment module, configured to align the face region to obtain an aligned face image;

[0034] A judgment module, configured to judge whether the aligned face image contains a face;

[0035] A determination module, configured to determine a high-definition face position and head posture result in the original image;

[0036] The tracking module is used to obtain the next frame image of the current frame and obtain the initial face image in the next frame image according to the high-definition face position in the original image of the current frame.

[0037] In one embodiment of the present invention, it further comprises:

[0038] an enlargement module, configured to enlarge the original image by a predetermined multiple to obtain a face region on the enlarged image;

[0039] The interception module is used to intercept the face area from the enlarged image to obtain an initial face image.

[0040] In one embodiment of the present invention, the alignment module includes:

[0041] An extraction submodule, configured to extract facial key point information of the facial area;

[0042] A calculation submodule, configured to calculate a similarity transformation matrix for achieving face alignment using the facial key point information;

[0043] The alignment submodule is used to align the face region using a similarity transformation matrix.

[0044] The present invention also provides a computer device comprising a memory and a processor; the memory stores a computer program that can be run by the processor; when the processor runs the computer program, it executes the steps of any one of the above-mentioned video stream-based head posture methods.

[0045] The present invention also provides a computer-readable storage medium comprising a program code. When the program code is run on an electronic device, the program code is used to enable the electronic device to execute any of the steps of the video stream-based head gesture method described above.

[0046] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0047] 1. This application uses an ultra-lightweight facial key point model to extract low-precision facial key points in the original image, and uses the similarity transformation matrix calculated by the extracted low-precision facial key point coordinates and the standard facial key point coordinates to obtain a face alignment image in the original image, and uses a face judgment model to judge whether the original image contains a face and whether face detection needs to be re-enabled, and then uses the facial key point contour model to perform high-definition processing on the aligned face image, thereby improving the accuracy of face tracking and reducing time consumption.

[0048] 2. This application uses the face position and head posture results obtained by face detection, ultra-lightweight face key point detection, similarity transformation, and high-precision face key point detection in the current frame image. After amplification, it is used as the initial face position of the adjacent next frame image, and then ultra-lightweight face key point detection, similarity transformation, and high-precision face key point detection are used to form a complete face tracking model. The face key points of the next frame image of this application are obtained through calculation. It is only necessary to detect the face position in one frame image, and there is no need to perform face detection on each frame image. Subsequent face tracking no longer relies on face detection, preventing the face from restarting face detection or further processing images that do not contain faces during rapid movement, thereby causing delays in the face key point effect and affecting the processing speed of the head posture algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a head posture algorithm in the prior art;

[0050] Figure 2 is a flow chart of a head posture method according to an embodiment of the present invention;

[0051] Figure 3 yes Figure 2 Specific flow chart of step S2;

[0052] Figure 4 yes Figure 2 Specific flow chart of step S3;

[0053] Figure 5 yes Figure 2 Specific flow chart of step S5;

[0054] Figure 6 is a schematic diagram of a head posture device according to one embodiment of the present invention.

[0055] Reference numerals:

[0056] 100, acquisition module; 200, detection module; 300, alignment module; 400, judgment module; 500, determination module; 600, tracking module. DETAILED DESCRIPTION

[0057] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings, but it should be understood that the protection scope of the present invention is not limited by the specific embodiments.

[0058] Unless expressly stated otherwise, throughout the specification and claims, the term "comprise" or variations such as "include" or "comprising", etc., will be understood to include the stated elements or components but not to exclude other elements or other components.

[0059] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating directions or positional relationships, are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or components referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the present invention; the terms "first", "second", and "third" are only used for descriptive purposes and should not be understood as indicating or implying relative importance. In addition, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, an indirect connection through an intermediate medium, or it can be internal communication between two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0060] like Figure 2-6 As shown, a preferred embodiment of the present invention provides a head posture algorithm based on video stream, comprising the following steps:

[0061] S1. Obtain the original image of the current frame to be tested containing the face;

[0062] S2 detects and captures the face area in the original image to obtain an initial face image;

[0063] S3 inputs the initial face image into the ultra-lightweight face key point model, aligns the initial face image to obtain an aligned face image;

[0064] S4. Sending the aligned face image to the face recognition model to determine whether the aligned face image contains a face, if so, proceed to the next step, if not, return to step S1;

[0065] S5. Input the aligned face image into the face key point contour model, perform high-definition processing on the aligned face image, and obtain the position coordinates of the high-definition face detection frame and the head posture result;

[0066] S6 obtains the original image of the next adjacent frame in the video file, and captures the initial face image of the next adjacent frame according to the position coordinates of the high-definition face detection frame of the current frame image;

[0067] Repeat steps S3-S6 to output the high-definition face detection frame and head posture results of the subsequent consecutive frame images in sequence.

[0068] In an embodiment of the present application, a head posture algorithm based on a video stream is used to detect the original image of any frame in the real-time camera data. The original image is an image of the facial key points to be determined. These images can be, for example, images to be used to add facial special effects or to perform facial AR operations. In addition, the image to be detected can be a real-time image taken by a terminal device, or a video frame image extracted from an existing video. The present disclosure does not impose any special restrictions on the resolution, image background, facial posture, number of faces, image color temperature, etc. of the image to be detected.

[0069] Here, facial landmarks refer to information that reflects facial features, such as eyes, eyebrows, nose, mouth, and facial contours. Facial landmark coordinates refer to the coordinates of these facial landmarks. The coordinates of each facial landmark can be represented by a two-dimensional array, for example, (x1, y1) or (x2, y2).

[0070] like Figure 3 As shown, in a further embodiment of the present application, step S2 specifically includes the following contents:

[0071] S21. Detect the original image and obtain the position coordinates of the first face detection frame [l, t, r, b], where (l, t) are the horizontal and vertical coordinates of the upper left corner of the first face detection frame on the original image, and (r, b) are the horizontal and vertical coordinates of the lower right corner of the first face detection frame on the original image.

[0072] S22. Scale up the original image to obtain an enlarged image, detect the face area in the enlarged image, and obtain the position coordinates of the second face detection frame [l', t', r', b'], where (l', t') are the horizontal and vertical coordinates of the upper left corner of the second face detection frame on the enlarged image, and (r', b') are the horizontal and vertical coordinates of the lower right corner of the second face detection frame on the enlarged image.

[0073] In this embodiment, the magnification ratio can be interpolated on the original image according to the magnification factor selected by the user (for example, the optional magnification factors include 1x, 2x, 4x, etc.), thereby magnifying the original image to the corresponding size. Optionally, the embodiment of the present application uses bilinear interpolation. While the original image is magnified into the first image, the facial key point information in the first image is also magnified to the corresponding size, thereby obtaining the facial key point information in the first image.

[0074] S23. Use the second face detection frame to capture the face portion in the enlarged image to obtain an initial face image. The height of the initial face image is b'-t', and the width is r'-l'.

[0075] S3. Input the initial face image into the ultra-lightweight face key point model, align the initial face image, and obtain an aligned face image.

[0076] like Figure 4 In a further embodiment of the present application, step S3 specifically includes the following contents:

[0077] S31. Input the initial face image into the ultra-lightweight face key point model and output the first low-precision face key point coordinate set [x′0, y′0, x′1, y′1, …x′ n-1 , y′ n-1 ], where n represents the total number of facial key points, (x′0, y′0) represents the horizontal and vertical coordinates of the first facial key point, (x′ n-1 , y′ n-1 ) represents the horizontal and vertical coordinates of the nth facial key point.

[0078] By inputting the initial facial image into the ultra-lightweight facial landmark model, the facial landmark information in the initial facial image can be obtained. Using the ultra-lightweight facial landmark model to locate the face position greatly improves the accuracy of face tracking, and in actual use, the time consumption is low and the results are very stable.

[0079] S32. Obtain a second low-precision facial key point coordinate set [x″0, y″0, x″1, y″1, …x″] of the face on the original image by adding the horizontal coordinate and vertical coordinate of each key point in the first low-precision facial key point coordinate set to the horizontal coordinate and vertical coordinate of the upper left corner of the second face detection frame on the original image. n-1 , y″ n-1 ].

[0080] S33. By operating the second low-precision facial key point coordinate set with the preset standard facial key point coordinate set, a similarity transformation matrix M1 is obtained. where R 00 =R 11 , R 01 =-R 10 , R 00 、R 01 、R 10 、R 11 Represents the rotation and scaling coefficients of the similarity transformation matrix, T x 、T y Represents the offset coefficient of the similarity transformation matrix; wherein the standard facial key point coordinate set is a preset facial key point coordinate set used as a reference base value.

[0081] S34. Based on the similarity transformation matrix M1 and the original image, the initial face image is aligned to obtain an aligned face image.

[0082] S4. The aligned face image is fed into the face recognition model to determine whether it contains a face. If so, the process proceeds to the next step; if not, the process returns to step S2. In this step, the face recognition model determines whether the aligned face image contains a face, and then determines whether to track the face. If the aligned face image does contain a face, tracking is maintained. If not, face detection and tracking is terminated, and face detection is re-enabled on the original image.

[0083] S5. Input the aligned face image into the face key point contour model, perform high-definition processing on the aligned face image, and obtain the position coordinates of the high-definition face detection frame and the head posture results.

[0084] like Figure 5 As shown, in a further embodiment of the present application, step S5 specifically includes the following steps:

[0085] S51. Input the aligned face image into the face key point contour model, and output the first high-precision face key point coordinate set [x″′0, y″′0, x″′1, y″′1, …x″′ n-1 , y″′ n-1 ] and the head posture estimation result (x, y, z), where x is the rotation angle of the face head posture around the X axis, y is the rotation angle of the face head posture around the Y axis, and z is the rotation angle of the face head posture around the Z axis.

[0086] S52. Based on the inverse matrix of the similarity transformation matrix M1, the coordinates of each key point in the first high-precision facial key point coordinate set are inversely transformed to obtain a second high-precision facial key point coordinate set [x0, y0, x1, y1, ...x n-1 ,y n-1 ] and the rotation angle θ of the similarity transformation matrix.

[0087] S53. Determine the position coordinates of the third face detection frame on the original image by calculating the maximum and minimum values ​​of the horizontal and vertical coordinates in the second high-precision facial key point coordinate set.

[0088] S54. Scale up the original image, and calculate the position coordinates of the high-definition face detection frame and the head posture result (x, y, Z) using the position coordinates of the third face detection frame, where Z = z + θ, and Z is the rotation angle of the head posture of the high-definition face detection frame around the Z axis.

[0089] S6 obtains the original image of the next adjacent frame, and captures the initial face image of the next adjacent frame according to the position coordinates of the high-definition face detection frame of the current frame image;

[0090] Repeat steps S3-S6 to output high-definition face detection frames for subsequent frames. Using the coordinates of the high-definition face detection frame in the previous frame as the coordinates of the second detection frame in the next frame, and capturing the initial face image of the next frame, eliminates the need to perform face detection on every frame, improving head posture efficiency.

[0091] This application conducts a face tracking comparison test by using different algorithms on the same imaging device. The experimental results are shown in Table 1.

[0092]

[0093] Table 1 is a comparison table of the head posture algorithm of the embodiment of the present application and the original head posture algorithm

[0094] Among them, the existing algorithm is to crop the face image from the original image and then feed it into the face contour key point model to obtain the face key points and head posture results, such as Figure 1 shown.

[0095] The comparison algorithm performs face detection on the original image to obtain a face image, and sends the face image to the face judgment model to determine whether the initial image contains a face. If not, the tracking is exited and the face detection is performed again on the original image; if it contains a face, the tracking state is maintained, and the initial image containing the face is sent to the face contour model. Among them, the specific processing steps of the face detection, face judgment model and face contour model in the comparison algorithm are the same as the head posture algorithm in the embodiment of the present application, and will not be repeated here.

[0096] It can be seen from the comparison data that compared with the existing algorithms and comparison algorithms, the head posture algorithm of the present application has a small average total time consumption and high stability, and has a better effect of reducing face tracking jitter and improving head posture stability.

[0097] The training data in the embodiments of the present application are all private data, which are obtained through semi-automatic labeling, manual detection and manual labeling.

[0098] This application uses the face position and head posture results obtained by face detection, ultra-lightweight face key point detection, similarity transformation, and high-precision face key point detection in the current frame image. After amplification, it is used as the initial face position of the adjacent next frame image, and then ultra-lightweight face key point detection, similarity transformation, and high-precision face key point detection are used to form a complete model for face tracking. The face key points of the next frame image of this application are obtained through calculation. It is only necessary to detect the face position in one frame image, and there is no need to perform face detection on each frame image. Subsequent face tracking no longer relies on face detection, preventing the face from restarting face detection or further processing images that do not contain faces during rapid movement, thereby causing delays in the face key point effect and affecting the speed of head posture algorithm processing.

[0099] like Figure 6 As shown, a preferred embodiment of the present application further provides a head posture device based on video stream, comprising:

[0100] The acquisition module 100 is used to acquire the original image of the current frame to be tested containing the face;

[0101] A detection module 200 is configured to detect a face region in the original image;

[0102] An alignment module 300 is configured to align the face region to obtain an aligned face image;

[0103] A determination module 400 is configured to determine whether the aligned face image contains a face;

[0104] A determination module 500 is used to determine the high-definition face position and head posture result in the original image;

[0105] The tracking module 600 is used to obtain the next frame image of the current frame, and obtain the initial face image in the next frame image according to the high-definition face position in the original image of the current frame.

[0106] In a further embodiment of the present application, the head posture device based on video stream further includes:

[0107] an enlargement module, configured to enlarge the original image by a predetermined multiple to obtain a face region on the enlarged image;

[0108] The interception module is used to intercept the face area from the enlarged image to obtain an initial face image.

[0109] In a further embodiment of the present application, the alignment module includes:

[0110] An extraction submodule, configured to extract facial key point information of the facial area;

[0111] A calculation submodule, configured to calculate a similarity transformation matrix for achieving face alignment using the facial key point information;

[0112] The alignment submodule is used to align the face region using a similarity transformation matrix.

[0113] This application obtains the original image of the current frame to be tested containing a face through an acquisition module, and the magnification module magnifies the original image in proportion. The detection module is used to detect the face area in the original image and cut out to obtain an initial face image. The initial face image is input into the ultra-lightweight face key point model in the alignment module, the face key point information of the face area is extracted, the similarity transformation matrix is ​​calculated, and the face area is aligned using the similarity transformation matrix to obtain an aligned face image, and it is determined whether the aligned face image contains a face. If it does not contain a face, the face detection tracking is exited and the original image is re-detected; if it contains a face, the high-definition face position and head posture result in the original image are determined, and the high-definition face position of the current frame is used as the initial face image of the next frame image. There is no need to perform face detection on each frame image, which greatly reduces the time consumption of face detection. Since the initial face position of the next frame image is obtained through the high-definition face position of the previous frame image, it can avoid the situation where the face is moving quickly and the initial face position obtained in the next frame does not contain a complete face or no face at all, thereby restarting the face detection or sending the image containing the incomplete face into the face contour model for further processing, avoiding single-frame jumps in the results or delays in key point effects. The head posture results can be used to determine the rotation angle of the face area in the original image, expand the scope of face detection, improve the speed and accuracy of face detection tracking, solve the angle mutation caused by blinking, and enable face detection to run smoothly on low-end machines with limited computing power.

[0114] A preferred embodiment of the present application further provides a computer device, comprising a memory and a processor; the memory stores a computer program that can be run by the processor; when the processor runs the computer program, it executes the steps of the head posture method based on video stream.

[0115] A preferred embodiment of the present application further provides a computer-readable storage medium, which includes program code. When the program code is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the head gesture method based on video stream.

[0116] As used herein, any reference to memory, storage, database, or other medium may include nonvolatile and / or volatile memory. Suitable nonvolatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0117] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0118] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0119] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0120] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0121] The foregoing descriptions of specific exemplary embodiments of the present invention are for purposes of illustration and description. These descriptions are not intended to limit the invention to the precise forms disclosed, and it is apparent that many variations and modifications are possible in light of the foregoing teachings. The exemplary embodiments have been selected and described for the purpose of explaining the specific principles of the invention and their practical application, thereby enabling those skilled in the art to realize and utilize a variety of exemplary embodiments of the invention and various options and modifications. The scope of the invention is intended to be defined by the claims and their equivalents.

Claims

1. A head posture method based on video stream, characterized in that: The following steps are involved: S1. Obtain the original image of the current frame to be tested containing a face in the input video file; S2 detects and captures the face area in the original image to obtain an initial face image; S3 inputs the initial face image into the ultra-lightweight face key point model, aligns the initial face image to obtain an aligned face image; S4. The aligned face image is fed into the face recognition model to determine whether the aligned face image contains a face. If so, proceed to the next step, if not, return to step S1; S5. Input the aligned face image into the face key point contour model, perform high-definition processing on the aligned face image to obtain the position coordinates of the high-definition face detection frame and the head posture results; Step S5 specifically includes the following contents: S51. Input the aligned face image into the face key point contour model and output the first high-precision face key point coordinate set of the face. And head pose estimation results , where x is the rotation angle of the face head posture around the X axis, y is the rotation angle of the face head posture around the Y axis, and z is the rotation angle of the face head posture around the Z axis; S52. Based on the inverse matrix of the similarity transformation matrix M1, the coordinates of each key point in the first high-precision facial key point coordinate set are inversely transformed to obtain a second high-precision facial key point coordinate set of the face on the original image. and the rotation angle of the similarity transformation matrix ; S53 by calculating the maximum and minimum values ​​of the horizontal and vertical coordinates of the second high-precision facial key point coordinate set, determining the position coordinates of the third face detection frame on the original image; S54. Scale up the original image and calculate the position coordinates of the high-definition face detection frame and the head posture result using the position coordinates of the third face detection frame , where Z= + , Z is the rotation angle of the head posture of the high-definition face detection frame around the Z axis; S6 obtains the original image of the next adjacent frame in the video file, based on the position coordinates of the high-definition face detection frame of the current frame image, captures the initial face image of the next adjacent frame image; Repeat steps S3-S6 to output the high-definition face detection frame and head posture results of the subsequent consecutive frame images in sequence.

2. The head posture method based on video stream according to claim 1, characterized in that The step S2 specifically includes the following contents: S21. Detect the original image and obtain the position coordinates [l, t, r, b] of a first face detection frame, where (l, t) are the horizontal and vertical coordinates of the upper left corner of the first face detection frame on the original image, and (r, b) are the horizontal and vertical coordinates of the lower right corner of the first face detection frame on the original image; S22. Proportionally enlarge the original image to obtain an enlarged image, detect the face region in the enlarged image, and obtain the position coordinates of a second face detection frame [l', t', r', b'], where (l', t') are the horizontal and vertical coordinates of the upper left corner of the second face detection frame on the enlarged image, and (r', b') are the horizontal and vertical coordinates of the lower right corner of the second face detection frame on the enlarged image; S23. Use the second face detection frame to capture the face portion in the enlarged image to obtain an initial face image, where the height of the initial face image is b'-t' and the width is r'-l'.

3. The head posture method based on video stream according to claim 2, characterized in that: The step S3 specifically includes the following contents: S31. Inputting the initial face image into the ultra-lightweight face key point model, outputting the first low-precision face key point coordinate set of the face , where n represents the total number of facial key points, Indicates the horizontal and vertical coordinates of the first facial key point, Represents the horizontal and vertical coordinates of the nth facial key point; S32. Obtain a second low-precision facial key point coordinate set of the face on the original image by adding the horizontal coordinate and vertical coordinate of each key point in the first low-precision facial key point coordinate set to the horizontal coordinate and vertical coordinate of the upper left corner of the second face detection frame on the original image. ; S33. By the operation between the second low-precision face key point coordinate set and the preset standard face key point coordinate set, the similarity transformation matrix M1 is obtained, M1 = ,in = , =- , represents the rotation and scaling coefficients of the similarity transformation matrix, Represents the offset coefficient of the similarity transformation matrix; S34. Based on the similarity transformation matrix M1 and the original image, align the initial face image to obtain an aligned face image.

4. The head posture method based on video stream according to claim 1, characterized in that The step S6 specifically includes the following contents: Obtain the original image of the adjacent next frame, use the position coordinates of the high-definition face detection frame in the current frame image as the position coordinates of the second detection frame of the adjacent next frame image, and intercept the initial face image of the adjacent next frame image based on the position coordinates of the second detection frame of the adjacent next frame image.

5. A head posture device based on video stream, using the head posture method based on video stream according to any one of claims 1 to 4, characterized in that: include: An acquisition module is used to obtain the original image of the current frame to be tested containing the face; A detection module, configured to detect a face region in the original image; An alignment module, configured to align the face region to obtain an aligned face image; A judgment module, configured to judge whether the aligned face image contains a face; a determination module for aligning a facial image input with a facial key point contour model output of a first high-precision facial key point coordinate set of the face, performing an inverse coordinate transformation on the coordinates of each key point in the first high-precision facial key point coordinate set based on an inverse matrix of a similarity transformation matrix, obtaining a second high-precision facial key point coordinate set of the face on the original image, determining the position coordinates of a third face detection frame on the original image by calculating the maximum and minimum values ​​of the horizontal and vertical coordinates in the second high-precision facial key point coordinate set, proportionally amplifying the original image, and calculating the position coordinates of a high-definition face detection frame and a head posture result based on the position coordinates of the third face detection frame to determine the high-definition face position and head posture result in the original image; The tracking module is used to obtain the next frame image of the current frame and obtain the initial face image in the next frame image according to the high-definition face position in the original image of the current frame.

6. The head posture device based on video stream according to claim 5, characterized in that: Also includes: an enlargement module, configured to enlarge the original image by a predetermined multiple to obtain a face region on the enlarged image; The interception module is used to intercept the face area from the enlarged image to obtain an initial face image.

7. The head posture device based on video stream according to claim 5, characterized in that: The alignment module includes: An extraction submodule, configured to extract facial key point information of the facial area; A calculation submodule, configured to calculate a similarity transformation matrix for achieving face alignment using the facial key point information; The alignment submodule is used to align the face region using a similarity transformation matrix.

8. A computer device comprising a memory and a processor; the memory stores a computer program that can be run by the processor; when the processor runs the computer program, it executes the steps of the video stream-based head posture method as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that It includes program code, and when the program code is run on an electronic device, the program code is used to enable the electronic device to execute the steps of the video stream-based head gesture method according to any one of claims 1 to 4.

Images