Three-dimensional face angle labeling method, device, system and storage medium

Abstract

The application provides a three-dimensional face angle labeling method, device, system and storage medium, comprising obtaining continuous frame three-dimensional face point cloud diagrams based on continuous frame face images collected by a depth camera; in response to a selection operation of a user on a frame in the continuous frames, presenting a three-dimensional face point cloud diagram corresponding to the selected frame on a human-computer interaction interface; in response to an adjustment instruction of the user on the presented three-dimensional face point cloud diagram, adjusting an initial posture of a face in the three-dimensional face point cloud diagram to a final posture corresponding to the adjustment instruction; recording a pitch angle adjustment angle, a yaw angle adjustment angle and a roll angle adjustment angle from the initial posture to the final posture to obtain a three-dimensional face angle of the selected frame, realizing construction of a real three-dimensional face point cloud diagram, directly performing angle labeling on the three-dimensional face point cloud diagram, finally obtaining a face angle of the three-dimensional face point cloud diagram, and improving the reliability of a labeling result of the three-dimensional face angle.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically provides a three-dimensional face angle annotation method, device, system, and storage medium. Background Technology

[0002] 3D face angle annotation is a crucial foundational step in 3D face recognition and analysis. Accurate 3D face angle annotation plays a vital role in both scientific research and practical applications, such as face orientation localization, facial expression analysis, face pose analysis, and recognition after face alignment. Therefore, researching and obtaining accurate 3D face angles is of great significance and has always been a popular research topic in fields such as machine vision, image processing, and pattern recognition.

[0003] However, existing research primarily predicts face angles based on a single 2D image. Publicly available datasets such as AFW and AFLW are all 2D face datasets and pose angle data. However, labeling face pose based solely on a 2D face image introduces significant errors. This is because face pose refers to the orientation of a person's face in real 3D space, requiring three angles for description: pitch (rotation about the x-axis), yaw (rotation about the y-axis), and roll (rotation about the z-axis). 2D images lose information about the direction perpendicular to the image plane. Therefore, labeling the angles in 3D space based on a single 2D image is ambiguous and yields unreliable results. Furthermore, most 3D face angle annotation methods in related technologies use a 3D virtual face fitted by a 3DMM. This fitted virtual face is not obtained from real 3D face data obtained through structured light, laser, or other 3D scanning equipment, further complicating the reliability of the 3D face angle annotation results.

[0004] Therefore, obtaining accurate three-dimensional facial angles is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] To overcome the above-mentioned defects, the present invention is proposed to provide a method, device, system and storage medium for three-dimensional face angle annotation that solves or at least partially solves the technical problem of poor reliability of three-dimensional face angle annotation results.

[0006] In a first aspect, the present invention provides a three-dimensional face angle annotation method, comprising:

[0007] A continuous-frame 3D face point cloud map is obtained from continuous-frame face images acquired by a depth camera.

[0008] In response to the user's selection operation of a frame in the continuous frames, the three-dimensional face point cloud map corresponding to the selected frame is presented on the human-computer interaction interface;

[0009] In response to the user's adjustment command for the presented 3D face point cloud map, the initial pose of the face in the 3D face point cloud map is adjusted to the final pose corresponding to the adjustment command; wherein, the pitch angle of the final pose is a first preset angle, the yaw angle of the final pose is a second preset angle, and the roll angle of the final pose is a third preset angle.

[0010] Record the pitch angle adjustment angle, yaw angle adjustment angle, and roll angle adjustment angle from the initial attitude to the final attitude to obtain the three-dimensional face angle of the selected frame.

[0011] Furthermore, in the aforementioned three-dimensional face angle annotation method, obtaining a continuous frame three-dimensional face point cloud map based on continuous frame face images acquired by a depth camera includes:

[0012] Each frame of face depth image is obtained based on each frame of face image captured by a depth camera;

[0013] Based on the focal length and aperture center of the depth camera in the first coordinate axis direction and the second coordinate axis direction, point cloud data of each frame of the face depth image is obtained through coordinate transformation; the pitch angle is the rotation angle about the first coordinate axis, and the yaw angle is the rotation angle about the second coordinate axis;

[0014] The point cloud data of each frame of the face depth image is registered to obtain the continuous frame three-dimensional face point cloud map.

[0015] Furthermore, in the aforementioned three-dimensional face angle annotation method, in response to the user's adjustment command for the presented three-dimensional face point cloud map, the initial pose of the face in the three-dimensional face point cloud map is adjusted to the final pose corresponding to the adjustment command, including:

[0016] Define a first mapping relationship between a first function key of a first input device and an adjustment action that translates and rotates the face pose;

[0017] In response to the user's operation command on the first function key, the initial posture is adjusted to the final posture based on the first mapping relationship.

[0018] Furthermore, in the above-described three-dimensional face angle annotation method, the first input device is a keyboard, and the first function keys are the direction keys in the main key area and the direction keys in the auxiliary key area of ​​the keyboard;

[0019] The step of adjusting the initial posture to the final posture based on the mapping relationship in response to the user's operation command on the first function key includes:

[0020] In response to a touch of the arrow keys in the main key area of ​​the keyboard, the initial posture is translated, wherein the translation distance corresponding to each touch of the arrow keys can be configured;

[0021] In response to a touch of the arrow keys in the auxiliary key area of ​​the keyboard, the initial posture is rotated, wherein the rotation angle corresponding to each touch of the arrow keys can be configured.

[0022] Furthermore, in the aforementioned three-dimensional face angle annotation method, the process of configuring the translation distance corresponding to each touch of the directional keys includes:

[0023] If two adjacent touches of the directional keys in the main key area are in opposite directions, the initial translation distance corresponding to the next touch is reduced based on a preset translation adjustment step size to obtain the adjusted translation distance corresponding to the next touch.

[0024] The process of configuring the rotation angle corresponding to each touch of the arrow keys includes:

[0025] If two adjacent touches of the directional keys in the auxiliary key area are in opposite directions, the initial rotation angle corresponding to the next touch is reduced based on a preset rotation adjustment step size to obtain the adjustment rotation angle corresponding to the next touch.

[0026] Furthermore, in the aforementioned three-dimensional face angle annotation method, before adjusting the initial pose of the face in the three-dimensional face point cloud map to the final pose corresponding to the adjustment command in response to the user's adjustment command, the method further includes:

[0027] Define a second mapping relationship between the second function key of the second input device and the observation action of changing and scaling the viewpoint of the face pose;

[0028] In response to the user's operation command on the second function key, the shape and distribution of the face point cloud under different viewing angles and zoom states are presented based on the second mapping relationship, so that the user can observe the face details.

[0029] Furthermore, in the aforementioned three-dimensional face angle annotation method, the second input device is a mouse, and the second function keys are the left mouse button and the scroll wheel;

[0030] The step of responding to the user's operation command on the second function key and presenting the shape and distribution of the face point cloud under different viewpoints and zoom states based on the second mapping relationship includes:

[0031] In response to the user dragging the three-dimensional face point cloud map with the left mouse button, the shape and distribution of the face point cloud are presented from different perspectives.

[0032] In response to the user zooming in and out on the 3D face point cloud using the mouse wheel, the shape and distribution of the face point cloud are presented under different zoom levels.

[0033] Furthermore, the aforementioned three-dimensional face angle annotation method also includes:

[0034] Based on the three-dimensional face angle of the selected frame corresponding to the selected frame, the three-dimensional face angle of the adjacent frame is obtained through a point cloud matching algorithm.

[0035] Based on the three-dimensional face angles corresponding to the adjacent frames, the three-dimensional face angles corresponding to the adjacent frames are obtained through a point cloud matching algorithm until all consecutive frames are labeled.

[0036] Furthermore, in the above-described 3D face angle annotation method, the step of obtaining the 3D face angles corresponding to adjacent frames through a point cloud matching algorithm based on the 3D face angles of the selected frame corresponding to the selected frame includes:

[0037] The first relative angle between the point cloud map of the selected frame and the point cloud map of the adjacent frame is obtained through the point cloud matching algorithm.

[0038] Based on the first relative angle and the three-dimensional face angle of the selected frame corresponding to the selected frame, the three-dimensional face angle corresponding to the adjacent frame is obtained;

[0039] Based on the 3D face angles corresponding to the adjacent frames, the 3D face angles corresponding to the adjacent frames are obtained through a point cloud matching algorithm, including:

[0040] A second relative angle between the point cloud map of the adjacent frame and the point cloud map of the adjacent frame is obtained through a point cloud matching algorithm.

[0041] Based on the second relative angle and the three-dimensional face angle corresponding to the adjacent frame, the three-dimensional face angle corresponding to the adjacent frame is obtained.

[0042] Furthermore, the aforementioned three-dimensional face angle annotation method also includes:

[0043] In response to the user's selection of frames in a series of frames, the three-dimensional face angle corresponding to the selected frame and the intermediate process information of the annotation are presented on the human-computer interaction interface so that the user can verify the results of automatic annotation.

[0044] In a second aspect, the present invention provides a three-dimensional face angle annotation device, including a processor and a storage device, the storage device being adapted to store a plurality of program codes, the program codes being adapted to be loaded and run by the processor to perform the three-dimensional face angle annotation method described above.

[0045] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored, the program codes being adapted to be loaded and run by a processor to perform the three-dimensional face angle annotation method described in any of the preceding claims.

[0046] In the fourth aspect, a three-dimensional face angle annotation system is provided, including:

[0047] The 3D face angle annotation device described above;

[0048] The first input device is used to receive user adjustment instructions for the presented 3D face point cloud map.

[0049] The above-described technical solutions of the present invention have at least one or more of the following beneficial effects:

[0050] In implementing the technical solution of this invention, after obtaining a continuous frame three-dimensional face point cloud map based on continuous frame face images acquired by a depth camera, in response to the user's selection operation of a frame in the continuous frame, the three-dimensional face point cloud map corresponding to the selected frame is presented on the human-computer interaction interface; and in response to the user's adjustment command of the presented three-dimensional face point cloud map, the initial pose of the face in the three-dimensional face point cloud map is adjusted to the final pose corresponding to the adjustment command; the pitch angle adjustment angle, yaw angle adjustment angle, and roll angle adjustment angle from the initial pose to the final pose are recorded to obtain the three-dimensional face angle of the selected frame, realizing direct angle annotation of the real three-dimensional face, and the obtained three-dimensional face angle is more accurate. Attached Figure Description

[0051] The disclosure of this invention will become more readily understood with reference to the accompanying drawings. It will be readily understood by those skilled in the art that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. Furthermore, similar numbers in the drawings are used to denote similar components, wherein:

[0052] Figure 1 This is a schematic diagram of the main steps of a three-dimensional face angle annotation method according to an embodiment of the present invention;

[0053] Figure 2 It is a coordinate system describing facial orientation in three-dimensional space;

[0054] Figure 3 This is a schematic diagram of a human face depth image;

[0055] Figure 4 yes Figure 3 Corresponding 3D face point cloud diagram;

[0056] Figure 5 yes Figure 4A schematic diagram showing the adjustment of the corresponding 3D face point cloud map to the final pose;

[0057] Figure 6 These are schematic diagrams of 3D face point cloud maps from different perspectives;

[0058] Figure 7 This is a main structural block diagram of a three-dimensional face angle annotation device according to an embodiment of the present invention. Detailed Implementation

[0059] Some embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0060] In the description of this invention, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and may also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.

[0061] In related technologies, when annotating 3D face angles, there is a relatively large error in annotating face pose based on only a 2D face image. If a 3D virtual face fitted by 3DMM is used for annotation, the fitted 3D virtual face is not the real 3D face data obtained by 3D scanning equipment such as structured light and laser, which makes the annotation results of 3D face angles unreliable.

[0062] Therefore, in order to solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0063] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a three-dimensional face angle annotation method according to an embodiment of the present invention. Figure 1As shown, the three-dimensional face angle annotation method in this embodiment of the invention mainly includes the following steps 101-104.

[0064] Step 101: Obtain a continuous frame 3D face point cloud map based on the continuous frame face images acquired by the depth camera;

[0065] In a specific implementation, each frame of face depth image can be obtained based on each frame of face image captured by a depth camera; based on the focal length and aperture center of the depth camera in the first and second coordinate axes, point cloud data of each frame of face depth image is obtained through coordinate transformation; the pitch angle is the rotation angle about the first coordinate axis, and the yaw angle is the rotation angle about the second coordinate axis; the point cloud data of each frame of face depth image are registered to obtain the continuous frame three-dimensional face point cloud map. Wherein, the pitch angle is the rotation angle about the first coordinate axis, the yaw angle is the rotation angle about the second coordinate axis, and the roll angle is the rotation angle about the third coordinate axis. The first coordinate axis can be the x-axis, the second coordinate axis can be the y-axis, and the third coordinate axis can be the z-axis. Figure 2 As shown. Figure 2 It is a coordinate system that describes the orientation of the face in three-dimensional space.

[0066] Specifically, after capturing each frame of speckle or infrared image by the depth camera, basic digital image processing algorithms such as white balance, wide dynamic range, and exposure compensation are used to ensure that the speckle or infrared image captured by the depth camera does not have obvious color casts or unclear blurring of dark and bright areas after processing. A face detection algorithm is then used to determine whether a face exists in each frame, whether multiple faces exist, the location of each face is determined, and the bounding box of each face is obtained, resulting in the face depth image of each frame. Figure 3 This is a schematic diagram of a human face depth image. (For example...) Figure 3 As shown, the bounding boxes in the face depth image are face bounding boxes, and point cloud reconstruction only reconstructs the face portion within the face bounding boxes. The face detection algorithm is primarily based on the YOLO algorithm, with improvements resulting in an extremely lightweight detection model. While maintaining face detection accuracy, it significantly accelerates the detection speed, achieving real-time performance on any computer's CPU.

[0067] After obtaining each frame of face depth image, the point cloud data of the face depth image can be obtained by coordinate transformation based on the focal length and aperture center of the depth camera in the first coordinate axis direction and the second coordinate axis direction; the point cloud data of the face depth image is then registered to obtain the three-dimensional face point cloud map. Figure 4 yes Figure 3 A schematic diagram of the corresponding 3D face point cloud.

[0068] like Figure 4 As shown, the 3D face point cloud maps are the front view (a), left view (b), and top view (c). The 3D face point cloud maps are viewed from three perspectives. The front view (a) primarily examines whether the face has a roll angle, i.e., whether there is a tilted head, such as... Figure 4 As shown, the face has an angle tilted to the left; the left view (b) mainly checks whether the face has a yaw angle, that is, whether it shakes its head to the left or right; the top view (c) mainly checks whether the face has a pitch angle.

[0069] Step 102: In response to the user's selection operation of a frame in the continuous frames, the three-dimensional face point cloud map corresponding to the selected frame is presented on the human-computer interaction interface.

[0070] In a specific implementation, a user can select the desired 3D face point cloud image from a series of 3D face point cloud images. In response to the user's selection operation of the frame in the series of frames, the 3D face point cloud image corresponding to the selected frame can be presented on the human-computer interaction interface, so that the user can annotate the 3D face point cloud image corresponding to the selected frame on the human-computer interaction interface.

[0071] Step 103: In response to the user's adjustment command for the presented 3D face point cloud map, adjust the initial pose of the face in the 3D face point cloud map to the final pose corresponding to the adjustment command.

[0072] In one specific implementation, after the user inputs the required adjustment command on the presented 3D face point cloud map, a response can be made, adjusting the initial pose of the face in the 3D face point cloud map to the final pose corresponding to the adjustment command. The pitch angle of the final pose is a first preset angle, the yaw angle is a second preset angle, and the roll angle is a third preset angle.

[0073] In some embodiments, the first preset angle, the second preset angle, and the third preset angle can all be 0°. That is, the desired final pose is when there is no pitch, no yaw, and no roll in the 3D face point cloud image, as shown below. Figure 5 As shown in (a), (b), and (c). Figure 5 yes Figure 4 The corresponding 3D face point cloud map is adjusted to the final pose diagram.

[0074] In a specific implementation, a first mapping relationship can be defined between a first function key of the first input device and adjustment actions for translating and rotating the face pose. After the user operates the first function key through the first input device, the system can respond to the user's operation command on the first function key and generate the required adjustment command based on the first mapping relationship to adjust the initial pose to the final pose corresponding to the adjustment command. The first mapping relationship may include the translation distance and rotation angle corresponding to the operation command on the first function key.

[0075] Specifically, the first input device can be a keyboard, and the first function keys are the arrow keys in the main key area and the arrow keys in the auxiliary key area of ​​the keyboard. The initial posture can be translated in response to touch of the arrow keys in the main key area, wherein the translation distance corresponding to each touch of the arrow keys can be configured. Alternatively, the initial posture can be rotated in response to touch of the arrow keys in the auxiliary key area, wherein the rotation angle corresponding to each touch of the arrow keys can be configured. The arrow keys in the main key area and the arrow keys in the auxiliary key area can be set according to actual needs. For example, the arrow keys in the main key area can be the up, down, left, and right arrow keys on a keyboard, and the arrow keys in the auxiliary key area can be the number keys on a numeric keypad or the letter keys on a keyboard; this embodiment does not impose specific limitations.

[0076] In a specific implementation, the translation distance corresponding to each touch of the arrow keys can be configured manually or automatically. For example, in manual configuration, a preset translation distance can be set for each click of an arrow key in the main key area. For instance, clicking the up button once will translate upwards by 5mm; pressing and holding the up button for more than 2 seconds will translate upwards at a preset speed, and so on. Other keys can be set similarly. Of course, the preset translation distance and / or preset translation speed for each arrow key in the main key area can be different, partially the same, or all the same.

[0077] During automatic configuration, if two adjacent touches of the directional keys in the main key area are in opposite directions, the initial translation distance corresponding to the subsequent touch is reduced based on a preset translation adjustment step size to obtain the adjusted translation distance for the subsequent touch. In other words, if two adjacent touches are in opposite directions, it means the translation distance of the previous touch was too large, requiring a reverse translation. However, if the reverse translation is performed according to the initial translation distance, it will return to the position of the previous touch. Therefore, the translation distance needs to be reduced. In this case, the initial translation distance corresponding to the subsequent touch can be directly reduced according to a preset translation adjustment step size to obtain the adjusted translation distance for the subsequent touch, eliminating the need for the user to manually input the reverse translation distance and simplifying the operation process.

[0078] Similarly, the rotation angle corresponding to each touch of the arrow keys can be configured manually or automatically. For example, in manual configuration, a preset rotation angle can be set for each click of an arrow key in the main key area. For instance, clicking the letter A key once rotates it 5° clockwise, and clicking the letter S key once rotates it 5° counterclockwise. Holding down the letter A key for more than 2 seconds will rotate it clockwise at a preset speed, and so on. Other keys can be set similarly. Of course, the preset rotation angle and / or preset rotation speed for each arrow key in the main key area can be different, partially the same, or all the same.

[0079] During automatic configuration, if two adjacent touches of the directional keys in the auxiliary key area are in opposite directions, the initial rotation angle corresponding to the subsequent touch is reduced based on a preset rotation adjustment step size to obtain the adjusted rotation angle for the subsequent touch. In other words, if two adjacent touches are in opposite directions, it means the rotation angle of the previous touch was too large, requiring a reverse rotation. However, if the rotation is performed according to the initial rotation angle, it will return to the position of the previous touch. Therefore, the rotation angle needs to be reduced. In this case, the initial rotation angle corresponding to the subsequent touch can be directly reduced according to a preset rotation adjustment step size to obtain the adjusted rotation angle for the subsequent touch, eliminating the need for the user to manually input the reverse rotation angle and simplifying the operation process.

[0080] Step 104: Record the pitch angle adjustment angle, yaw angle adjustment angle, and roll angle adjustment angle from the initial attitude to the final attitude to obtain the three-dimensional face angle of the selected frame.

[0081] In a specific implementation process, after the 3D face point cloud map completes the angle adjustment according to each adjustment command, the current adjustment angles of the pitch angle, yaw angle, and roll angle can be recorded. Until the 3D face point cloud map is adjusted to the final posture, all the current adjustment angles of the pitch angle, yaw angle, and roll angle are accumulated. Finally, the total pitch angle, yaw angle, and roll angle adjustments of the 3D face point cloud map from the initial posture to the final posture are obtained, thus obtaining the 3D face angle of the selected frame.

[0082] It should be noted that the 3D face angle of the selected frame can be directly adjusted using pitch, yaw, and roll angles, or it can be a quaternion or rotation matrix converted from pitch, yaw, and roll angles. This embodiment does not impose specific limitations. After obtaining the 3D face angle of the selected frame, it can be saved according to a preset data format.

[0083] The 3D face angle annotation method in this embodiment obtains a continuous frame 3D face point cloud map based on continuous frame face images acquired by a depth camera. In response to the user's selection operation of a frame in the continuous frame, the 3D face point cloud map corresponding to the selected frame is presented on the human-computer interaction interface. In response to the user's adjustment command for the presented 3D face point cloud map, the initial pose of the face in the 3D face point cloud map is adjusted to the final pose corresponding to the adjustment command. The pitch angle, yaw angle, and roll angle adjustment angles from the initial pose to the final pose are recorded to obtain the 3D face angle of the selected frame. This method directly annotates the angles of the real 3D face, resulting in more accurate 3D face angles.

[0084] In a specific implementation, before step 103, "in response to the user's adjustment command for the presented 3D face point cloud map, adjust the initial pose of the face in the 3D face point cloud map to the final pose corresponding to the adjustment command," the following steps may also be performed:

[0085] (1) Define a second mapping relationship between the second function key of the second input device and the observation action of changing and scaling the viewpoint of the face pose;

[0086] The second mapping relationship may include viewpoint conversion information and viewpoint scaling information corresponding to the operation command of the second function key.

[0087] (2) In response to the user's operation command on the second function key, the shape and distribution of the face point cloud under different viewing angles and zoom states are presented based on the second mapping relationship so that the user can observe the face details.

[0088] In one specific implementation, the second input device can be a mouse, and the second function key can be the left mouse button and the scroll wheel. It can respond to the user dragging the 3D face point cloud image using the left mouse button, displaying the shape and distribution of the face point cloud from different viewpoints; it can also respond to the user zooming the 3D face point cloud image using the mouse wheel, displaying the shape and distribution of the face point cloud at different zoom levels. In this way, the user can more intuitively observe facial details, determine the required adjustment commands, and more quickly adjust the initial pose of the face in the 3D face point cloud image to the final pose corresponding to the adjustment command.

[0089] Figure 6 These are schematic diagrams of 3D face point cloud maps from different perspectives, such as... Figure 6 As shown, Figure 6 This explanation will use only the front view of a 3D face point cloud as an example. Figure 6 Part (a) is about Figure 4 (a) The main view of the 3D face point cloud obtained after dragging the mouse to change the perspective; Figure 6 Part (b) is about Figure 4 (a) Drag the mouse to change the viewpoint and use the scroll wheel to zoom in and out of the main view of the 3D face point cloud map.

[0090] In a specific implementation, after obtaining the 3D face angle of the selected frame corresponding to the 3D face point cloud map of the selected frame, the 3D face angle of the adjacent frame can be obtained through a point cloud matching algorithm based on the 3D face angle of the selected frame. Based on the 3D face angle of the adjacent frame, the 3D face angle of the frame adjacent to it can be obtained through a point cloud matching algorithm until all consecutive frames are labeled, thereby improving the labeling rate and saving the data according to a preset data format.

[0091] Specifically, a first relative angle between the point cloud map of the selected frame and the point cloud maps of adjacent frames can be obtained through a point cloud matching algorithm. Based on the first relative angle and the 3D face angle of the selected frame, the 3D face angle corresponding to the adjacent frame is obtained. A second relative angle between the point cloud map of the adjacent frame and the point cloud map of its adjacent frame can be obtained through a point cloud matching algorithm. Based on the second relative angle and the 3D face angle corresponding to the adjacent frame, the 3D face angle corresponding to the adjacent frame is obtained.

[0092] In a specific implementation, in response to the user's selection of frames in a series of frames, the three-dimensional face angle corresponding to the selected frame and the intermediate process information of the annotation are presented on the human-computer interaction interface so that the user can verify the automatic annotation results and ensure the accuracy of the three-dimensional face angle corresponding to the three-dimensional face point cloud map of each frame.

[0093] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effects of the present invention, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of the present invention.

[0094] Those skilled in the art will understand that all or part of the processes in the method of the above embodiment of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc. It should be noted that the content included in the computer-readable storage medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable storage medium does not include electrical carrier signals and telecommunication signals.

[0095] Furthermore, the present invention also provides a three-dimensional face angle annotation device.

[0096] See appendix Figure 7 , Figure 7 This is a main structural block diagram of a three-dimensional face angle annotation device according to an embodiment of the present invention. Figure 7 As shown, the three-dimensional face angle annotation device of this embodiment may include a processor 70 and a storage device 71. The storage device 71 is adapted to store multiple program codes, which are adapted to be loaded and run by the processor 70 to execute the three-dimensional face angle annotation method of the above embodiment.

[0097] For ease of explanation, only the parts relevant to the embodiments of the present invention are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The control device may be a control device device comprising various electronic devices.

[0098] Furthermore, the present invention also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to the present invention, the computer-readable storage medium can be configured to store a program for executing the three-dimensional face angle annotation method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described three-dimensional face angle annotation method. For ease of explanation, only the parts related to the embodiments of the present invention are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of the present invention. The computer-readable storage medium can be a storage device comprising various electronic devices. Optionally, in the embodiments of the present invention, the computer-readable storage medium is a non-transitory computer-readable storage medium.

[0099] Furthermore, the present invention also provides a three-dimensional face angle annotation system. This three-dimensional face angle annotation system may include the three-dimensional face angle annotation device and a first input device as described in the above embodiments. The first input device is used to receive adjustment instructions from the user on the presented three-dimensional face point cloud map.

[0100] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, a part of its hardware, or a combination of software and hardware. Therefore, the number of modules shown in the figures is merely illustrative.

[0101] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of the present invention; therefore, the technical solutions after splitting or combining will fall within the protection scope of the present invention.

[0102] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A three-dimensional face angle annotation method, characterized in that, include: A continuous-frame 3D face point cloud map is obtained from continuous-frame real face images captured by a depth camera. In response to the user's selection operation of a frame in the continuous frames, the three-dimensional face point cloud map corresponding to the selected frame is presented on the human-computer interaction interface; In response to the user's adjustment command for the presented 3D face point cloud map, the initial pose of the face in the 3D face point cloud map is adjusted to the final pose corresponding to the adjustment command; wherein, the pitch angle of the final pose is a first preset angle, the yaw angle of the final pose is a second preset angle, and the roll angle of the final pose is a third preset angle. Record the pitch angle adjustment angle, yaw angle adjustment angle, and roll angle adjustment angle from the initial attitude to the final attitude to obtain the three-dimensional face angle of the selected frame; The method of obtaining a continuous frame 3D face point cloud map from continuous frame real face images acquired by a depth camera includes: Acquire each frame of real human face image captured by the depth camera; Face detection is performed on each frame of the real face depth image to obtain the face bounding box of each face in each frame of the real face depth image; Based on the face bounding box, obtain the real face depth image for each frame; Point cloud data of the depth image of each real face is obtained based on each frame of the real face image.

2. The three-dimensional face angle annotation method according to claim 1, characterized in that, Point cloud data of the depth image of each real face image is obtained based on each real face image, including: Based on the focal length and aperture center of the depth camera in the first coordinate axis direction and the second coordinate axis direction, point cloud data of each frame of the real face depth image is obtained through coordinate transformation; the pitch angle is the rotation angle about the first coordinate axis, and the yaw angle is the rotation angle about the second coordinate axis; The point cloud data of each frame of the real face depth image is registered to obtain the continuous frame three-dimensional face point cloud map.

3. The three-dimensional face angle annotation method according to claim 1, characterized in that, In response to a user's adjustment command for the presented 3D face point cloud image, the initial pose of the face in the 3D face point cloud image is adjusted to a final pose corresponding to the adjustment command, including: Define a first mapping relationship between a first function key of a first input device and an adjustment action that translates and rotates the face pose; In response to the user's operation command on the first function key, the initial posture is adjusted to the final posture based on the first mapping relationship.

4. The three-dimensional face angle annotation method according to claim 3, characterized in that, The first input device is a keyboard, and the first function keys are the arrow keys in the main key area and the arrow keys in the auxiliary key area of ​​the keyboard; The step of adjusting the initial posture to the final posture based on the mapping relationship in response to the user's operation command on the first function key includes: In response to a touch of the arrow keys in the main key area of ​​the keyboard, the initial posture is translated, wherein the translation distance corresponding to each touch of the arrow keys can be configured; In response to a touch of the arrow keys in the auxiliary key area of ​​the keyboard, the initial posture is rotated, wherein the rotation angle corresponding to each touch of the arrow keys can be configured.

5. The three-dimensional face angle annotation method according to claim 4, characterized in that, The process of configuring the translation distance corresponding to each touch of the arrow keys includes: If two adjacent touches of the directional keys in the main key area are in opposite directions, the initial translation distance corresponding to the next touch is reduced based on a preset translation adjustment step size to obtain the adjusted translation distance corresponding to the next touch. The process of configuring the rotation angle corresponding to each touch of the arrow keys includes: If two adjacent touches of the directional keys in the auxiliary key area are in opposite directions, the initial rotation angle corresponding to the next touch is reduced based on a preset rotation adjustment step size to obtain the adjustment rotation angle corresponding to the next touch.

6. The three-dimensional face angle annotation method according to claim 3, characterized in that, Before adjusting the initial pose of the face in the three-dimensional face point cloud map to the final pose corresponding to the adjustment command in response to the user's adjustment command, the method further includes: Define a second mapping relationship between the second function key of the second input device and the observation action of changing and scaling the viewpoint of the face pose; In response to the user's operation command on the second function key, the shape and distribution of the face point cloud under different viewing angles and zoom states are presented based on the second mapping relationship, so that the user can observe the face details.

7. The three-dimensional face angle annotation method according to claim 6, characterized in that, The second input device is a mouse, and the second function keys are the left mouse button and the scroll wheel; The step of responding to the user's operation command on the second function key and presenting the shape and distribution of the face point cloud under different viewpoints and zoom states based on the second mapping relationship includes: In response to the user dragging the three-dimensional face point cloud map with the left mouse button, the shape and distribution of the face point cloud are presented from different perspectives. In response to the user zooming in and out on the 3D face point cloud using the mouse wheel, the shape and distribution of the face point cloud are presented under different zoom levels.

8. The three-dimensional face angle annotation method according to any one of claims 1-7, characterized in that, Also includes: Based on the three-dimensional face angle of the selected frame corresponding to the selected frame, the three-dimensional face angle of the adjacent frame is obtained through a point cloud matching algorithm. Based on the three-dimensional face angles corresponding to the adjacent frames, the three-dimensional face angles corresponding to the adjacent frames are obtained through a point cloud matching algorithm until all consecutive frames are labeled.

9. The three-dimensional face angle annotation method according to claim 8, characterized in that, The step of obtaining the 3D face angles of adjacent frames based on the 3D face angles of the selected frame and the selected frame using a point cloud matching algorithm includes: The first relative angle between the point cloud map of the selected frame and the point cloud map of the adjacent frame is obtained through the point cloud matching algorithm. Based on the first relative angle and the three-dimensional face angle of the selected frame corresponding to the selected frame, the three-dimensional face angle corresponding to the adjacent frame is obtained; Based on the 3D face angles corresponding to the adjacent frames, the 3D face angles corresponding to the adjacent frames are obtained through a point cloud matching algorithm, including: A second relative angle between the point cloud map of the adjacent frame and the point cloud map of the adjacent frame is obtained through a point cloud matching algorithm. Based on the second relative angle and the three-dimensional face angle corresponding to the adjacent frame, the three-dimensional face angle corresponding to the adjacent frame is obtained.

10. The three-dimensional face angle annotation method according to claim 8, characterized in that, Also includes: In response to the user's selection of frames in a series of frames, the three-dimensional face angle corresponding to the selected frame and the intermediate process information of the annotation are presented on the human-computer interaction interface so that the user can verify the results of automatic annotation.

11. A three-dimensional face angle annotation device, comprising a processor and a storage device, wherein the storage device is adapted to store multiple lines of program code, characterized in that, The program code is adapted to be loaded and run by the processor to perform the three-dimensional face angle annotation method according to any one of claims 1 to 10.

12. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the three-dimensional face angle annotation method according to any one of claims 1 to 10.

13. A three-dimensional face angle annotation system, characterized in that, include: The three-dimensional face angle annotation device as described in claim 11; The first input device is used to receive user adjustment instructions for the presented 3D face point cloud map.

Images