Three-dimensional key point generation method, device, robot and storage medium

Abstract

The present application discloses a method, device, robot, and storage medium for generating three-dimensional key points, which are used to improve the stability and rationality of generated three-dimensional key points. The method comprises: acquiring an image to be detected, performing key point detection on the image to be detected, obtaining a plurality of two-dimensional initial key points corresponding to the image to be detected, and the confidence levels corresponding to the plurality of two-dimensional initial key points; determining a target matrix based on the plurality of two-dimensional initial key points and the confidence levels corresponding to the plurality of two-dimensional initial key points; inputting the target matrix into a completion model, performing key point completion on the two-dimensional initial key points based on the target matrix, and obtaining two-dimensional completed key points; inputting the two-dimensional completed key points into a conversion model, and calling the conversion model to convert the two-dimensional completed key points into three-dimensional key points.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to a three-dimensional key point generation method, device, robot and storage medium. Background Art

[0002] In the robotics industry, there is a method for generating three-dimensional key points based on the two-dimensional key points of the human body. By using three-dimensional key points to model the human body in three dimensions, the recognition and judgment of human posture can be realized, so that the robot can recognize and predict pedestrian limb movements. Combining the recognition results with the robot's motion control can also realize robot obstacle avoidance, interactive control and other functions.

[0003] In the existing technology, the method for generating three-dimensional key points of the human body is generally as follows: first, the coordinates of the two-dimensional key points of the human body are detected through information such as pictures and video streams, and then the two-dimensional coordinates are input into the deep learning trained model to calculate the corresponding three-dimensional space coordinates, thereby generating three-dimensional key points.

[0004] In the application process of the above method, when performing two-dimensional key point detection, several two-dimensional key points are often missing in the picture, or two-dimensional key point detection errors occur. If this kind of full amount or data containing erroneous two-dimensional key points is directly used to calculate three-dimensional key points, it will cause severe jitter, resulting in poor quality of the output three-dimensional key points. Summary of the Invention

[0005] The present application provides a three-dimensional key point generation method, device, robot and storage medium for improving the stability and rationality of generated three-dimensional key points.

[0006] The first aspect of the present application provides a three-dimensional key point generation method, comprising:

[0007] Acquire an image to be detected, perform key point detection on the image to be detected, and obtain a plurality of two-dimensional initial key points corresponding to the image to be detected, as well as confidence scores corresponding to each of the plurality of two-dimensional initial key points;

[0008] Determine a target matrix according to the plurality of two-dimensional initial key points and the confidence levels corresponding to the plurality of two-dimensional initial key points;

[0009] Inputting the target matrix into the completion model, performing key point completion on the two-dimensional initial key points according to the target matrix to obtain two-dimensional completed key points;

[0010] The two-dimensional completion key points are input into a conversion model, and the conversion model is called to convert the two-dimensional completion key points into three-dimensional key points.

[0011] Optionally, the completion model includes: a first fully connected module, a feature extraction module and a second fully connected module;

[0012] Inputting the target matrix into the completion model, and performing key point completion on the two-dimensional initial key points according to the target matrix to obtain the two-dimensional completed key points includes:

[0013] Inputting the target matrix into the completion model, performing feature mapping processing on the target matrix through the first fully connected module to generate first features;

[0014] Performing feature extraction processing on the first feature by the feature extraction module to generate a second feature;

[0015] The second feature is subjected to key point mapping processing by the second fully connected module to obtain two-dimensional completed key points.

[0016] Optionally, the feature extraction module includes a linear layer, a normalization layer, an activation layer, and a residual superposition layer, and performing feature extraction processing on the first feature by the feature extraction module to generate the second feature includes:

[0017] Performing linear feature extraction processing on the first feature through the linear layer to generate a first intermediate feature;

[0018] performing batch normalization processing on the first intermediate features through the normalization layer to generate second intermediate features;

[0019] activating the second intermediate feature through the activation layer to generate a third intermediate feature;

[0020] The first feature and the third intermediate feature are superimposed on each other through the residual superposition layer to generate a second feature.

[0021] Optionally, determining a target matrix according to the multiple two-dimensional initial key points and the confidence levels corresponding to the multiple two-dimensional initial key points includes:

[0022] Binarizing the confidences corresponding to the plurality of two-dimensional initial key points according to a preset threshold, and generating a confidence vector according to the binarized confidences;

[0023] A target matrix is ​​determined using the multiple two-dimensional initial key points and the confidence vector.

[0024] Optionally, the image to be detected is a human body image. After inputting the two-dimensional completion key points into the conversion model and calling the conversion model to convert the two-dimensional completion key points into three-dimensional key points, the method further includes:

[0025] Determining human body motion based on the three-dimensional key points of multiple consecutive frames;

[0026] Execute a target task according to the human body action.

[0027] Optionally, the method further includes training the completion model, wherein the training of the completion model includes:

[0028] Obtain a sample data set, wherein the sample data set includes several groups of two-dimensional real key points;

[0029] Randomly selecting a set of two-dimensional real key points from the sample data set, randomly setting the two-dimensional real key points, and obtaining a confidence vector;

[0030] Multiplying the two-dimensional true key point and the confidence vector to obtain a key point matrix;

[0031] Inputting the key point matrix into a first initial network to obtain two-dimensional completed key points, wherein the first initial network includes a first fully connected module, a feature extraction module, and a second fully connected module;

[0032] Calculating a first loss value between the two-dimensional completed key point and the two-dimensional real key point;

[0033] The first initial network is iteratively trained according to the first loss value to obtain a trained completion model.

[0034] Optionally, the sample data set further includes three-dimensional real key points corresponding to the two-dimensional real key points, and the method further includes training the conversion model, wherein the training of the conversion model includes:

[0035] Inputting the two-dimensional completed key points into a second initial network to obtain three-dimensional output key points, wherein the structure of the second initial network is the same as that of the first initial network;

[0036] Calculating a second loss value between the three-dimensional real key point and the three-dimensional output key point according to the confidence vector;

[0037] The second initial network is iteratively trained according to the second loss value to obtain a trained conversion model.

[0038] A second aspect of the present application provides a three-dimensional key point generation device, comprising:

[0039] a data acquisition unit, configured to acquire an image to be detected, perform key point detection on the image to be detected, and obtain a plurality of two-dimensional initial key points corresponding to the image to be detected, as well as confidence levels corresponding to the plurality of two-dimensional initial key points;

[0040] a data determining unit, configured to determine a target matrix based on the plurality of two-dimensional initial key points and the confidence levels corresponding to the plurality of two-dimensional initial key points;

[0041] a completion processing unit, configured to input the target matrix into a completion model, and perform key point completion on the two-dimensional initial key points according to the target matrix to obtain two-dimensional completed key points;

[0042] The conversion processing unit is used to input the two-dimensional completion key points into a conversion model, and call the conversion model to convert the two-dimensional completion key points into three-dimensional key points.

[0043] Optionally, the completion model includes: a first fully connected module, a feature extraction module and a second fully connected module;

[0044] The completion processing unit is specifically used for:

[0045] Inputting the target matrix into the completion model, performing feature mapping processing on the target matrix through the first fully connected module to generate first features;

[0046] Performing feature extraction processing on the first feature by the feature extraction module to generate a second feature;

[0047] The second feature is subjected to key point mapping processing by the second fully connected module to obtain two-dimensional completed key points.

[0048] Optionally, the feature extraction module includes a linear layer, a normalization layer, an activation layer, and a residual superposition layer, and the completion processing unit is further configured to:

[0049] Performing linear feature extraction processing on the first feature through the linear layer to generate a first intermediate feature;

[0050] performing batch normalization processing on the first intermediate features through the normalization layer to generate second intermediate features;

[0051] activating the second intermediate feature through the activation layer to generate a third intermediate feature;

[0052] The first feature and the third intermediate feature are superimposed on each other through the residual superposition layer to generate a second feature.

[0053] Optionally, the data determination unit is specifically configured to:

[0054] Binarizing the confidences corresponding to the plurality of two-dimensional initial key points according to a preset threshold, and generating a confidence vector according to the binarized confidences;

[0055] A target matrix is ​​determined using the multiple two-dimensional initial key points and the confidence vector.

[0056] Optionally, the device further comprises: a task execution unit;

[0057] The task execution unit is used to:

[0058] Determining human body motion based on the three-dimensional key points of multiple consecutive frames;

[0059] Execute a target task according to the human body action.

[0060] Optionally, the device further comprises: a model training unit;

[0061] The model training unit is specifically used for:

[0062] Obtain a sample data set, wherein the sample data set includes several groups of two-dimensional real key points;

[0063] Randomly selecting a set of two-dimensional real key points from the sample data set, randomly setting the two-dimensional real key points, and obtaining a confidence vector;

[0064] Multiplying the two-dimensional true key point and the confidence vector to obtain a key point matrix;

[0065] Inputting the key point matrix into a first initial network to obtain two-dimensional completed key points, wherein the first initial network includes a first fully connected module, a feature extraction module, and a second fully connected module;

[0066] Calculating a first loss value between the two-dimensional completed key point and the two-dimensional real key point;

[0067] The first initial network is iteratively trained according to the first loss value to obtain a trained completion model.

[0068] Optionally, the model training unit is further used to:

[0069] Inputting the two-dimensional completed key points into a second initial network to obtain three-dimensional output key points, wherein the structure of the second initial network is the same as that of the first initial network;

[0070] Calculating a second loss value between the three-dimensional real key point and the three-dimensional output key point according to the confidence vector;

[0071] The second initial network is iteratively trained according to the second loss value to obtain a trained conversion model.

[0072] A third aspect of the present application provides a robot, comprising:

[0073] processor, memory, input and output units, and buses;

[0074] The processor is connected to the memory, the input and output unit, and the bus;

[0075] The memory stores a program, and the processor calls the program to execute the first aspect and any optional three-dimensional key point generation method in the first aspect.

[0076] In a fourth aspect, the present application provides a computer-readable storage medium having a program stored thereon. When the program is executed on a computer, the program executes the first aspect and any optional three-dimensional key point generation method in the first aspect.

[0077] It can be seen from the above technical solutions that this application has the following advantages:

[0078] The three-dimensional key point generation method provided by this application is divided into two stages: two-dimensional completion and three-dimensional conversion. First, the detected two-dimensional key points are completed by the completion model, and then the completed two-dimensional key points are converted into three-dimensional key points by the conversion model. Through the above method, even when a large number of key points are missing in the picture, the correct three-dimensional space coordinates can be output for the visible key points, and stable and reasonable three-dimensional space coordinates can be output for the invisible key points. The distribution of the two-dimensional key points input to the conversion model is guaranteed to be close to reality to the greatest extent, so that the stability and rationality of the finally generated three-dimensional key points are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0080] Figure 1 A schematic diagram of the hardware structure of the robot provided in this application;

[0081] Figure 2 A schematic diagram of the mechanical structure of the robot provided in this application;

[0082] Figure 3 A schematic diagram of an embodiment of a three-dimensional key point generation method provided in this application;

[0083] Figure 4 Schematic diagram of the network structure of the completion model and the conversion model in the three-dimensional key point generation method provided by this application;

[0084] Figure 5A schematic flow chart of another embodiment of the three-dimensional key point generation method provided by this application;

[0085] Figure 6 A flowchart of an embodiment of the training method for the completion model and the conversion model provided in this application;

[0086] Figure 7 Schematic diagram of various training methods for the completion model and conversion model provided in this application;

[0087] Figure 8 This is a schematic diagram of the structure of an embodiment of a device for generating three-dimensional coordinates of a human body provided by this application;

[0088] Figure 9 This is a schematic structural diagram of an embodiment of the robot provided in this application. DETAILED DESCRIPTION

[0089] The present application provides a three-dimensional key point generation method, device, robot and storage medium for improving the stability and rationality of generated three-dimensional key points.

[0090] It should be understood that the specific embodiments described herein are intended only to explain the present invention and are not intended to limit the present invention. It should be noted that the method for generating three-dimensional key points provided in this application can be applied to terminals as well as servers. For example, the terminal can be a robot, a smartphone or computer, a tablet computer, a smart TV, a smartwatch, a portable computer terminal, or a fixed terminal such as a desktop computer. For ease of description, the following description will be based on the terminal execution entity.

[0091] In the subsequent description, suffixes such as "module," "component," or "unit" used to represent components are used only to facilitate the description of the present invention and have no specific meaning. Therefore, "module," "component," or "unit" may be used interchangeably.

[0092] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of the hardware structure of a robot 100 according to one embodiment of the present invention. Figure 1 In the embodiment shown, the robot 100 includes a mechanical unit 101, a communication unit 102, a sensor unit 103, an interface unit 104, a storage unit 105, a control module 110, and a power supply 111. The various components of the robot 100 can be connected in any manner, including wired or wireless connections. It will be understood by those skilled in the art that Figure 1The specific structure of the robot 100 shown in the figure does not constitute a limitation of the robot 100. The robot 100 may include more or fewer components than shown in the figure. Some components are not necessary components of the robot 100 and can be omitted or combined as needed without changing the essence of the invention.

[0093] The following combination Figure 1 The following describes the various components of the robot 100:

[0094] The mechanical unit 101 is the hardware of the robot 100. Figure 1 As shown, the mechanical unit 101 may include a driving plate 1011, a motor 1012, and a mechanical structure 1013. Figure 2 As shown, the mechanical structure 1013 may include a body 1014, extendable legs 1015, and a foot 1016. In other embodiments, the mechanical structure 1013 may further include an extendable mechanical arm (not shown), a rotatable head structure 1017, a swingable tail structure 1018, a cargo structure 1019, a saddle structure 1020, a camera structure 1021, etc. It should be noted that the various component modules of the mechanical unit 101 can be one or more, and can be arranged according to specific circumstances. For example, there can be four legs 1015, each leg 1015 can be configured with three motors 1012, and the corresponding number of motors 1012 is 12.

[0095] The communication unit 102 can be used to receive and send signals and can also communicate with the network and other devices. For example, after receiving instructions from a remote control or other robot 100 to move in a specific direction at a specific speed according to a specific gait, the communication unit 102 can be transmitted to the control module 110 for processing. The communication unit 102 can include, for example, a WiFi module, a 4G module, a 5G module, a Bluetooth module, an infrared module, etc.

[0096] The sensing unit 103 is used to obtain information data about the robot 100's surroundings and monitor parameter data of various components within the robot 100, and transmits the data to the control module 110. The sensing unit 103 includes a variety of sensors, such as sensors for obtaining information about the surrounding environment: lidar (for long-range object detection, distance determination, and / or velocity determination), millimeter-wave radar (for short-range object detection, distance determination, and / or velocity determination), cameras, infrared cameras, and a global navigation satellite system (GNSS). Sensors for monitoring various components within the robot 100 include an inertial measurement unit (IMU) (for measuring velocity, acceleration, and angular velocity), a foot sensor (for monitoring the foot's foot contact point location, foot posture, ground contact force magnitude and direction), and a temperature sensor (for detecting component temperature). Other sensors that may be configured for the robot 100, such as load sensors, touch sensors, motor angle sensors, and torque sensors, are not described here.

[0097] The interface unit 104 may be used to receive input (e.g., data information, power, etc.) from an external device and transmit the received input to one or more components within the robot 100, or may be used to output (e.g., data information, power, etc.) to an external device. The interface unit 104 may include a power port, a data port (e.g., a USB port), a memory card port, a port for connecting a device having an identification module, an audio input / output (I / O) port, a video I / O port, etc.

[0098] The storage unit 105 is used to store software programs and various data. The storage unit 105 may primarily include a program storage area and a data storage area. The program storage area may store operating system programs, motion control programs, and application programs (such as a text editor), while the data storage area may store data generated by the robot 100 during use (such as various sensor data acquired by the sensor unit 103 and log file data). Furthermore, the storage unit 105 may include high-speed random access memory and non-volatile memory, such as disk storage, flash memory, or other volatile solid-state memory.

[0099] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0100] The input unit 107 can be used to receive input digital or character information. Specifically, the input unit 107 may include a touch panel 1071 and other input devices 1072. The touch panel 1071, also known as a touch screen, can collect user touch operations (such as operations performed by the user using the palm, finger or suitable accessories on the touch panel 1071 or near the touch panel 1071) and drive the corresponding connection device according to a pre-set program. The touch panel 1071 may include two parts: a touch detection device 1073 and a touch controller 1074. Among them, the touch detection device 1073 detects the user's touch direction and detects the signal caused by the touch operation, and transmits the signal to the touch controller 1074; the touch controller 1074 receives the touch information from the touch detection device 1073 and converts it into touch point coordinates, which are then sent to the control module 110. It can also receive commands sent by the control module 110 and execute them. In addition to the touch panel 1071, the input unit 107 may also include other input devices 1072. Specifically, other input devices 1072 may include but are not limited to one or more of a remote control operating handle, etc., which are not specifically limited here.

[0101] Furthermore, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the control module 110 to determine the type of touch event. Subsequently, the control module 110 provides corresponding visual output on the display panel 1061 according to the type of touch event. Figure 1 In the embodiment, the touch panel 1071 and the display panel 1061 are used as two independent components to realize input and output functions respectively. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize input and output functions, which is not limited here.

[0102] The control module 110 is the control center of the robot 100. It uses various interfaces and lines to connect the various components of the entire robot 100, and controls the robot 100 as a whole by running or executing software programs stored in the storage unit 105 and calling data stored in the storage unit 105.

[0103] The power supply 111 is used to supply power to each component. The power supply 111 may include a battery and a power control board. The power control board is used to control functions such as battery charging, discharging, and power consumption management. Figure 1 In the illustrated embodiment, the power supply 111 is electrically connected to the control module 110. In other embodiments, the power supply 111 may also be electrically connected to the sensor unit 103 (such as a camera, radar, speaker, etc.) and the motor 1012. It should be noted that each component can be connected to a different power supply 111 or powered by the same power supply 111.

[0104] Based on the above embodiments, specifically, in some embodiments, a terminal device can be used to establish a communication connection with the robot 100. During communication between the terminal device and the robot 100, the terminal device can send command information to the robot 100. The robot 100 can receive the command information via the communication unit 102 and, upon receiving the command information, transmit the command information to the control module 110, so that the control module 110 can process the command information to obtain a target speed value. Terminal devices include, but are not limited to, mobile phones with image capture capabilities, tablet computers, servers, personal computers, wearable smart devices, and other electrical devices.

[0105] The command information can be determined based on preset conditions. In one embodiment, the robot 100 may include a sensing unit 103, which can generate command information based on the current environment of the robot 100. The control module 110 can use the command information to determine whether the current speed of the robot 100 meets the corresponding preset conditions. If so, the robot 100's current speed and gait will be maintained. If not, a target speed and gait will be determined based on the corresponding preset conditions, thereby controlling the robot 100 to move at the target speed and gait. Environmental sensors may include temperature sensors, air pressure sensors, visual sensors, and sound sensors. The command information may include temperature information, air pressure information, image information, and sound information. The communication method between the environmental sensors and the control module 110 may be wired or wireless. Wireless communication methods include, but are not limited to, wireless networks, mobile communication networks (3G, 4G, 5G, etc.), Bluetooth, and infrared.

[0106] The following is an explanation of the three-dimensional key point generation method provided by this application. Figure 3 , Figure 3 An embodiment of a three-dimensional key point generation method provided by the present application includes:

[0107] Step 301: Acquire an image to be detected, perform key point detection on the image to be detected, and obtain multiple two-dimensional initial key points corresponding to the image to be detected, as well as confidence levels corresponding to the multiple two-dimensional initial key points.

[0108] The terminal obtains the image to be detected and performs key point detection on the image to be detected. Key point detection specifically refers to detecting the coordinate positions of certain feature points in the image. It should be noted that in this embodiment, key point detection can be performed on human bodies, animals, or specific categories of objects (such as hand bones, vehicles, etc.), and the specific method is not limited here.

[0109] After performing key point detection on the image to be detected, the terminal can obtain the coordinate positions of multiple two-dimensional initial key points in the image to be detected, as well as the confidence level of each key point. The confidence level of the key point is used to characterize the accuracy of the detected key point position, and the confidence level is between 0 and 1. There are many ways for the terminal to implement two-dimensional key point detection, such as through deep learning, manual labeling, etc. Regardless of the method used, after obtaining the two-dimensional initial key points and the corresponding confidence levels, the three-dimensional key point generation method provided in this embodiment can be used to generate corresponding three-dimensional key points from the two-dimensional initial key points.

[0110] Step 302: Determine a target matrix based on the multiple two-dimensional initial key points and the confidence levels corresponding to the multiple two-dimensional initial key points.

[0111] Since several two-dimensional key points are often missing in the picture or the two-dimensional key points are detected incorrectly during two-dimensional key point detection, and the confidence level can reflect these problems to a certain extent, after the terminal detects the positions and confidence levels of multiple two-dimensional initial key points in the image to be detected, it determines the target matrix based on the coordinates and confidence levels of the two-dimensional initial key points. The target matrix also contains the coordinate information and confidence level information of the two-dimensional initial key points, preparing for subsequent completion steps.

[0112] Step 303: Input the target matrix into the completion model, and perform key point completion on the two-dimensional initial key points according to the target matrix to obtain two-dimensional completed key points.

[0113] In this embodiment, the process of generating 3D key points from 2D key points is divided into two stages: the first stage is the completion of the 2D key points, and the second stage is the conversion of the completed 2D key points to 3D key points.

[0114] The first stage is specifically implemented through a completion model. The completion model is trained with a sample data set of two-dimensional key points. The terminal inputs the target matrix into the completion model. The completion model outputs the completed two-dimensional completion key points based on the coordinate information and confidence information of the two-dimensional initial key points contained in the target matrix. The two-dimensional completion key points are predicted and completed for the invisible key points based on the two-dimensional initial key points to ensure that the two-dimensional key points converted in the second stage are complete and reasonable.

[0115] Step 304: Input the two-dimensional completion key points into the conversion model, and call the conversion model to convert the two-dimensional completion key points into three-dimensional key points.

[0116] The second stage is implemented using a conversion model trained on a dataset of paired 2D / 3D keypoints. In image processing, keypoints are essentially features, and the conversion model extracts these features from 2D keypoint data and generates corresponding 3D keypoints. The terminal inputs the completed 2D keypoints into the conversion model, which then processes them and converts them into 3D keypoints.

[0117] In this embodiment, the process of generating 3D key points from 2D key points is divided into two stages: 2D completion and 3D conversion. First, the detected 2D key points are completed using a completion model, and then the completed 2D key points are converted into 3D key points using a conversion model. This method allows the output of correct 3D spatial coordinates for visible key points, even when a large number of key points are missing from the image, while also outputting stable and reasonable 3D spatial coordinates for invisible key points. This ensures that the distribution of 2D key points input into the conversion model is as close to reality as possible, significantly improving the stability and rationality of the resulting 3D key points.

[0118] See also Figure 4 In some specific embodiments, the completion model and the conversion model have the same structure, both including a first fully connected module 401, a feature extraction module 402, and a second fully connected module 403. The following describes the 3D key point generation method provided by this application in detail in conjunction with the specific structures of the completion model and the conversion model. Figure 5 , Figure 5 An embodiment of a three-dimensional key point generation method provided by the present application includes:

[0119] Step 501: Acquire an image to be detected, perform key point detection on the image to be detected, and obtain multiple two-dimensional initial key points corresponding to the image to be detected, as well as confidence levels corresponding to the multiple two-dimensional initial key points.

[0120] In this embodiment, step 501 is similar to step 401 in the above embodiment and will not be described in detail here. In this embodiment and subsequent embodiments, the scenario where the image to be detected is a human body image is mainly described, specifically taking the terminal detecting 17 two-dimensional initial key points and the confidence levels corresponding to the 17 key points as an example.

[0121] Step 502: Binarize the confidences corresponding to the multiple two-dimensional initial key points according to a preset threshold, and generate a confidence vector according to the binarized confidences.

[0122] The human body can be captured in a camera image, showing either the entire body or just half of the body. This can lead to false detections of 2D key points or missing key points. Using a model to directly convert 2D to 3D can severely degrade performance. The confidence level obtained from 2D key point detection can, to a certain extent, reflect the accuracy of the key points. To mitigate the impact of false detections or missing key points, this embodiment uses a preset threshold to binarize the confidence level, reducing the impact of unreliable 2D key points.

[0123] Specifically, the terminal binarizes the confidence scores of the multiple detected two-dimensional initial keypoints to 0 and 1 based on a preset threshold, and generates a confidence vector based on the binarized data. A 1*17 confidence vector is composed of the confidence scores of the 17 two-dimensional initial keypoints. In some specific embodiments, the preset threshold can be set to 0.5, and the terminal sets confidence scores less than 0.5 to 0 and confidence scores greater than 0.5 to 1 based on the threshold.

[0124] Step 503: Determine a target matrix using multiple two-dimensional initial key points and confidence vectors.

[0125] The terminal multiplies the detected two-dimensional initial key points and the binarized confidence vector to obtain the target matrix, that is, multiplying the b*17*2 two-dimensional initial key point coordinates and the corresponding positions of the b*17*1 confidence vector, and then flattening the two dimensions to obtain the b*34 target matrix. The target matrix carries the coordinate information and confidence information of the two-dimensional initial key points, and the confidence information is binarized to minimize the impact of unreliable two-dimensional key points.

[0126] Step 504: Input the target matrix into the completion model, perform feature mapping processing on the target matrix through the first fully connected module, and generate the first feature.

[0127] In this embodiment, the completion model includes a first fully connected module 401, a feature extraction module 402, and a second fully connected module 403. When the terminal inputs the target matrix into the completion model, it first passes through the first fully connected module 401. This fully connected input module is a multilayer perceptron (MLP) structure. The MLP structure includes at least an input layer, a hidden layer, and an output layer. The layers are fully connected, that is, any neuron in the previous layer is connected to all neurons in the next layer. After the terminal inputs the target matrix (b*34) into the completion model, the first fully connected module performs feature mapping on the target matrix to generate the first feature (b*1024). This feature mapping process specifically refers to the process of mapping the target matrix into a feature matrix.

[0128] Step 505: Perform feature extraction processing on the first feature through a feature extraction module to generate a second feature.

[0129] See also Figure 4 In this embodiment, the feature extraction module 402 further includes a linear layer 402a, a normalization layer 402b, an activation layer 402c, and a residual superposition layer 402d. Step 505 further includes:

[0130] 1) Perform linear feature extraction processing on the first feature through the linear layer to generate a first intermediate feature.

[0131] The terminal inputs the first feature into feature extraction module 402, which first performs linear feature extraction on the first feature through linear layer 402a to obtain a first intermediate feature (b*1024). Linear layer 402a, also known as a fully connected layer, has each neuron connected to the previous neuron to achieve a linear combination of the previous layer, thus achieving a linear transformation of the feature.

[0132] 2) The first intermediate features are batch normalized through the normalization layer to generate the second intermediate features.

[0133] The first intermediate feature continues to pass through the normalization layer (Batch norm) 402b, which performs batch normalization on the channel dimension of the first intermediate feature (b*1024) to generate the second intermediate feature (b*1024). The function of normalization layer 402b is to prevent gradient explosion and gradient vanishing, reduce model error rate, and improve model performance.

[0134] 3) Activate the second intermediate feature through the activation layer to generate the third intermediate feature.

[0135] The second intermediate feature continues to pass through activation layer 402c, which performs a nonlinear transformation on the second intermediate feature (b*1024) to obtain the third intermediate feature (b*1024). Activation layer 402c can alleviate the vanishing gradient problem to a certain extent, allowing the model to be deeper. Activation layer 402c can use ReLU or Sigmoid as the activation function, but the specifics are not limited here.

[0136] It should be noted that steps 1), 2), and 3) can be repeated more than twice, and the feature obtained in the last step is used as the third intermediate feature to achieve a better feature extraction effect.

[0137] 4) The first feature and the third intermediate feature are superimposed through the residual superposition layer to generate the second feature.

[0138] Residual stacking layer 402d superimposes the initial input first feature (b*1024) with the third intermediate feature (b*1024) as the residual structure, and repeats steps 1), 2), and 3) above to obtain the second feature (b*1024). This residual stacking layer 402d solves the difficulty of optimizing deep networks and allows for deeper network structures to achieve higher accuracy.

[0139] It should be noted that the feature extraction module 402 may also include a Dropout layer, in which neurons are randomly reset to zero with a probability of 0.5, so that the training of one neuron does not depend on another neuron, which weakens the synergy between features, reduces overfitting, and makes the model more generalizable.

[0140] Step 506: Perform key point mapping processing on the second feature through the second fully connected module to obtain two-dimensional completed key points.

[0141] The terminal performs keypoint mapping on the second feature through the second fully connected module 403 to obtain three-dimensional keypoints. This keypoint mapping process specifically maps the feature matrix to three-dimensional keypoints. The second fully connected module 403 also has an MLP structure. The second feature (b*1024) is input to the second fully connected module 403 to obtain an output vector of b*34. The reshape transformation is then performed to obtain the two-dimensional completed coordinates of b*17*2.

[0142] 507. Input the two-dimensional completion key points into the conversion model, and call the conversion model to convert the two-dimensional completion key points into three-dimensional key points.

[0143] In this embodiment, the conversion model has the same structure as the completion model, but uses different training data, resulting in different parameters and weights in the trained model. The difference between the conversion model and the completion model is that the conversion model takes two-dimensional coordinates as input and outputs three-dimensional coordinates, while the completion model uses two-dimensional coordinates as input and output.

[0144] Specifically, the conversion model inputs the two-dimensional completed coordinates of b*17*2, and flattens the two dimensions to obtain a b*34 matrix; first, it passes through the first fully connected module (input b*34 and output b*1024), then passes through the feature extraction module, and goes through the linear layer, normalization layer, activation layer and residual superposition layer in the feature extraction module (input b*1024 and output b*1024), and then passes through the second fully connected module (input b*1024 and output b*48), and after reshape transformation, the three-dimensional key point coordinates of b*16*3 are obtained.

[0145] Step 508: Determine human body motion based on the three-dimensional key points of multiple consecutive frames.

[0146] In this embodiment, 2D keypoints are detected and 3D keypoints are generated for the human body. By acquiring 3D keypoint results from multiple consecutive frames, the terminal can determine the human body's movements. This allows for human pose estimation and human-computer interaction. Compared to 2D keypoints, 3D keypoints contain more information, enabling more accurate human pose estimation.

[0147] Step 509: Execute the target task according to the human body movement.

[0148] The terminal can perform different target tasks based on different human movements. Human posture estimation can help computers and robots better understand people's position and behavior. With human posture, computers and robots can easily execute instructions and become more intelligent.

[0149] Key point detection of the human body is currently a widely used field. This embodiment describes a method for generating three-dimensional key points of the human body. The process of generating three-dimensional key points from two-dimensional key points is divided into two stages: two-dimensional completion and three-dimensional conversion. First, the detected two-dimensional key points of the human body are completed using a completion model, and then the completed two-dimensional key points are converted into three-dimensional key points using a conversion model. Through this method, even if a large number of human key points are missing in the image (only half of the body is captured), the correct three-dimensional spatial coordinates can be output for the visible key points, while stable and reasonable three-dimensional spatial coordinates are output for the invisible key points. This ensures that the distribution of the two-dimensional key points input to the conversion model is close to reality, which significantly improves the stability and rationality of the finally generated three-dimensional key points.

[0150] Furthermore, different scenarios require different numbers of key points, and different datasets also have different numbers of key points labeled. In this case, the only solution is to train a separate model for each scenario, which results in excessive model redundancy. However, in this embodiment, a full set of key point definitions is found as the key point definition, and 2D completion training is performed for each data set to obtain the corresponding 2D completion model. Then, a conversion model is used to perform 2D-to-3D conversion, allowing a single model to cover various scenarios requiring different numbers of key points.

[0151] The following describes the training methods for the completion model and conversion model in this application. Figure 6 , Figure 6 The training method for the completion model and conversion model provided in this application includes:

[0152] Step 601: Obtain a sample data set, where the sample data set includes several groups of two-dimensional real key points.

[0153] During the training phase, a dataset must be prepared, consisting of multiple sets of matched 2D / 3D keypoint coordinate data. Taking 17 keypoints as an example, the dataset should contain multiple sets of sample data, each containing both the 2D and 3D true keypoint coordinates for the 17 keypoints. It should be noted that only the 2D true coordinates are used during training of the completion model.

[0154] Step 602: randomly select a set of two-dimensional real key points from the sample data set, randomly set the two-dimensional real key points, and obtain a confidence vector.

[0155] The terminal randomly selects a set of two-dimensional real key points from the sample data set, and randomly masks the set of two-dimensional real key points to obtain a confidence vector containing 0 and 1.

[0156] Step 603: Multiply the two-dimensional true key point and the confidence vector to obtain a key point matrix.

[0157] The terminal multiplies the corresponding positions of the two-dimensional real key points and the confidence vector to obtain the masked key point matrix.

[0158] Step 604: Input the key point matrix into the first initial network to obtain two-dimensional completed key points. The first initial network includes a first fully connected module, a feature extraction module, and a second fully connected module.

[0159] The terminal inputs the key point matrix into the first initial network to obtain a two-dimensional completed key point. The first initial network structure is as follows: Figure 4 As shown, the data flow in the first initial network is as follows Figure 5 The corresponding embodiments are shown and will not be described again here.

[0160] Step 605: Calculate a first loss value between the two-dimensional completed key point and the two-dimensional true key point.

[0161] The terminal calculates the first loss value between the two-dimensional completed key point and the two-dimensional true key point. The first loss value can be calculated by the mean square error loss function MSELoss, or by the cross-entropy loss function Cross-entropy, which is not limited here.

[0162] Step 606: Iteratively train the first initial network according to the first loss value to obtain a trained completion model.

[0163] The terminal trains the first initial network based on the first loss value, backpropagates the first loss value to update the parameters of the first initial network, and then reselects a set of two-dimensional real key points and repeats steps 602 to 605 to train the first initial network until a preset number of training times is reached. The trained first initial network is then determined as the completion model. The training method can be a mini-batch gradient descent method or a stochastic gradient descent method, which is not specifically limited here.

[0164] Step 607: Input the two-dimensional completed key points into the second initial network to obtain three-dimensional output key points. The structure of the second initial network is the same as that of the first initial network.

[0165] See also Figure 7 , Figure 7 Schematic diagram of various training methods for the completion model and the conversion model. The completion model is mainly trained by the two-dimensional real key points in the sample data set, but when training the conversion model, it can be trained directly by the matched two-dimensional real coordinates and three-dimensional real coordinates in the sample data set, and can also be trained by the two-dimensional completion coordinates and three-dimensional real coordinates output in the above step 604.

[0166] Specifically, the terminal inputs the two-dimensional completion key points into the second initial network to obtain three-dimensional output key points. The second initial network structure is as follows: Figure 4 As shown, the data flow in the second initial network is as follows Figure 5 The corresponding embodiments are shown and will not be described again here.

[0167] Step 608: Calculate a second loss value between the 3D real key point and the 3D output key point based on the confidence vector;

[0168] When the second initial network is trained using the two-dimensional completed coordinates and the three-dimensional real coordinates, the terminal specifically only calculates the second loss value between the three-dimensional real key point and the three-dimensional output key point at a confidence level of 1 to achieve stable mapping of visible points.

[0169] Step 609: Iteratively train the second initial network according to the second loss value to obtain a trained conversion model.

[0170] The terminal iteratively trains the second initial network based on the second loss value, backpropagating the second loss value to update the parameters of the second initial network until a preset number of training cycles are reached, and then determines the trained second initial network as the conversion model. The training method can also be mini-batch gradient descent or stochastic gradient descent, though the specifics are not limited here.

[0171] See also Figure 8 , Figure 8 An embodiment of a device for three-dimensional modeling of a human body provided in this application includes:

[0172] The data acquisition unit 801 is used to acquire an image to be detected, perform key point detection on the image to be detected, and obtain multiple two-dimensional initial key points corresponding to the image to be detected, as well as confidence levels corresponding to the multiple two-dimensional initial key points;

[0173] A data determination unit 802 is configured to determine a target matrix based on the plurality of two-dimensional initial key points and the confidence levels corresponding to the plurality of two-dimensional initial key points;

[0174] The completion processing unit 803 is used to input the target matrix into the completion model, and perform key point completion on the two-dimensional initial key points according to the target matrix to obtain two-dimensional completed key points;

[0175] The conversion processing unit 804 is configured to input the two-dimensional completion key points into the conversion model, and call the conversion model to convert the two-dimensional completion key points into three-dimensional key points.

[0176] Optionally, the completion model includes: a first fully connected module, a feature extraction module, and a second fully connected module;

[0177] The completion processing unit 803 is specifically used for:

[0178] Input the target matrix into the completion model, perform feature mapping on the target matrix through the first fully connected module to generate the first feature;

[0179] Performing feature extraction processing on the first feature by a feature extraction module to generate a second feature;

[0180] The second feature is mapped to key points through the second fully connected module to obtain two-dimensional completed key points.

[0181] Optionally, the feature extraction module includes a linear layer, a normalization layer, an activation layer, and a residual superposition layer, and the completion processing unit 803 is further configured to:

[0182] Performing linear feature extraction processing on the first feature through a linear layer to generate a first intermediate feature;

[0183] Performing batch normalization on the first intermediate features through a normalization layer to generate second intermediate features;

[0184] Activate the second intermediate feature through the activation layer to generate a third intermediate feature;

[0185] The first feature and the third intermediate feature are superimposed through the residual superposition layer to generate the second feature.

[0186] Optionally, the data determining unit 802 is specifically configured to:

[0187] Binarize the confidences corresponding to the multiple two-dimensional initial key points according to a preset threshold, and generate a confidence vector based on the binarized confidences;

[0188] The target matrix is ​​determined by multiple 2D initial key points and confidence vectors.

[0189] Optionally, the device further includes: a task execution unit 805;

[0190] The task execution unit 805 is used to:

[0191] Determine human body movements based on 3D key points of multiple consecutive frames;

[0192] Perform target tasks based on human actions.

[0193] Optionally, the apparatus further includes: a model training unit 806;

[0194] The model training unit 806 is specifically used to:

[0195] Obtain a sample data set, which contains several sets of two-dimensional real key points;

[0196] Randomly select a set of two-dimensional real key points from the sample data set, randomly set the two-dimensional real key points, and obtain the confidence vector;

[0197] Multiply the two-dimensional true key points and the confidence vector to obtain the key point matrix;

[0198] Inputting the key point matrix into a first initial network to obtain two-dimensional completed key points, the first initial network including a first fully connected module, a feature extraction module, and a second fully connected module;

[0199] Calculate the first loss value between the two-dimensional completed key point and the two-dimensional true key point;

[0200] The first initial network is iteratively trained according to the first loss value to obtain a trained completion model.

[0201] Optionally, the model training unit 806 is further configured to:

[0202] Input the two-dimensional completed key points into the second initial network to obtain three-dimensional output key points. The structure of the second initial network is the same as that of the first initial network.

[0203] Calculate a second loss value between the three-dimensional true key point and the three-dimensional output key point according to the confidence vector;

[0204] The second initial network is iteratively trained according to the second loss value to obtain a trained conversion model.

[0205] In the device of this embodiment, the functions of each unit are the same as those described above. Figure 3 or Figure 5 The steps in the method embodiment shown correspond to each other and will not be repeated here.

[0206] This application also provides a robot, see Figure 9 , Figure 9 An embodiment of a robot provided in this application includes:

[0207] Processor 901, memory 902, input and output unit 903, bus 904;

[0208] The processor 901 is connected to the memory 902, the input and output unit 903, and the bus 904;

[0209] The memory 902 stores a program, and the processor 901 calls the program to execute any of the above three-dimensional key point generation methods.

[0210] The present application also relates to a computer-readable storage medium having a program stored thereon, wherein when the program is run on a computer, the computer is caused to execute any of the above three-dimensional key point generation methods.

[0211] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0212] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0213] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0214] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0215] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk, and other media that can store program code.

Claims

1. A three-dimensional key point generation method, characterized in that: The method comprises: Acquire an image to be detected, perform key point detection on the image to be detected, and obtain a plurality of two-dimensional initial key points corresponding to the image to be detected, as well as confidence scores corresponding to each of the plurality of two-dimensional initial key points; Determine a target matrix according to the plurality of two-dimensional initial key points and the confidence levels corresponding to the plurality of two-dimensional initial key points; Inputting the target matrix into the completion model, completing the two-dimensional initial key points according to the target matrix to obtain two-dimensional completed key points; The two-dimensional completion key points are input into a conversion model, and the conversion model is called to convert the two-dimensional completion key points into three-dimensional key points.

2. The method according to claim 1, characterized in that The completion model includes: a first fully connected module, a feature extraction module and a second fully connected module; Inputting the target matrix into the completion model, and performing key point completion on the two-dimensional initial key points according to the target matrix to obtain the two-dimensional completed key points includes: Inputting the target matrix into the completion model, performing feature mapping processing on the target matrix through the first fully connected module to generate first features; Performing feature extraction processing on the first feature by the feature extraction module to generate a second feature; The second feature is subjected to key point mapping processing by the second fully connected module to obtain two-dimensional completed key points.

3. The method according to claim 2, characterized in that The feature extraction module includes a linear layer, a normalization layer, an activation layer, and a residual superposition layer. The feature extraction module performs feature extraction processing on the first feature to generate the second feature includes: Performing linear feature extraction processing on the first feature through the linear layer to generate a first intermediate feature; performing batch normalization processing on the first intermediate features through the normalization layer to generate second intermediate features; activating the second intermediate feature through the activation layer to generate a third intermediate feature; The first feature and the third intermediate feature are superimposed on each other through the residual superposition layer to generate a second feature.

4. The method according to claim 1, wherein The determining of the target matrix according to the plurality of two-dimensional initial key points and the confidence levels corresponding to the plurality of two-dimensional initial key points includes: Binarizing the confidences corresponding to the plurality of two-dimensional initial key points according to a preset threshold, and generating a confidence vector according to the binarized confidences; A target matrix is ​​determined using the multiple two-dimensional initial key points and the confidence vector.

5. The method according to claim 1, characterized in that The image to be detected is a human body image. After inputting the two-dimensional completion key points into the conversion model and calling the conversion model to convert the two-dimensional completion key points into three-dimensional key points, the method further includes: Determining human body motion based on the three-dimensional key points of multiple consecutive frames; Execute a target task according to the human body action.

6. The method according to any one of claims 1 to 5, characterized in that The method further includes training the completion model, wherein the training the completion model includes: Obtain a sample data set, wherein the sample data set includes several groups of two-dimensional real key points; Randomly selecting a set of two-dimensional real key points from the sample data set, randomly setting the two-dimensional real key points, and obtaining a confidence vector; Multiplying the two-dimensional true key point and the confidence vector to obtain a key point matrix; Inputting the key point matrix into a first initial network to obtain two-dimensional completed key points, wherein the first initial network includes a first fully connected module, a feature extraction module, and a second fully connected module; Calculating a first loss value between the two-dimensional completed key point and the two-dimensional real key point; The first initial network is iteratively trained according to the first loss value to obtain a trained completion model.

7. The method according to claim 6, characterized in that The sample data set also includes three-dimensional real key points corresponding to the two-dimensional real key points. The method further includes training the conversion model, and the training of the conversion model includes: Inputting the two-dimensional completed key points into a second initial network to obtain three-dimensional output key points, wherein the structure of the second initial network is the same as that of the first initial network; Calculating a second loss value between the three-dimensional real key point and the three-dimensional output key point according to the confidence vector; The second initial network is iteratively trained according to the second loss value to obtain a trained conversion model.

8. A three-dimensional key point generation device, characterized in that: The device comprises: a data acquisition unit, configured to acquire an image to be detected, perform key point detection on the image to be detected, and obtain a plurality of two-dimensional initial key points corresponding to the image to be detected, as well as confidence levels corresponding to the plurality of two-dimensional initial key points; a data determining unit, configured to determine a target matrix based on the plurality of two-dimensional initial key points and the confidence levels corresponding to the plurality of two-dimensional initial key points; a completion processing unit, configured to input the target matrix into a completion model, and perform key point completion on the two-dimensional initial key points according to the target matrix to obtain two-dimensional completed key points; The conversion processing unit is used to input the two-dimensional completion key points into a conversion model, and call the conversion model to convert the two-dimensional completion key points into three-dimensional key points.

9. A robot, characterized in that: The robot comprises: processor, memory, input and output units, and buses; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 7.

Images