Three-dimensional posture classification method based on two-dimensional key points and related devices

Abstract

This application applies to the field of image processing technology and provides a method and related apparatus for three-dimensional pose classification based on two-dimensional key points, enabling the three-dimensional pose classification of an object to be obtained using only a single two-dimensional image of the target object. The method primarily comprises: obtaining two-dimensional key point information from a two-dimensional image of the target object using a pre-trained deep learning network; determining key point structural features of the target object based on the two-dimensional key point information; and inferring the three-dimensional pose classification of the target object using the key point structural features.

Description

Technical Field

[0001] The embodiments of the present application belong to the field of image processing technology, and in particular relate to a three-dimensional posture classification method based on two-dimensional key points and related devices. Background Art

[0002] Two-dimensional images captured by cameras lose depth information, making it impossible to directly perceive the three-dimensional pose of the target object when processing the two-dimensional images using computers. However, in industries such as security monitoring, autonomous driving, smart elderly care, aviation, and airports, there are numerous applications where computers are required to identify the three-dimensional pose of people, vehicles, and aircraft in two-dimensional images. There is even a need to identify the pose in physical space, such as the spatial orientation of corridors, tunnels, and roads. In order to enable computers to directly perceive the three-dimensional pose of target objects in combination with cameras, that is, to enable computer vision to accurately perceive the three-dimensional pose of target objects, the industry currently has two relatively mature three-dimensional pose estimation methods.

[0003] Direct estimation method: Estimates 3D key points directly from images through an end-to-end network. This method requires the input to be annotated with 3D annotations. This method has fewer constraints, but requires the input and training set to be annotated with 3D annotations. The application of the direct estimation method has very high requirements for the input camera and is costly. In addition, such datasets are rare and are all captured in a controlled environment, resulting in the trained models often not being applicable to actual scenarios.

[0004] Two-stage estimation method: first obtain the two-dimensional key point information of the target object, then input the two-dimensional key point information sequence into the subsequent network, and then "upgrade" it to three-dimensional key points, that is, output three-dimensional posture; this two-dimensional key point information sequence must contain additional information, such as three-dimensional annotations, two-dimensional key points at different times, two-dimensional key points at different perspectives, etc.; in applications based on human posture, you can also input the human body parameter model according to the two-dimensional human body key point information, and then project it to obtain the two-dimensional key points for fitting the human body model optimization method; in summary, the two-stage estimation method is to first obtain the two-dimensional key point information sequence, and then obtain the three-dimensional key point estimation. This method has good accuracy and good effect. It has wide requirements on the data set and a wide range of applications, but the second stage, that is, upgrading from two-dimensional key points to three-dimensional key points, requires a large amount of data, so the calculation is large and relatively slow. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a three-dimensional posture classification method based on two-dimensional key points and related devices, which realizes that only a two-dimensional image of the target object is needed to obtain the three-dimensional posture classification of the object. The learning and reasoning processes of the embodiments of the present application do not require the three-dimensional coordinate information of the key points of the target object. Compared with the direct estimation method and the two-stage estimation method, both of which require the three-dimensional coordinate information of the key points to complete the classification, the embodiments of the present application completely save the computational amount and cost required to obtain the three-dimensional data of the key points, and have good economy and practicality.

[0006] The first aspect of the present application provides a three-dimensional posture classification method based on two-dimensional key points, comprising:

[0007] Obtain 2D key point information in the 2D image of the target object through a pre-trained deep learning network;

[0008] Determining key point structural features of the target object based on the two-dimensional key point information;

[0009] The three-dimensional posture classification of the target object is inferred based on the key point structural features.

[0010] Optionally, the two-dimensional key point information includes two-dimensional key point coordinates of the target object;

[0011] Determining the key point structural features of the target object according to the two-dimensional key point information includes:

[0012] The relative structural features between the two-dimensional key point coordinates of the target object are used as the key point structural features of the target object.

[0013] Optionally, the relative structural features are characterized by a relative distance l i and angle a i Expressed as (a i ,l i ), using the relative structural features between the two-dimensional key point coordinates of the target object as the key point structural features of the target object includes:

[0014] Three two-dimensional key point coordinates are selected from all two-dimensional key point coordinates as a group to form several groups of two-dimensional key fixed combination sequences formed by the three two-dimensional key point coordinates, wherein the three two-dimensional key point coordinates of each group of two-dimensional key fixed combination sequences are converted into the key point structural features of the target object including:

[0015] Determine the coordinates of the three two-dimensional key points (X c ,Y c ), (X m ,Y m ), (X t ,Yt );

[0016] The (X c ,Y c ), the (X m ,Y m ), the (X t ,Y t ) is converted to (a i ,l i );

[0017] in,

[0018] c2m_x=X c -X m

[0019] c2m_y=Y c -Y m

[0020] t2m_x=X t -X m

[0021] t2m_y=Y t -Y m

[0022] c2m_deg=arctan(c2m_y / c2m_x)

[0023] t2m_deg=arctan(t2m_y / t2m_x)

[0024]

[0025] a i =(t2m_deg-c2m_deg) / D

[0026] l i =L*t2m_leg / (t2m_leg+c2m_leg)

[0027] The arctan represents an inverse tangent function, the D is a positive integer greater than 0, and the L is a positive integer greater than 0;

[0028] The (a i ,l i ) as the first key point structural feature;

[0029] The first key point structural feature is regarded as the key point structural feature.

[0030] Optionally, the two-dimensional key point information includes the body coordinates of the target object, and determining the key point structural features of the target object according to the two-dimensional key point information includes:

[0031] Using the normalized structural features of the body coordinates of the target object as the key point structural features of the target object;

[0032] The using the normalized structural feature of the body coordinates of the target object as the second key point structural feature of the target object includes:

[0033] Determine the reference point (X) from the two-dimensional key point coordinates according to the preset rules z ,Y z );

[0034] Calculating the average distance from all the two-dimensional key point coordinates to the reference point;

[0035] The average distance is taken as the reference length l a ;

[0036] All 2D keypoint coordinates (X i ,Y i ) is converted into a normalized matrix (a i ,l i );

[0037] in,

[0038]

[0039] a i =arctan((Y i -Y z ) / (X i -X z )) / D

[0040]

[0041] l i =L*l z / (l z +l a )

[0042] The arctan represents an inverse tangent function, the D is a positive integer greater than 0, and the L is a positive integer greater than 0;

[0043] The normalized matrix (a i ,l i ) as the second key point structural feature;

[0044] The second key point structural feature is regarded as the key point structural feature.

[0045] Optionally, the two-dimensional key point information includes the body coordinates of the target object, and determining the key point structural features of the target object according to the two-dimensional key point information includes:

[0046] Using the normalized structural features of the body coordinates of the target object as the key point structural features of the target object;

[0047] The first key point structural feature and the second key point structural feature are fused into one key point structural feature.

[0048] Optionally, a reference point (X z ,Y z )include:

[0049] Randomly select a two-dimensional key point coordinate from all the two-dimensional key point coordinates as the reference point (X z ,Y z );

[0050] or,

[0051] Select the center points of two 2D key point coordinates from all 2D key point coordinates as the reference point (X z ,Y z );

[0052] or,

[0053] Select the centroid of several two-dimensional key point coordinates as the reference point (X z ,Y z ).

[0054] Optionally, estimating the three-dimensional posture classification of the target object by using the key point structural features includes:

[0055] Pre-establishing mappings between different preset key point structural features of the target object and different three-dimensional posture classifications of the preset target object;

[0056] Calculating the correlation between the key point structural feature and the preset key point structural feature, wherein the correlation includes distance, similarity or activation degree;

[0057] If the degree of correlation between the key point structural feature and the preset key point structural feature exceeds a preset threshold, it is determined that the key point structural feature is consistent with the target preset key point structural feature, and the corresponding mapping of the target preset key point structural feature is determined to be the three-dimensional posture classification of the target object, and the target preset key point structural feature is one of the preset key point structural features.

[0058] The second aspect of the present application is a three-dimensional posture classification device based on two-dimensional key points, comprising:

[0059] A two-dimensional key point information acquisition module is used to obtain two-dimensional key point information in a two-dimensional image of a target object through a pre-trained deep learning network;

[0060] A key point structural feature determination module, configured to determine the key point structural features of the target object based on the two-dimensional key point information;

[0061] The three-dimensional posture classification module is used to infer the three-dimensional posture classification of the target object based on the key point structural features.

[0062] Optionally, the two-dimensional key point information includes two-dimensional key point coordinates of the target object;

[0063] When determining the key point structural features of the target object based on the two-dimensional key point information, the key point structural feature determination module is specifically configured to:

[0064] The relative structural features between the two-dimensional key point coordinates of the target object are used as the key point structural features of the target object.

[0065] Optionally, the relative structural features are characterized by a relative distance l i and angle a i Expressed as (a i ,l i ), the key point structural feature determination module is specifically configured to:

[0066] Three two-dimensional key point coordinates are selected from all two-dimensional key point coordinates as a group to form several groups of two-dimensional key fixed combination sequences formed by the three two-dimensional key point coordinates, wherein the three two-dimensional key point coordinates of each group of two-dimensional key fixed combination sequences are converted into the key point structural features of the target object including:

[0067] Determine the coordinates of the three two-dimensional key points (X c ,Y c ), (X m ,Y m ), (X t ,Y t );

[0068] The (X c ,Y c ), the (X m ,Y m ), the (X t ,Y t) is converted to (a i ,l i );

[0069] in,

[0070] c2m_x=X c -X m

[0071] c2m_y=Y c -Y m

[0072] t2m_x=X t -X m

[0073] t2m_y=Y t -Y m

[0074] c2m_deg=arctan(c2m_y / c2m_x)

[0075] t2m_deg=arctan(t2m_y / t2m_x)

[0076]

[0077] a i =(t2m_deg-c2m_deg) / D

[0078] l i =L*t2m_leg / (t2m_leg+c2m_leg)

[0079] The arctan represents an inverse tangent function, the D is a positive integer greater than 0, and the L is a positive integer greater than 0;

[0080] The (a i ,l i ) as the first key point structural feature;

[0081] The first key point structural feature is regarded as the key point structural feature.

[0082] Optionally, the two-dimensional key point information includes the body coordinates of the target object, and the key point structural feature determination module determines the key point structural features of the target object according to the two-dimensional key point information, specifically for:

[0083] Using the normalized structural feature of the body coordinates of the target object as the second key point structural feature of the target object;

[0084] When the key point structural feature determination module uses the normalized structural feature of the body coordinates of the target object as the second key point structural feature of the target object, it is specifically configured to:

[0085] Determine the reference point (X) from the two-dimensional key point coordinates according to the preset rules z ,Y z );

[0086] Calculating the average distance from all the two-dimensional key point coordinates to the reference point;

[0087] The average distance is taken as the reference length l a ;

[0088] All 2D keypoint coordinates (X i ,Y i ) is converted into a normalized matrix (a i ,l i );

[0089] in,

[0090]

[0091] a i =arctan((Y i -Y z ) / (X i -X z )) / D

[0092]

[0093] l i =L*l z / (l z +l a )

[0094] The arctan represents an inverse tangent function, the D is a positive integer greater than 0, and the L is a positive integer greater than 0;

[0095] The normalized matrix (a i ,l i ) as the second key point structural feature;

[0096] The second key point structural feature is regarded as the key point structural feature.

[0097] Optionally, the two-dimensional key point information includes the body coordinates of the target object, and the key point structural feature determination module determines the key point structural features of the target object according to the two-dimensional key point information, specifically for:

[0098] Using the normalized structural feature of the body coordinates of the target object as the second key point structural feature of the target object;

[0099] The first key point structural feature and the second key point structural feature are fused into one key point structural feature.

[0100] Optionally, the key point structural feature determination module determines the reference point (X z ,Y z ), specifically used for:

[0101] Randomly select a two-dimensional key point coordinate from all the two-dimensional key point coordinates as the reference point (X z ,Y z );

[0102] or,

[0103] Select the center points of two 2D key point coordinates from all 2D key point coordinates as the reference point (X z ,Y z );

[0104] or,

[0105] Select the centroid of several two-dimensional key point coordinates as the reference point (X z ,Y z ).

[0106] Optionally, when estimating the three-dimensional posture classification of the target object through the key point structural features, the three-dimensional posture classification module is specifically configured to:

[0107] Pre-establishing mappings between different preset key point structural features of the target object and different three-dimensional posture classifications of the preset target object;

[0108] Calculating the correlation between the key point structural feature and the preset key point structural feature, wherein the correlation includes distance, similarity or activation degree;

[0109] If the degree of correlation between the key point structural feature and the preset key point structural feature exceeds a preset threshold, it is determined that the key point structural feature is consistent with the target preset key point structural feature, and the corresponding mapping of the target preset key point structural feature is determined to be the three-dimensional posture classification of the target object, and the target preset key point structural feature is one of the preset key point structural features.

[0110] A third aspect of the present application provides an electronic device, including:

[0111] Processor, memory;

[0112] The processor is connected to the memory via a bus;

[0113] The memory stores a program;

[0114] When the processor executes the program stored in the memory, the method described in any one of the first aspects is implemented.

[0115] A fourth aspect of the present application provides a computer-readable storage medium, characterized in that instructions are stored in the computer storage medium, and when the instructions are executed on a computer, the computer executes any one of the methods described in the first aspect.

[0116] A fifth aspect of the present application provides a computer program product, which, when executed on a computer, enables the computer to perform the method as described in any one of the first aspects above.

[0117] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0118] The three-dimensional posture classification method based on two-dimensional key points of the embodiment of the present application obtains two-dimensional key point information in the two-dimensional image of the target object through a pre-trained deep learning network. The two-dimensional key point information includes key information that can reflect the three-dimensional posture of the target object; then determines the key point structural features of the target object based on the two-dimensional key point information. The key point structural features are key information that can directly reflect the three-dimensional posture of the target object; and then infers the three-dimensional posture classification of the target object through the key point structural features. It can be seen that the embodiment of the present application realizes that only one two-dimensional image of the target object is needed to obtain the three-dimensional posture classification of the object. The learning and reasoning processes of the embodiment of the present application do not require the three-dimensional coordinate information of the key points of the target object. Compared with the direct estimation method and the two-stage estimation method, which both require the three-dimensional coordinate information of the key points to complete the classification, the embodiment of the present application completely saves the computational amount and cost required to obtain the three-dimensional data of the key points, and has good economy and practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0119] Figure 1 This is a flow chart of an embodiment of a three-dimensional posture classification method based on two-dimensional key points of the present application;

[0120] Figure 2 This is a schematic structural diagram of an embodiment of a three-dimensional posture classification device based on two-dimensional key points of the present application;

[0121] Figure 3 Define schematic diagrams for key points on the human body;

[0122] Figure 4 This is a schematic diagram of the network model structure of Example 1 of the present application;

[0123] Figure 5 Schematic diagram of the model structure of the deep learning network for two-dimensional key point output in this application;

[0124] Figure 6 This is a schematic diagram of the relative structure of the three key points of Example 1 of this application;

[0125] Figure 7 This is a schematic diagram of the network model structure of Example 2 of this application;

[0126] Figure 8 This is a schematic diagram of the normalized expression of the ontology in Example 2 of this application;

[0127] Figure 9 This is a structural diagram of the electronic device according to the fourth embodiment of the present application. DETAILED DESCRIPTION

[0128] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0129] Example 1

[0130] See also Figure 1 , Figure 1 This is a flow chart of a three-dimensional posture classification method based on two-dimensional key points of a target object provided in the first embodiment of the present application. This embodiment is used to classify three-dimensional postures based on two-dimensional key point information of a target object in a two-dimensional image. The method can be performed by a three-dimensional posture classification device based on two-dimensional key points of a target object. The device can be implemented in software and / or hardware and can be integrated into an electronic device, which can be a mobile terminal or a computer. Figure 1 As shown, the present embodiment provides a 3D posture classification method based on 2D key points of a target object, which may include:

[0131] Identify the target object area where the target object is located from a two-dimensional image of the target object, that is, determine the area of ​​the target object to be classified, and obtain the two-dimensional key point information of the target object through a pre-trained deep learning network. The image can be an image taken by a camera or an image obtained from a video frame.

[0132] Among them, the area of ​​the target object to be classified can be determined by methods such as motion area segmentation or target size and target type to determine the target image. The so-called target image is the area occupied by the target object to be classified in the two-dimensional image of the target object. In the "bottom-up" key point acquisition deep network, the two-dimensional key point coordinates of the target object are determined by the target image area; in the "top-down" key point acquisition deep network, the image can be segmented by the target image area, and the target image area is sent to the deep network to determine the two-dimensional key point coordinates of the target image. The overall network model of this embodiment is shown in FIG. Figure 4 .

[0133] For the pre-training of the deep learning network, the same type of target object corresponds to multiple 3D postures to be classified and recognized. Therefore, multiple training images are required for each 3D posture. For these training images, the 2D coordinate positions of each key point in each image should be calibrated. For the same type of target object, the positions of each key point should be consistent. For example, for the key points of the human body, the industry mainstream key point definition can be used. Figure 3 The image and keypoint annotations are fed into a deep learning network for learning. The output is a heatmap of each keypoint, from which the 2D coordinates of each keypoint can be obtained. The 2D keypoint information of the target object is called a 2D keypoint heatmap.

[0134] Train a deep learning network that can output two-dimensional key points, or directly use a pre-trained network that has already been trained, such as a network that outputs human key points, see Figure 5 Using VGG19 as the base network, two identical feature maps are fed into the stage subnetwork. Each stage subnetwork contains two branch networks: one for predicting key points and one for predicting connections. Each branch network contains multiple convolutional layers with 3x3 and 1x1 convolution kernels, as well as a loss adjustment layer. The feature map output by each stage subnetwork is concat-operated with the base feature map as the feature map output of this stage and passed to the next stage subnetwork. Multiple stage subnetworks will be iterated, and the last stage subnetwork will output a heat map of the key points and a prediction of the connection method. During training, the key points and connection method predictions of each stage subnetwork are compared with the key points and connection methods annotated in the training samples to determine the loss value. The subnetwork then performs reverse parameter adjustments based on the loss value until training is complete. After completing the above-mentioned key point deep learning network pre-training, inputting an image of a target object can output a heat map of the key points of the target object.

[0135] For each key point, first find the peak value of each key point and its peak coordinates (X p ,Y p ); Each keypoint has at least one peak coordinate.

[0136] This embodiment can use three key points to express the relative structure, see Figure 6 , define the three-point combination sequence in advance according to the typical characteristics of the object, that is, to clearly know which three-point combination can better capture the structural changes, and optionally, all three key points can be arranged and combined.

[0137] For the key point c, take the heat map of the key point c and take its coordinates as (X c ,Y c ), take out a sequence from the three key point sequences related to it, such as [key point c, key point m, key point t], and obtain the peak coordinates of the other two key points (X m ,Y m ),(X t ,Y t ), use relative distance and angle to express, that is, (X c ,Y c ), (X m ,Y m ), (X t ,Y t ) to (a i ,l i ), the conversion formula is as follows:

[0138] c2m_x=X c -X m

[0139] c2m_y=Y c -Y m

[0140] t2m_x=X t -X m

[0141] t2m_y=Y t -Y m

[0142] c2m_deg=arctan(c2m_y / c2m_x)

[0143] t2m_deg=arctan(t2m_y / t2m_x)

[0144]

[0145] a i =(t2m_deg-c2m_deg) / D

[0146] l i =L*t2m_leg / (t2m_leg+c2m_leg)

[0147] Wherein, arctan represents the inverse tangent function; D is a positive integer greater than 0. In this embodiment, D can adopt a precision of 5. The smaller D is, the higher the precision is. In actual application, it can be configured, for example, to configure D to 2, 3, etc.; L is a positive integer greater than 0. In this embodiment, L adopts 64. The larger the L value is, the higher the precision range is. In actual application, it can be configured, for example, to configure L to 128, 256, etc. Figure 2 The three combined coordinates of the dimensional key point are converted into a new structural feature map. In order to facilitate the subsequent processing with the 3D posture classification deep network, a i It must be an integer and is limited to 0-71, with every 5 degrees corresponding to an integer. i It must also be an integer and limited to 0-63. It traverses all sequences of key point c and completes the conversion of the key point c heat map into a new structural feature map; then it traverses all key points according to the above steps. The number of key points is n, and the heat maps of all key points are converted into new structural feature maps. The dimension of the new structural feature map is [72, 64, n]. The three-dimensional posture classification deep learning network is trained. In this embodiment, VGG19 is used as the basic network. The three-dimensional posture classification deep learning network outputs a vector and sends it to the softmax layer for classification. During training, the structural feature map is sent to the posture representation network. The softmax classification layer compares it with the three-dimensional posture classification code marked in the training sample to determine the difference loss value. Then, the three-dimensional posture classification deep learning network and the softmax classification layer perform reverse parameter adjustment according to the difference loss value until the training is completed. After completing the above network pre-training, the prediction of the three-dimensional posture classification can be output.

[0148] It can be seen that the embodiment of the present application realizes that only one two-dimensional image of the target object is needed to obtain the three-dimensional posture classification of the object. The learning and reasoning processes of the embodiment of the present application do not require the three-dimensional coordinate information of the key points of the target object. Compared with the direct estimation method and the two-stage estimation method, both of which require the three-dimensional coordinate information of the key points to complete the classification, the embodiment of the present application completely saves the computational amount and cost required to obtain the three-dimensional data of the key points, and has good economy and practicality.

[0149] Example 2

[0150] This embodiment is an improvement and perfection of the first embodiment. The first embodiment uses a three-keypoint relative expression structure. This expression structure is very sensitive to changes in the depth direction because of its invariance to translation and rotation, but not so sensitive to changes in the horizontal and vertical directions. In order to achieve better three-dimensional posture perception and accurate perception in the horizontal, vertical and depth directions, this embodiment adds a body coordinate normalization structure expression method. The network structure model is shown in Figure 7, in order to better provide a 3D posture classification method based on 2D key points of a target object, the method may include:

[0151] Identify the target object area where the target object is located from a two-dimensional image of the target object, that is, determine the area of ​​the target object to be classified, and obtain the two-dimensional key point information of the target object through a pre-trained deep learning network. The image can be an image taken by a camera or an image obtained from a video frame.

[0152] Among them, the area of ​​the target object to be classified can be determined by methods such as motion area segmentation or target size and target type to determine the target image. The so-called target image is the area occupied by the target object to be classified in the two-dimensional image of the target object. In the "bottom-up" key point acquisition deep network, the two-dimensional key point coordinates of the target object are determined by the target image area; in the "top-down" key point acquisition deep network, the image can be segmented by the target image area, and the target image area is sent to the deep network to determine the two-dimensional key point coordinates of the target image. The overall network model of this embodiment is shown in FIG. Figure 4 .

[0153] For the pre-training of the deep learning network, the same type of target object corresponds to multiple 3D postures to be classified and recognized. Therefore, multiple training images are required for each 3D posture. For these training images, the 2D coordinate positions of each key point in each image should be calibrated. For the same type of target object, the positions of each key point should be consistent. For example, for the key points of the human body, the industry mainstream key point definition can be used. Figure 3 The image and keypoint annotations are fed into a deep learning network for learning. The output is a heatmap of each keypoint, from which the 2D coordinates of each keypoint can be obtained. The 2D keypoint information of the target object is called a 2D keypoint heatmap.

[0154] Train a deep learning network that can output two-dimensional key points, or directly use a pre-trained network that has already been trained, such as a network that outputs human key points, see Figure 5Using VGG19 as the base network, two identical feature maps are fed into the stage subnetwork. Each stage subnetwork contains two branch networks: one for predicting key points and one for predicting connections. Each branch network contains multiple convolutional layers with 3x3 and 1x1 convolution kernels, as well as a loss adjustment layer. The feature map output by each stage subnetwork is concat-operated with the base feature map as the feature map output of this stage and passed to the next stage subnetwork. Multiple stage subnetworks will be iterated, and the last stage subnetwork will output a heat map of the key points and a prediction of the connection method. During training, the key points and connection method predictions of each stage subnetwork are compared with the key points and connection methods annotated in the training samples to determine the loss value. The subnetwork then performs reverse parameter adjustments based on the loss value until training is complete. After completing the above-mentioned key point deep learning network pre-training, inputting an image of a target object can output a heat map of the key points of the target object.

[0155] For each key point, first find the peak value of each key point and its peak coordinates (X p ,Y p ); Each keypoint has at least one peak coordinate.

[0156] First, the relative distance and relative angle between the three key points are used to express the two-dimensional key point structure. The three-point combination sequence is defined in advance according to the typical characteristics of the object, that is, it is necessary to clearly determine which three-point combination can better capture the structural changes. Optionally, all three key points can be arranged and combined.

[0157] For the key point c, take the heat map of the key point c and take its coordinates as (X c ,Y c ), take out a sequence from the three key point sequences related to it, such as [key point c, key point m, key point t], and obtain the peak coordinates of the other two key points (X m ,Y m ),(X t ,Y t ), use relative distance and angle to express, that is, (X c ,Y c ), (X m ,Y m ), (X t ,Y t ) to (a i ,l i ), the conversion formula is as follows:

[0158] c2m_x=X c -X m

[0159] c2m_y=Y c -Y m

[0160] t2m_x=X t -X m

[0161] t2m_y=Y t -Y m

[0162] c2m_deg=arctan(c2m_y / c2m_x)

[0163] t2m_deg=arctan(t2m_y / t2m_x)

[0164]

[0165] a i =(t2m_deg-c2m_deg) / D

[0166] l i =L*t2m_leg / (t2m_leg+c2m_leg)

[0167] Wherein, arctan represents the inverse tangent function; D is a positive integer greater than 0. In this embodiment, D can adopt a precision of 5. The smaller D is, the higher the precision is. In actual application, it can be configured, for example, to configure D to 2, 3, etc.; L is a positive integer greater than 0. In this embodiment, L adopts 64. The larger the L value is, the higher the precision range is. In actual application, it can be configured, for example, to configure L to 128, 256, etc. Figure 2 The three combined coordinates of the 3D key point are converted into a new structural feature map. In order to facilitate the subsequent processing with the 3D posture classification deep network, a i It must be an integer and is limited to 0-71, with every 5 degrees corresponding to an integer. i It must also be an integer and limited to 0-63. It traverses all sequences of key point c and completes the conversion of the key point c heat map into a new structural feature map. Then, it traverses all key points according to the above steps. The number of key points is n, and the heat maps of all key points are converted into new structural feature maps. The dimension of the new structural feature map is [72, 64, n].

[0168] Secondly, this embodiment adds the normalization of the body coordinates to convert the two-dimensional key point heat map into a new structural feature map, see Figure 8 , assuming that for the last n key points, the peak coordinates of key point 1 are selected as the reference coordinates (X z ,Y z), substitute the peak coordinates of all key points into the following formula according to the ascending sequence of key points 1 to n to calculate the average distance l a As a reference length, it is as follows:

[0169]

[0170] There is already a reference point (X z ,Y z ), and the reference length l a , except for the coordinates of the reference point, i.e., key point 1, each coordinate (X i ,Y i ) is substituted into the following formula, and the coordinate value, i.e., the confidence, remains unchanged. Then, the heat map of the key point i can be converted to the expression after the normalization of the body coordinates (a i ,l i ) feature map, that is, the normalized feature map, in order to facilitate the subsequent processing with the posture classification deep network, a i It must be an integer and is limited to 0-71, with every 5 degrees corresponding to an integer. i It must also be an integer and must be between 0 and 63.

[0171] a i =arctan((Y i -Y z ) / (X i -X z )) / 5

[0172]

[0173] l i =64*l z / (l z +l a )

[0174] According to the above steps, traverse all other key points, the number of key points is n-1, and convert the heat map of all key points into a new structural feature map. The dimension of the new structural feature map is [72, 64, n-1].

[0175] The posture representation deep learning network is trained. This embodiment uses VGG19 as the basic network and adds a feature merging layer. It can use methods such as residual attention network or fully connected network to fuse the posture representation network output of the normalized structural expression of the body coordinates and the posture representation network output of the relative structural expression of the three key points. In this embodiment, the output vectors of the two networks are concat into one vector, and then sent to the softmax layer for classification. During training, the structural feature map is sent to the posture representation network. The softmax classification layer will compare it with the posture classification code marked in the training sample to determine the difference loss value. Then the posture representation network subnet, feature merging layer and softmax classification layer will reversely adjust the parameters according to the difference loss value until the training is completed. After completing the above-mentioned network pre-training, the posture classification prediction can be output.

[0176] According to the technical solution of the embodiment of this application, only a 2D image of the target object is needed to obtain the 3D pose classification of the object, and the 3D coordinate information of the key points is not required for both the learning and reasoning processes. Compared with the direct estimation method and the two-stage estimation method, which both require the 3D coordinate information of the key points to complete the classification, this method completely saves the computational effort and cost of obtaining the 3D data of the key points, and is very cost-effective and practical, and has better accuracy than the first embodiment.

[0177] Example 3

[0178] Figure 2 This is a structural diagram of a three-dimensional posture classification device based on two-dimensional key points of a target object provided in the third embodiment of this application, which can execute the three-dimensional posture classification method provided in the embodiment of this application to achieve the corresponding functional modules and effects of the execution method. Figure 2 As shown, the three-dimensional posture classification device 200 based on two-dimensional key points may include:

[0179] A two-dimensional key point information acquisition module 201 is used to acquire two-dimensional key point information in a two-dimensional image of a target object through a pre-trained deep learning network;

[0180] A key point structural feature determination module 202 is configured to determine the key point structural features of the target object based on the two-dimensional key point information;

[0181] The 3D pose classification module 203 is configured to infer the 3D pose classification of the target object based on the key point structural features, for example, by inputting the key point structural features into a pre-trained 3D pose classification deep learning network to determine the 3D pose classification output of the target object.

[0182] Optionally, the two-dimensional key point information includes the body coordinates of the target object, and the two-dimensional key point information includes the two-dimensional key point coordinates of the target object;

[0183] When determining the key point structural features of the target object based on the two-dimensional key point information, the key point structural feature determination module 202 is specifically configured to:

[0184] The relative structural features between the two-dimensional key point coordinates of the target object are used as the key point structural features of the target object.

[0185] Optionally, the relative structural features are characterized by a relative distance l i and angle a i Expressed as (a i ,l i ), the key point structural feature determining module 202, when using the relative structural features between the two-dimensional key point coordinates of the target object as the key point structural features of the target object, is specifically configured to:

[0186] Three two-dimensional key point coordinates are selected from all two-dimensional key point coordinates as a group to form several groups of two-dimensional key fixed combination sequences formed by the three two-dimensional key point coordinates, wherein the three two-dimensional key point coordinates of each group of two-dimensional key fixed combination sequences are converted into the key point structural features of the target object including:

[0187] Determine the coordinates of the three two-dimensional key points (X c ,Y c ), (X m ,Y m ), (X t ,Y t );

[0188] The (X c ,Y c ), the (X m ,Y m ), the (X t ,Y t ) is converted to (a i ,l i );

[0189] in,

[0190] c2m_x=X c -X m

[0191] c2m_y=Y c -Y m

[0192] t2m_x=X t -X m

[0193] t2m_y=Y t -Ym

[0194] c2m_deg=arctan(c2m_y / c2m_x)

[0195] t2m_deg=arctan(t2m_y / t2m_x)

[0196]

[0197] a i =(t2m_deg-c2m_deg) / D

[0198] l i =L*t2m_leg / (t2m_leg+c2m_leg)

[0199] The arctan represents an inverse tangent function, the D is a positive integer greater than 0, and the L is a positive integer greater than 0;

[0200] The (a i ,l i ) as the first key point structural feature;

[0201] The first key point structural feature is regarded as the key point structural feature.

[0202] Optionally, the two-dimensional key point information includes the body coordinates of the target object, and the key point structural feature determination module 202 is specifically configured to:

[0203] Using the normalized structural feature of the body coordinates of the target object as the second key point structural feature of the target object;

[0204] When the key point structural feature determination module 202 uses the normalized structural feature of the body coordinates of the target object as the second key point structural feature of the target object, it is specifically configured to:

[0205] Determine the reference point (X) from the two-dimensional key point coordinates according to the preset rules z ,Y z );

[0206] Calculating the average distance from all the two-dimensional key point coordinates to the reference point;

[0207] The average distance is taken as the reference length l a ;

[0208] All 2D keypoint coordinates (X i ,Y i ) is converted into a normalized matrix (ai ,l i );

[0209] in,

[0210]

[0211] a i =arctan((Y i -Y z ) / (X i -X z )) / D

[0212]

[0213] l i =L*l z / (l z +l a )

[0214] The arctan represents an inverse tangent function, the D is a positive integer greater than 0, and the L is a positive integer greater than 0;

[0215] The normalized matrix (a i ,l i ) as the second key point structural feature;

[0216] The second key point structural feature is regarded as the key point structural feature.

[0217] Optionally, the two-dimensional key point information includes the body coordinates of the target object, and the key point structural feature determination module 202 is specifically configured to:

[0218] Using the normalized structural feature of the body coordinates of the target object as the second key point structural feature of the target object;

[0219] The first key point structural feature and the second key point structural feature are fused into one key point structural feature.

[0220] Optionally, the key point structural feature determination module 202 determines the reference point (X z ,Y z ), specifically used for:

[0221] Randomly select a two-dimensional key point coordinate from all the two-dimensional key point coordinates as the reference point (X z ,Y z );

[0222] or,

[0223] Select the center points of two 2D key point coordinates from all 2D key point coordinates as the reference point (X z ,Y z );

[0224] or,

[0225] Select the centroid of several two-dimensional key point coordinates as the reference point (X z ,Y z ).

[0226] Optionally, when estimating the three-dimensional posture classification of the target object by using the key point structural features, the three-dimensional posture classification module 203 is specifically configured to:

[0227] Pre-establishing mappings between different preset key point structural features of the target object and different three-dimensional posture classifications of the preset target object;

[0228] Calculating the correlation between the key point structural feature and the preset key point structural feature, wherein the correlation includes distance, similarity or activation degree;

[0229] If the degree of correlation between the key point structural feature and the preset key point structural feature exceeds a preset threshold, it is determined that the key point structural feature is consistent with the target preset key point structural feature, and the corresponding mapping of the target preset key point structural feature is determined to be the three-dimensional posture classification of the target object, and the target preset key point structural feature is one of the preset key point structural features.

[0230] It can be seen that the embodiment of the present application realizes that only one two-dimensional image of the target object is needed to obtain the three-dimensional posture classification of the object. The learning and reasoning processes of the embodiment of the present application do not require the three-dimensional coordinate information of the key points of the target object. Compared with other devices that run direct estimation method and two-stage estimation method, which need to obtain the three-dimensional coordinate information of the key points to complete classification, the embodiment of the present application completely saves the computational amount and cost required to obtain the three-dimensional data of the key points, and has good economy and practicality.

[0231] Example 4

[0232] The present application also provides an electronic device, such as Figure 9As shown, a block diagram of an electronic device for a three-dimensional posture classification method based on two-dimensional key points of a target object provided in Example 4 of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0233] like Figure 9 As shown, the electronic device includes: one or more processors 901, a memory 902, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 9In the example, a processor 901 is used. Memory 902 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor, causing the at least one processor to perform the method for classifying the three-dimensional posture of two-dimensional key points provided in this application. The computer-readable storage medium of this application stores computer instructions for causing a computer to perform the method for classifying the three-dimensional posture of two-dimensional key points provided in this application. Memory 902, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the method for classifying the three-dimensional posture of two-dimensional key points in the embodiments of this application. Processor 901 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in memory 902, thereby implementing the method for classifying the three-dimensional posture of two-dimensional key points in the above-mentioned method embodiment. Memory 902 can include a program storage area and a data storage area. The program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created by the use of an electronic device based on the method for classifying the three-dimensional posture of two-dimensional key points. In addition, the memory 902 may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 902 may optionally include a memory remotely located relative to the processor 901, and these remote memories may be connected to the electronic device of the three-dimensional posture classification method of two-dimensional key points via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0234] An electronic device for a three-dimensional posture classification method based on two-dimensional key points may further include: a wireless network interface 903 and an input / output interface 904. The processor 901, the memory 902, the wireless network interface 903 and the input / output interface 904 may be connected to the processor 901 via a bus or other means. Figure 9 The bus connection is taken as an example. The input and output interface 904 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device based on the three-dimensional posture classification method of the key points of the two-dimensional image of the target object, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, an indicator stick, one or more mouse buttons, a trackball, a joystick and other input devices. The input and output interface 904 can be connected to a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0235] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementation in one or more computer programs that can be executed and / or interpreted on a programmable system comprising at least one programmable processor, which can be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device. These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for the programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., a disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor. To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input). The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with embodiments of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by digital data communication (e.g., a communication network) in any form or medium. Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network. The computer system can include clients and servers.A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0236] The device according to the technical solution of the embodiment of the present application only requires a single 2D image of the target object to classify its 3D pose, without requiring the 3D coordinates of key points during both the learning and reasoning processes. Compared to devices implementing direct estimation and two-stage estimation methods, which both require the 3D coordinates of key points for classification, this method completely eliminates the computational effort and cost of acquiring 3D data of key points, resulting in excellent economical and practical performance.

[0237] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0238] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements or improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A three-dimensional posture classification method based on two-dimensional key points, characterized in that: include: Obtain 2D key point information in the 2D image of the target object through a pre-trained deep learning network; Determining key point structural features of the target object based on the two-dimensional key point information; Inferring the three-dimensional posture classification of the target object through the key point structural features; The two-dimensional key point information includes the two-dimensional key point coordinates of the target object; Determining the key point structural features of the target object according to the two-dimensional key point information includes: Using relative structural features between the two-dimensional key point coordinates of the target object as key point structural features of the target object; The relative structural features are characterized by the relative distance l i and angle a i Expressed as (a i ,l i ), using the relative structural features between the two-dimensional key point coordinates of the target object as the key point structural features of the target object includes: Three two-dimensional key point coordinates are selected from all two-dimensional key point coordinates as a group to form several groups of two-dimensional key fixed combination sequences formed by the three two-dimensional key point coordinates, wherein the three two-dimensional key point coordinates of each group of two-dimensional key fixed combination sequences are converted into the key point structural features of the target object including: Determine the coordinates of the three two-dimensional key points (X c ,Y c ), (X m ,Y m ), (X t ,Y t ); The (X c ,Y c ), the (X m ,Y m ), the (X t ,Y t ) is converted to (a i ,l i ); in, c2m_x=X c -X m c2m_y=Y c -AND m t2m_x=X t -X m t2m_y=Y t -AND m c2m_deg=arctan(c2m_y / c2m_x) t2m_deg=arctan(t2m_y / t2m_x) a i =(t2m_deg-c2m_deg) / D l i =L*t2m_leg / (t2m_leg+c2m_leg) The arctan represents an inverse tangent function, the D is a positive integer greater than 0, and the L is a positive integer greater than 0; The (a i ,l i ) as the first key point structural feature; The first key point structural feature is regarded as the key point structural feature.

2. The three-dimensional posture classification method based on two-dimensional key points according to claim 1 is characterized in that: The two-dimensional key point information includes the body coordinates of the target object, and determining the key point structural features of the target object based on the two-dimensional key point information includes: Using the normalized structural feature of the body coordinates of the target object as the second key point structural feature of the target object; The using the normalized structural feature of the body coordinates of the target object as the second key point structural feature of the target object includes: Determine the reference point (X) from the two-dimensional key point coordinates according to the preset rules z ,Y z ); Calculating the average distance from all the two-dimensional key point coordinates to the reference point; The average distance is taken as the reference length l a ; All 2D keypoint coordinates (X i ,Y i ) is converted into a normalized matrix (a i ,l i ); in, to i =arctan((Y i -AND z ) / (X i -X z )) / D l i ＝L*l z / (l z +l a ) The arctan represents an inverse tangent function, the D is a positive integer greater than 0, and the L is a positive integer greater than 0; The normalized matrix (a i ,l i ) as the second key point structural feature; The second key point structural feature is regarded as the key point structural feature.

3. The three-dimensional posture classification method based on two-dimensional key points according to claim 1, characterized in that: The two-dimensional key point information includes the body coordinates of the target object, and determining the key point structural features of the target object according to the two-dimensional key point information includes: Using the normalized structural feature of the body coordinates of the target object as the second key point structural feature of the target object; The first key point structural feature and the second key point structural feature are fused into one key point structural feature.

4. The three-dimensional posture classification method based on two-dimensional key points according to claim 2, characterized in that: Determine the reference point (X) from the two-dimensional key point coordinates according to the preset rules z ,Y z )include: Randomly select a two-dimensional key point coordinate from all the two-dimensional key point coordinates as the reference point (X z ,Y z ); or, Select the center points of two 2D key point coordinates from all 2D key point coordinates as the reference point (X z ,Y z ); or, Select the centroid of several two-dimensional key point coordinates as the reference point (X z ,Y z ).

5. The three-dimensional posture classification method based on two-dimensional key points according to claim 1, characterized in that: Estimating the three-dimensional posture classification of the target object by using the key point structural features includes: Pre-establishing mappings between different preset key point structural features of the target object and different three-dimensional posture classifications of the preset target object; Calculating the correlation between the key point structural feature and the preset key point structural feature, wherein the correlation includes distance, similarity or activation degree; If the degree of correlation between the key point structural feature and the preset key point structural feature exceeds a preset threshold, it is determined that the key point structural feature is consistent with the target preset key point structural feature, and the corresponding mapping of the target preset key point structural feature is determined to be the three-dimensional posture classification of the target object, and the target preset key point structural feature is one of the preset key point structural features.

6. A three-dimensional posture classification device based on two-dimensional key points, characterized in that: include: A two-dimensional key point information acquisition module is used to obtain two-dimensional key point information in a two-dimensional image of a target object through a pre-trained deep learning network; A key point structural feature determination module, configured to determine the key point structural features of the target object based on the two-dimensional key point information; A three-dimensional posture classification module, used to infer the three-dimensional posture classification of the target object based on the key point structural features; The two-dimensional key point information includes the two-dimensional key point coordinates of the target object; When determining the key point structural features of the target object based on the two-dimensional key point information, the key point structural feature determination module is specifically configured to: Using relative structural features between the two-dimensional key point coordinates of the target object as key point structural features of the target object; The relative structural features are characterized by the relative distance l i and angle a i Expressed as (a i ,l i ), the key point structural feature determination module is specifically configured to: Three two-dimensional key point coordinates are selected from all two-dimensional key point coordinates as a group to form several groups of two-dimensional key fixed combination sequences formed by the three two-dimensional key point coordinates, wherein the three two-dimensional key point coordinates of each group of two-dimensional key fixed combination sequences are converted into the key point structural features of the target object including: Determine the coordinates of the three two-dimensional key points (X c ,Y c ), (X m ,Y m ), (X t ,Y t ); The (X c ,Y c ), the (X m ,Y m ), the (X t ,Y t ) is converted to (a i ,l i ); in, c2m_x=X c -X m c2m_y=Y c -AND m t2m_x=X t -X m t2m_y=Y t -AND m c2m_deg=arctan(c2m_y / c2m_x) t2m_deg=arctan(t2m_y / t2m_x) a i =(t2m_deg-c2m_deg) / D l i =L*t2m_leg / (t2m_leg+c2m_leg) The arctan represents an inverse tangent function, the D is a positive integer greater than 0, and the L is a positive integer greater than 0; The (a i ,l i ) as the first key point structural feature; The first key point structural feature is regarded as the key point structural feature.

7. An electronic device, characterized in that: include: Processor, memory; The processor is connected to the memory via a bus; The memory stores a program; When the processor executes the program stored in the memory, the method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, characterized in that The computer storage medium stores instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 1 to 5.

Images