Single person prone 2D posture recognition method based on joint feature encoding in depth image

Abstract

A method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding comprises the following steps: firstly, the depth image is normalized, and then the true value of the joint point coordinates in the depth image data set is feature coded, wherein the coding is divided into confidence region feature coding, offset information coding and scale coding; after combining the above feature coding information, the output of a convolutional neural network is formed; then, the network structure is designed, wherein the network structure includes a general network structure module and a custom network structure module, and the network training is realized by selecting a loss function, and after decoding and interpolating the result of the network output, a corresponding high-resolution thermal map of each joint point can be obtained, and then the positioning coordinates of each joint point can be obtained to form a human skeleton in a prone position, thereby realizing human two-dimensional posture recognition; the present invention can realize single person prone two-dimensional posture recognition based on single person joint point feature coding and only with a depth image as input.

Description

Technical Field

[0001] The invention belongs to the technical field of human posture estimation, and in particular relates to a method for recognizing a single-person lying two-dimensional posture in a depth image based on joint feature coding. Background Art

[0002] Human posture recognition is increasingly being used in the fields of medical rehabilitation, sports health, human-computer interaction, etc. Current human posture recognition methods can be mainly divided into color image-based and depth image-based input methods. Human posture recognition methods based on color images have large-scale posture data sets with mature annotations, which can be easily used for various deep network training. However, color images contain privacy information such as human face and skin texture, which is not conducive to direct use in medical and other fields. Compared with color images, depth images are not sensitive to information such as texture and lighting, so these privacy information can be avoided during the acquisition process.

[0003] Although there are some human pose recognition methods based on depth images, these methods ([1] Pons-Moll G, Taylor J, Shotton J, et al. Metric regression forests for correspondence estimation [J]. International journal of computer vision, 2015, 113 (3): 163-175. [2] Shotton J, Sharp T, Kipman A, et al. Real-time human pose recognition in parts from single depth images [J]. Communications of the ACM, 2013, 56 (1): 116-124. [3] Shotton J, Fitzgibbon A, Cook M, et al. Real-time human pose recognition in parts from single depth images [C]: IEEE, 2011: 1297-1304 [4] Wu Q, Xu G, Zhang S, et al. Human 3D pose estimation in a lying position by RGB-D images for medical diagnosis and rehabilitation [C]. 2020 42nd Annual International Conference ofthe IEEE Engineering in Medicine&Biology Society(EMBC).IEEE,2020:5802-5805.) can only recognize the posture of the human body in a standing position or needs to be combined with a color image to recognize the posture of the human body in a lying position. In the diagnosis and rehabilitation applications in hospitals, patients are in a lying position most of the time, and it is impossible to directly use the existing methods to rely only on depth images for posture recognition. Therefore, a method for recognizing the lying posture of the human body that can rely only on depth image input is urgently needed. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a single-person lying two-dimensional posture recognition method based on joint feature coding, which can realize single-person lying two-dimensional posture recognition based on single-person joint point feature coding and only with depth image as input, so as to meet the needs of behavioral detection and analysis of human lying posture in privacy-sensitive fields such as medical rehabilitation.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding comprises the following steps:

[0007] Step 1: Depth image normalization;

[0008] Step 2: Position and encode the joint point features;

[0009] Step 3: Combine joint feature encoding;

[0010] Step 4: Network structure design;

[0011] Step 5: Select the loss function;

[0012] Step 6: Deep network training process;

[0013] Step 7: Joint point positioning feature decoding;

[0014] Step 8: Generate the human body skeleton in lying position.

[0015] The normalization of the depth image in step 1 is determined by formula (1):

[0016]

[0017] In the formula, I d (i, j) is the normalized corresponding depth image I of pixel (i, j) d , D(i,j) is the unnormalized depth value of the input pixel (i,j), mean(D) is the mean of the pixel values ​​of the unnormalized depth image, and K is the selected human range threshold.

[0018] The K value in formula (1) in step 1 is determined by the specific collection object. When the human body is an adult, the K value is large; when the human body is an infant or a child, the K value is small.

[0019] The positioning code information in step 2 includes a confidence region code, and the confidence region range is determined by formula (2):

[0020] {φ(U1,V1,r)||(i,j)-(U1,V1)|≤r} (2)

[0021] Where (U1, V1) is the coordinate of the target key point, r is its nearby radius, and (i, j) is the pixel point in the surrounding range.

[0022] The positioning coding information in step 2 includes a confidence region code, and the true value of the code is 0 or 1, representing the probability that the pixel point belongs to the confidence region.

[0023] The positioning coding information in step 2 includes positioning coordinate information, and the positioning coordinate information is determined by the offset of pixels within the confidence range {φ} relative to the target key point.

[0024] The positioning coding information in step 2 includes human scale information, and the method for determining the human scale information is determined by formula (4):

[0025]

[0026] Where: s i,j is the individual scale in the image, {U} is the x-coordinate set of key points, and {V} is the y-coordinate set of key points.

[0027] The number of channels of the depth image input in the step 3 is not fixed, and the number of channels can be customized, which can be 1 channel or more channels.

[0028] The network unit in step 4 is composed of a convolutional neural network, but the structure is not fixed and can be freely replaced according to requirements such as accuracy and efficiency.

[0029] The output of the network unit in step 4 is divided into two categories, the first category is the confidence region classification probability output of the pixel point, and the second category is the offset regression output of the pixel point.

[0030] The loss function of the confidence region classification part in step 5 is the cross entropy loss function, while the loss function of the offset output is determined by the absolute value of the difference between the true value and the output.

[0031] The overall loss function in step 5 is represented by the cross entropy loss function loss P1 And the loss function loss of the offset output P2 It is obtained by adding weights, and the weight value is determined after the network optimization parameters.

[0032] In the network training process in step 6, a pre-training process is first performed to fix the network structure model parameters of the general part, and only the parameters of the custom network part are trained. The number of training rounds in the pre-training process is not fixed.

[0033] The preliminary positioning decoding of the joint point in step 7 is determined by formula (7):

[0034]

[0035] Where: is the estimated xy coordinate of the target point, (i, j) is the corresponding pixel point in the confidence region {φ} of the target point, (x i,j ,y i,j ) is the offset of the corresponding pixel.

[0036] In the step 7, a heat map of the original input resolution size is obtained by interpolation.

[0037] Before generating the heat map of the original input resolution by using the position coding information of the key points in the step 7, the pixel values ​​between the two amplified points need to be set to 0, and then the pixel values ​​set to 0 are interpolated.

[0038] The specific formula for interpolation in step seven is determined as shown in formula (8):

[0039]

[0040] Where: f(N i ,N j ) is the coordinate (N i ,N j ), N is a template of appropriate size centered at coordinate (i, j), and the point (N i ,N j ) at the scale output s i,j The rounded value is used as the template size.

[0041] In the step eight, a heat map of the original input resolution size of the corresponding key points is obtained by interpolation, and non-maximum suppression is performed on the heat map to obtain the corresponding key point coordinates, and the corresponding key points are connected to obtain the two-dimensional skeleton of the human body in a recumbent position.

[0042] The number of all joint points connected in step eight is not fixed and is adjusted according to task requirements.

[0043] Once the number of joint points is determined in step eight, the output structure of the network can be adjusted to adapt to the task through step four, or the appropriate joint point true values ​​can be directly selected in step six to participate in training, or selective connections can be made when connecting the joint points.

[0044] The beneficial effects of the present invention are:

[0045] Since the present invention encodes the positioning features of the joint points, the encoding information includes the offset vector and the scale information, and the depth image of the lying posture is trained by the convolutional neural network, the present invention can realize the two-dimensional posture recognition of the lying human body based on the depth image;

[0046] During decoding, a high-resolution joint point estimation heat map can be obtained by formula interpolation, which avoids the phenomenon of precision loss caused by the output of a low-resolution heat map by a deep network. Therefore, the present invention can obtain a highly reliable recognition result. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Flow chart of the method of the present invention.

[0048] Figure 2 is a normalized depth image of the embodiment.

[0049] Figure 3 Schematic diagram of joint point feature positioning coding in the embodiment, Figure (a) is the confidence region coding, the pixel value of the shaded area is set to 1, and the rest is 0; Figure (b) is the offset coding.

[0050] Figure 4 This is an example of network structure design for an embodiment.

[0051] Figure 5 This is a thermal diagram of the knee joint at the original resolution of the embodiment.

[0052] Figure 6 It is the true value annotation of the joint points of different data sets in the embodiment.

[0053] Figure 7 This is the recognition result of the embodiment. DETAILED DESCRIPTION

[0054] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0055] Reference Figure 1 , a method for recognizing a single person's lying two-dimensional posture in a depth image based on joint feature coding, comprising the following steps:

[0056] Step 1: Depth image normalization. The specific implementation is as follows: the depth image is normalized according to formula (1):

[0057]

[0058] In the formula, I d (i, j) is the normalized corresponding depth image I of pixel (i, j) d The value of , D(i,j) is the depth value of the input unnormalized pixel (i,j), mean(D) is the mean value of the pixel value of the unnormalized depth image, K is the selected human range threshold, which can be determined by the specific acquisition object. When the human body is an adult, the K value can be slightly larger, and when the human body is an infant or a child, the K value can be set to a slightly smaller value. The normalized depth image can be represented by a single-channel grayscale image. Figure 2 shows an example of a normalized depth image;

[0059] Step 2: Position and encode the joint point features in the depth image posture dataset. The specific implementation is as follows: the pixel points (i, j) within a radius r around the target key point (U1, V1) are attributed to the confidence region {φ} of the target key point:

[0060] {φ(U1,V1,r)|||(i,j)-(U1,V1)||≤r} (2)

[0061] Set the confidence level c of the pixel points in the range {φ} that meets equation (2) i,j =1, the rest of the points are 0, the feature encoding example of the confidence region is as follows Figure 3 (a)

[0062] The positioning coordinate information of the key point is determined by the offset of the pixel within the confidence range {φ} relative to the target key point. The offset calculation formula is shown in formula (3):

[0063]

[0064] Where: (x i,j ,y i,j ) is the offset of pixel (i, j) relative to the target key point, (i, j) is the pixel coordinate within the confidence range {φ} of the target key (U1, V1), and the feature encoding example of the offset information is as follows Figure 3 (b)

[0065] The encoding of human scale is also considered. Adding scale information to the prior information is beneficial to the subsequent decoding and clustering of key points. i,j The calculation formula is as follows:

[0066]

[0067] Where: s i,j is the individual scale in the image, {U} is the x-coordinate set of key points, and {V} is the y-coordinate set of key points;

[0068] Step 3: Combine joint feature encoding; the specific implementation is as follows: first, the number of channels of the input depth image can be customized. Taking three channels as an example, the grayscale normalized single-channel depth image is expanded to three channels; for the depth image I input in the training set m×n×3 , where m and n correspond to the length and width of the depth image I, respectively, and 3 represents the number of input channels; the key point feature encoding is performed to obtain the true value encoding information corresponding to the output in the network structure {J(c i,j ,x i,j ,y i,j ,s i,j )}, where J is the key point coordinate information encoding, (i, j) is the coordinate representation in the image, (c i,j ,x i,j ,y i,j ,s i,j) represent the confidence that the point at (i, j) is a key point, the x and y offsets relative to the true value point, and the scale of the human body relative to the image;

[0069] Step 4: Network structure design; the specific implementation is as follows. In order to ensure network training, the network structure is also designed. The network structure is composed of stacked convolutional neural units. The intermediate structure of the network has various forms and is mainly composed of two modules, namely the general network structure module and the custom network structure module. The output of the network mainly includes two categories: the first category is the confidence region classification probability output c of the pixel point i,j The second type is the offset regression output of the pixel point (x i,j ,y i,j ) and scale output s i,j ;by Figure 4 Taking the network structure as an example, the general network structure module adopts the residual network structure, while the custom network structure module contains two types of outputs;

[0070] Step 5: Select a suitable loss function; the specific implementation is as follows: c in the positioning code J i,j Defined as the probability that the pixel point (i, j) belongs to the corresponding key point, then the loss function of this part is loss P1 The cross entropy loss function in the form of L1 norm can be selected; while the loss function loss of the rest of the positioning code J P2 Determined by formula (5);

[0071] loss P2 =|g Pi -o Pi |, (5) In the formula, g Pi is the true value offset part of the i-th pixel in the positioning code J, and o Pi Encode the network output portion corresponding to the pixel point;

[0072] Add the two loss functions according to certain weights to obtain the final loss of the network training, as shown in formula (6);

[0073] loss = αloss P1 +βloss P2 , (6)

[0074] In the formula, loss P1 is the location code J in c i,j The loss function definition, loss P2 is the loss function definition of the remaining corresponding part of the positioning code J, [α, β,] is the weight of the corresponding loss function, which is determined by the optimization during the network training process;

[0075] Step 6: Deep network training process; After the above steps, the network needs to be trained. The specific implementation is as follows: before the formal training, pre-training is first performed to fix the model parameters of the general network structure module and train the parameters of the custom network structure module. After several rounds of training (taking 3 rounds as an example), the general network structure module is unfrozen and the model parameters of the entire network are trained;

[0076] Step 7: Joint point positioning feature decoding; Specifically, the preliminary positioning decoding of key points is expressed as formula (7):

[0077]

[0078] Where: is the estimated xy coordinate of the target point, (i, j) is the corresponding pixel point in the confidence region {φ} of the target point, (x i,j ,y i,j ) is the offset of the corresponding pixel point; the heat map of the original input resolution is generated through the position encoding information of the key point, the pixel values ​​between the two points after amplification are set to 0, and then the pixels set to 0 are interpolated, and the interpolation formula is formula (8):

[0079]

[0080] Where: f(N i ,N j ) is the coordinate (N i ,N j ) is the value after the pixel point interpolation accumulation, N is the template of the appropriate size centered at the coordinate (i, j) (taking the point (N) as the center). i ,N j ) at the scale output s i,j The rounded value is used as the template size for example); Figure 5 This is a heat map of the interpolated knee joint position. Note Figure 5 The resolution of is consistent with the resolution of the input depth image;

[0081] Step 8: Generate a human skeleton in a supine position; Through the above steps, the heat map of the original input resolution size of the corresponding key points can be interpolated, and the non-maximum suppression of the heat map can be performed to obtain the corresponding key point coordinates. The corresponding key points are connected to obtain the two-dimensional skeleton of the human body in a supine position; Figure 6 As shown in the figure, according to different task requirements, there may be different definitions, numbers or connection methods of joint points, and the output number of joint points can be adjusted according to different task requirements; once the number of joint points is determined, the output structure of the network can be adjusted to adapt to the task through step 4, or the appropriate joint point truth value can be directly selected in step 6 to participate in the training, or selective connection can be made when connecting the joint points; Figure 7 The human body posture results of some depth images recognized by this embodiment.

Claims

1. A method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding, characterized in that: The following steps are involved: Step 1: Depth image normalization; Step 2: Position and encode the joint point features; Specifically, the pixel points (i, j) within a radius r around the target key point (U1, V1) are attributed to the confidence region {φ} of the target key point: {φ(U1,V1,r)|||(i,j)-(U1,V1)||≤r} (2) Set the confidence level c of the pixel points in the range {φ} that meets equation (2) i,j =1, the rest of the points are 0; The positioning coordinate information of the key point is determined by the offset of the pixel within the confidence range {φ} relative to the target key point. The offset calculation formula is shown in formula (3): Where: (x i,j ,y i,j ) is the offset of pixel (i, j) relative to the target key point, (i, j) is the pixel coordinate within the confidence range {φ} of the target key (U1, V1); The encoding of human scale is also considered, the scale s i,j The calculation formula is as follows: Where: s i,j is the individual scale in the image, {U} is the x-coordinate set of key points, and {V} is the y-coordinate set of key points; Step 3: Combine joint feature encoding; Specifically, the grayscale normalized single-channel depth image is expanded into multiple channels; for the depth image I input in the training set m×n×t , where m, n correspond to the length and width of the depth image I respectively, and t represents the number of input channels; The key point features are encoded to obtain the true value encoding information corresponding to the output in the network structure {J(c i,j ,x i,j ,y i,j ,s i,j )}, where J is the key point coordinate information encoding, (i, j) is the coordinate representation in the image, (c i,j ,x i,j ,y i,j ,s i,j ) represent the confidence that the point at (i, j) is a key point, the x and y offsets relative to the true value point, and the scale of the human body relative to the image; Step 4: Network structure design; Specifically, the network structure is composed of stacked convolutional neural units, and the intermediate structure of the network consists of two modules, namely the general network structure module and the custom network structure module; the output of the network includes two categories: the first category is the confidence region classification probability output c of the pixel point i,j The second type is the offset regression output of the pixel point (x i,j ,y i,j ) and scale output s i,j ; The general network structure module adopts a residual network structure, while the custom network structure module contains two types of outputs; Step 5: Select the loss function; Step 6: Deep network training process; Step 7: Joint point positioning feature decoding; Specifically, the initial positioning decoding of the key points is expressed as formula (7): Where: is the estimated xy coordinate of the target point, (i, j) is the corresponding pixel point in the confidence region {φ} of the target point, (x i,j ,y i,j ) is the offset of the corresponding pixel point; the heat map of the original input resolution is generated through the position encoding information of the key point, the pixel values ​​between the two points after amplification are set to 0, and then the pixels set to 0 are interpolated, and the interpolation formula is formula (8): Where: f(N i ,N j ) is the coordinate (N i ,N j ), where N is a template of appropriate size centered at coordinate (i, j); Step 8: Generate the human body skeleton in lying position.

2. A method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding as claimed in claim 1, characterized in that: The normalization of the depth image in step 1 is determined by formula (1): In the formula, I d (i, j) is the normalized corresponding depth image I of pixel (i, j) d , D(i,j) is the unnormalized depth value of the input pixel (i,j), mean(D) is the mean of the pixel values ​​of the unnormalized depth image, and K is the selected human range threshold.

3. A method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding as claimed in claim 2, characterized in that: The K value in formula (1) in step 1 is determined by the specific collection object. When the human body is an adult, the K value is large; when the human body is an infant or a child, the K value is small.

4. The method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding as claimed in claim 1, characterized in that: The number of channels of the depth image input in step 3 is not fixed, and the number of channels can be customized.

5. The method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding as claimed in claim 1, characterized in that: The network unit in step 4 is composed of a convolutional neural network, but the stacking structure is not fixed and can be freely replaced according to the requirements of accuracy and efficiency.

6. The method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding as claimed in claim 1, characterized in that: The loss function of the confidence region classification part in step 5 is the cross entropy loss function, while the loss function of the offset output is determined by the absolute value of the difference between the true value and the output.

7. A method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding as claimed in claim 6, characterized in that: The overall loss function in step 5 is represented by the cross entropy loss function loss P1 And the loss function loss of the offset output P2 It is obtained by adding weights, and the weight value is determined after the network optimization parameters.

8. The method for recognizing a single person's prone two-dimensional posture from a depth image based on joint feature coding as claimed in claim 1, characterized in that: In the network training process in step 6, a pre-training process is first performed to fix the network structure model parameters of the general part, and only the parameters of the custom network part are trained. The number of training rounds in the pre-training process is not fixed.

9. The method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding as claimed in claim 1, characterized in that: In the step eight, a heat map of the original input resolution size of the corresponding key points is obtained by interpolation, and non-maximum suppression is performed on the heat map to obtain the corresponding key point coordinates, and the corresponding key points are connected to obtain the two-dimensional skeleton of the human body in a recumbent position.

10. The method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding as claimed in claim 9, characterized in that: The number of all joint points connected in step eight is not fixed and is adjusted according to task requirements.

11. The method for recognizing a single person's prone two-dimensional posture in a depth image based on joint feature coding as claimed in claim 9, characterized in that: Once the number of joint points is determined in step eight, the output structure of the network can be adjusted to adapt to the task through step four, or the appropriate joint point true values ​​can be directly selected in step six to participate in training, or selective connections can be made when connecting the joint points.

Images