A human pose estimation method and device fusing vision and pressure, and a medium

Abstract

The present application relates to a kind of fusion vision and pressure human posture estimation method, equipment and medium, in the method, image data obtained using monocular camera and pressure data obtained using sensor, and by constructing fusion image feature and pressure feature human posture recognition model, based on deep learning and data fusion, realize the three-dimensional estimation of human posture.The present application compared with prior art, realizes more accurate 3D human posture estimation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision, and in particular to a human pose estimation method and device fusing vision and pressure, and a medium. BACKGROUND

[0002] In recent years, 3D human pose estimation technology has gradually shown its importance in various fields such as human-computer interaction, virtual reality, augmented reality, and medical rehabilitation. This technology can track and locate the poses of various joints of the human body in three-dimensional space, providing key support for real-time interaction, motion analysis, medical rehabilitation, and other applications.

[0003] Currently, in order to improve the estimation accuracy of 3D human pose, research on human pose estimation mainly focuses on the design of deep convolutional neural networks. However, these methods have some limitations, especially in terms of global information acquisition. Considering that human pose can be regarded as a graph structure, where joints are nodes and bones are edges, graph neural networks (GNN) have emerged. Some studies use fully connected networks and GNNs to encode the relationship between local joints. In recent years, the Transformer model with strong global self-attention capabilities has also gradually attracted attention. However, regardless of which algorithm is used for prediction, there is a problem of inaccurate recognition of the lower body in the estimation results. This is because under the condition of using only a monocular camera, it cannot directly obtain depth information, which leads to problems such as floating or shifting of the feet in the estimation results. This problem is particularly prominent in dynamic scenes, complex poses, and lower body poses, which limits the accuracy and applicability of the method.

[0004] In the field of sensing, some work has explored human activities through pressure information between humans and their surroundings, which further demonstrates the potential and importance of pressure sensing technology in the field of human pose estimation. However, these methods are mainly suitable for poses that require extensive contact with the sensing surface, such as lying down. For most daily life poses that mainly involve contact between the feet and the sensing surface, this approach is not applicable. This limitation often leads to inaccurate estimation of upper body poses, as upper body parts such as arms and heads usually do not significantly affect the tactile signals from the feet.

[0005] Therefore, relying solely on visual or pressure methods is not reliable enough. SUMMARY

[0006] To solve the above problems, the present application proposes a human pose estimation method and device fusing vision and pressure, and a medium.

[0007] The specific scheme is as follows:

[0008] A human posture estimation method fusing vision and pressure, comprising the following steps:

[0009] S1: acquiring video frame images of a to-be-measured region by a monocular camera, acquiring pressure distribution data of the to-be-measured region by a pressure sensor, and grouping video frame image sequences and pressure distribution data sequences in the same time period into a training set;

[0010] S2: constructing a human posture recognition model and training the model by the training set;

[0011] The model comprises an image feature extraction module, a pressure feature extraction module and a multi-modal feature fusion module;

[0012] The input of the image feature extraction module is the video frame image sequence, and the network structure thereof comprises, in sequence, a first convolutional layer, a maximum pooling layer, five groups of residual modules and a second convolutional layer, and the first convolutional layer and the second convolutional layer are both connected with an activation function batch normalization and a leaky linear rectifier function;

[0013] The input of the pressure feature extraction module is the pressure distribution data sequence, and the network structure thereof comprises seven convolutional layers, wherein the second, fourth and seventh convolutional layers are connected with a maximum pooling layer after the same;

[0014] The input of the multi-modal feature fusion module is the fused features of the output of the image feature extraction module and the output of the pressure feature extraction module, the network structure thereof comprises five convolutional layers, and a conversion layer is inserted in the middle of the five convolutional layers to increase the resolution of the features; the first four convolutional layers of the five convolutional layers and the conversion layer in the middle are both connected with an activation function batch normalization and a leaky linear rectifier function, and an S-shaped activation function is connected after the last convolutional layer;

[0015] S3: estimating the human posture by the trained model.

[0016] Further, the number of residual blocks contained in each of the five groups of residual modules in the image feature extraction module is three, four, six, three and one respectively, and each residual block contains two 1x1 convolutional kernels and one convolutional kernel with a size of 3x3 and a padding of 1.

[0017] Further, the kernel size of the convolutional kernel in the first convolutional layer in the image feature extraction module is 7x7, the step is set to 2, the size of the maximum pooling layer is 3x3, and the step is set to 2, and the kernel size of the convolutional kernel in the second convolutional layer is 3x4.

[0018] Further, in the seven convolutional layers of the pressure feature extraction module, the first five layers and the seventh layer use a kernel with a size of 3x3 and a padding with a size of 1x1, the sixth layer uses a kernel with a size of 7x7 without padding; and the size of the maximum pooling layer applied is 2x2.

[0019] Further, the kernel size of each layer of the 5-layer convolutional layer included in the multi-modal feature fusion module is 3*3*3, and the padding is 1*1*1.

[0020] Further, the loss function of the model is :

[0021]

[0022]

[0023] wherein N represents the number of human key points, i represents the serial number of the human key points, N-1 is the number of bones in the human skeleton, H i represents the true value of the 3D key point confidence map, represents the predicted value of the 3D key point confidence map, represents the deviation value, K i is the bone length calculated according to the prediction result of the model, and respectively represent the minimum value and the maximum value of the preset bone length, and ||.|| represents the L2 norm.

[0024] Further, the manner of collecting the pressure distribution data of the to-be-measured region through the pressure sensor is that a plurality of pressure sensors are arranged in the to-be-measured region, and the data collected by all the pressure sensors is spliced to form the pressure distribution data.

[0025] Further, the method further comprises calibrating the data for splicing through a linear regression method and performing noise reduction processing on the spliced pressure distribution data.

[0026] A human posture estimation terminal device fusing vision and pressure, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the above-mentioned embodiment of the application when executing the computer program.

[0027] A computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by a processor to implement the steps of the method of the above-mentioned embodiment of the application.

[0028] The present application adopts the above technical solution, compared with the existing method, the present application comprehensively utilizes visual and pressure sensor data, and advanced technologies such as deep learning, and realizes more accurate 3D human posture estimation. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1Fig. 1 shows a flow chart of an embodiment of the present application.

[0030] Figure 2 Fig. 2 shows a schematic diagram of the arrangement of pressure sensors in this embodiment.

[0031] Figure 3 Fig. 3 shows a network overview diagram of the model in this embodiment.

[0032] Figure 4 Fig. 4 shows a schematic diagram of the network structure of the model in this embodiment.

[0033] Figure 5 Fig. 5 shows a quantitative effect diagram of the experiment in this embodiment.

[0034] Figure 6 Fig. 6 shows a qualitative effect diagram of the experiment in this embodiment. DETAILED DESCRIPTION

[0035] To further illustrate the embodiments, the present application provides accompanying drawings. These drawings are part of the disclosure of the present application, which mainly serve to illustrate the embodiments, and can be used to explain the operating principles of the embodiments in conjunction with the relevant description of the specification. With reference to these contents, those of ordinary skill in the art should be able to understand other possible implementations and advantages of the present application.

[0036] The present application will be further described in conjunction with the accompanying drawings and specific embodiments.

[0037] Embodiment 1

[0038] The embodiment of the present application provides a human posture estimation method fusing vision and pressure, as shown in Figure 1 The method comprises the following steps:

[0039] S1: Acquire video frame images of the to-be-measured region through a monocular camera, and acquire pressure distribution data of the to-be-measured region through a pressure sensor; and form a training set by combining a video frame image sequence and a pressure distribution data sequence in the same time period.

[0040] The monocular camera and the pressure sensor can be connected to the terminal device through a USB interface. In this embodiment, the monocular camera captures video data at a frame rate of 30 fps and a resolution of 720p. In the arrangement of the pressure sensor, a plurality of pressure sensors can be uniformly distributed in the to-be-measured region according to the size of the to-be-measured region, for example, four pressure sensors are arranged in the to-be-measured region in this embodiment, as shown in Figure 2As shown, the collected pressure distribution data is composed of the data of the 4 pressure sensors. In the data splicing of the pressure sensors, the data can be spliced according to the direction of the pressure data record and the physical position of the pressure sensor. Further, in order to improve the accuracy of subsequent attitude estimation, the data before splicing is converted by a linear regression method in this embodiment, and the noise data in the pressure distribution data after splicing is removed, and the specific method can adopt the existing bin method, and the pressure data is smoothed by bin mean, and each value in a bin is replaced by the average value of all values belonging to the same bin.

[0041] Each training data in the training set includes a video frame image sequence and a pressure distribution data sequence, and the label of the training data is a key point image corresponding to the human body attitude in the time period.

[0042] S2: Construct a human body attitude recognition model, and train the model through the training set.

[0043] As shown in Figure 3 and 4 , the human body attitude recognition model in this embodiment includes an image feature extraction module, a pressure feature extraction module and a multi-modal feature fusion module.

[0044] (1) The input of the image feature extraction module is a video frame image sequence, and its network structure includes a first convolutional layer, a maximum pooling layer, five groups of residual modules and a second convolutional layer in turn, and the first convolutional layer and the second convolutional layer are connected with an activation function batch normalization and a leaky linear rectifier function. By placing the activation function batch normalization and the leaky linear rectifier function before the weight layer, the training process of the model can be simplified, and the generalization ability of the model can be improved.

[0045] In this embodiment, the kernel size of the convolution kernel in the first convolutional layer is 7x7, the step is set to 2, the size of the maximum pooling layer is 3x3, and the step is set to 2, and the kernel size of the convolution kernel in the second convolutional layer is 3x4. The second convolutional layer is used to map the image features to a size suitable for subsequent feature fusion. Each group of the five groups of residual modules contains a different number of residual blocks, which are three, four, six, three and one respectively. Each residual block contains two 1x1 convolution kernels and one convolution kernel with a kernel size of 3x3 and a padding of 1. The size of the maximum pooling layer is 3x3 and the step is 2, which is used to reduce the input image data to a feature map of 160x180.

[0046] (2) The input of the pressure feature extraction module is a pressure distribution data sequence, and its network structure includes 7 convolutional layers, among which the 2nd, 4th and 7th convolutional layers are connected with a maximum pooling layer.

[0047] The first 5 layers and the 7th layer of convolutional layer in the embodiment use a kernel with a size of 3x3 and a padding with a size of 1x1, the 6th layer of convolutional layer uses a larger kernel (i.e. a kernel with a size of 7x7), without padding, and the maximum pooling layer applied has a size of 2x2.

[0048] (3) The input of the multi-modal feature fusion module is the fused feature of the output of the image feature extraction module and the output of the pressure feature extraction module, the network structure of which includes 5 layers of convolutional layers, and a conversion layer is inserted in the middle of the 5 layers of convolutional layers (e.g. Figure 4 inserted between the 2nd layer and the 3rd layer) to increase the resolution of the feature; the first 4 layers of convolutional layers of the 5 layers of convolutional layers and the conversion layer in the middle are all connected with an activation function batch normalization and a leaky linear rectifier function, and the last layer of convolutional layer is connected with a sigmoid activation function (Sigmoid) to finally regress the confidence of the 3D key point.

[0049] In the fusion of the two kinds of features, reconstruction can be performed in the order of x-y-z voxel space. The feature size of the pressure data becomes 1024x8x8x1, and the feature size of the image becomes 1024x8x1x10. Then, the two kinds of features are multiplied to obtain a new 3D fused feature with a size of 1024x8x8x10.

[0050] In the 5 layers of convolutional layers of the embodiment, the kernel size of each layer is 3x3x3, and the padding is 1x1x1; in the middle conversion layer, a kernel size of 2x2x2 is used, and the step size is set to 2 to increase the feature resolution from 8x8x10 to 16x16x20.

[0051] In order to better optimize the model, the model in the embodiment uses an Adam optimizer to optimize the model by minimizing the mean square error between the predicted value and the true value of the key point confidence map.

[0052] The specific loss function is:

[0053]

[0054]

[0055] wherein N represents the number of human key points, i represents the serial number of the human key point, N-1 is the number of bones in the human skeleton, H i represents the true value of the 3D key point confidence map, represents the predicted value of the 3D key point confidence map, represents a deviation value, K i is the bone length calculated according to the model prediction result, and respectively represent the minimum and maximum values of the preset bone length, and ||.|| represents the L2 norm.

[0056] S3: estimating the human pose through the trained model.

[0057] When the human pose needs to be estimated, the video frame image sequence and the pressure distribution data sequence of the to-be-measured region in a period of time are input into the trained model through the manner of step S1, and the result output by the model is taken as the estimation result of the human pose.

[0058] In order to evaluate the effect achieved by the method of the embodiment, the quantitative and qualitative methods are used for evaluation in the embodiment, wherein the quantitative method uses a data index most commonly used in the field of 3D human pose estimation to evaluate the prediction ability of the model, that is, the mean per joint position error (MPJPE).

[0059] Figure 5 The achievements achieved by the application in the quantitative index are shown. Compared with the Intelligent Carpet method, the application is obviously better in the MPJPE data distribution, and the prediction effect is improved by one order of magnitude at each key point. The prediction effect of the upper body and the lower body is more balanced, the accuracy of the overall pose is not much different, and the demand of the actual application is met. In addition, compared with other pure visual methods, the evaluation index of the application is better than those of the methods, and the average MPJPE is 56.0 mm, which is reduced by 23.6 mm than the best result (79.6 mm) of the pure visual method. For complex foot changes, such as "tap dance", "ballet" and "jump rope", the application achieves significant improvement compared with other methods.

[0060] Figure 6 The achievements achieved by the application in the qualitative index are shown, and the ability of the application to provide more accurate 3D human pose estimation can be intuitively embodied visually.

[0061] The method of the embodiment of the application plays an important role in the continuously developing human-computer interface, and has wide application prospects in the fields of immersive interaction in virtual reality games, posture monitoring in medical rehabilitation, motion evaluation in sports analysis and the like.

[0062] Embodiment two:

[0063] The application further provides a human pose estimation terminal device fusing vision and pressure, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps in the method embodiments of the above-mentioned embodiment one of the application when executing the computer program.

[0064] Further, as an executable solution, the human pose estimation terminal device fusing vision and pressure can be a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The human pose estimation terminal device fusing vision and pressure can include, but is not limited to, a processor, a memory. Those skilled in the art can understand that the above-mentioned composition of the human pose estimation terminal device fusing vision and pressure is only an example of the human pose estimation terminal device fusing vision and pressure, and does not constitute a limitation on the human pose estimation terminal device fusing vision and pressure, and can include more or fewer components than the above, or combine certain components, or different components, for example, the human pose estimation terminal device fusing vision and pressure can also include an input / output device, a network access device, a bus and the like, and the embodiments of the present application do not limit this.

[0065] Further, as an executable solution, the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor and the like, and the processor is a control center of the human pose estimation terminal device fusing vision and pressure, and connects various parts of the human pose estimation terminal device fusing vision and pressure through various interfaces and lines.

[0066] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the human pose estimation terminal device fusing vision and pressure by running or executing the computer program and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function; the data storage area can store data created according to the use of the mobile phone and the like. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, for example, a hard disk, a memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory devices.

[0067] The application further provides a computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, implements the steps of the method of the above-mentioned embodiment of the application.

[0068] The module / unit integrated with the fusion visual and pressure human body posture estimation terminal device, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the application can also complete all or part of the processes of the above-mentioned embodiment methods by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. When the computer program is executed by a processor, the steps of each method embodiment can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, an executable file or some intermediate form, etc. The computer readable medium can include any entity or device capable of carrying the computer program code, a recording medium, a U disk, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM) and a software distribution medium, etc.

[0069] Although the application is specifically shown and described in connection with the preferred embodiments, those skilled in the art should understand that various changes can be made in form and details without departing from the spirit and scope of the application as defined by the appended claims.

Claims

1. A method of human pose estimation fusing vision and pressure, characterized in that, The method comprises the following steps: S1: acquiring video frame images of a to-be-tested region by a monocular camera, acquiring pressure distribution data of the to-be-tested region by a pressure sensor, and forming a training set by a video frame image sequence and a pressure distribution data sequence in the same time period; S2: constructing a human posture recognition model and training the model by the training set; The model comprises an image feature extraction module, a pressure feature extraction module, and a multi-modal feature fusion module; The input of the image feature extraction module is the video frame image sequence, the network structure of the image feature extraction module comprises a first convolutional layer, a maximum pooling layer, five groups of residual modules, and a second convolutional layer in sequence, and the first convolutional layer and the second convolutional layer are both connected with an activation function batch normalization and a leaky linear rectifier function; The input of the pressure feature extraction module is the pressure distribution data sequence, and the network structure of the pressure feature extraction module comprises seven convolutional layers, wherein the second, fourth, and seventh convolutional layers are connected with a maximum pooling layer; The input of the multi-modal feature fusion module is the fused features of the output of the image feature extraction module and the output of the pressure feature extraction module, the network structure of the multi-modal feature fusion module comprises five convolutional layers, and a conversion layer is inserted in the middle of the five convolutional layers to increase the resolution of the features; the first four convolutional layers of the five convolutional layers and the conversion layer in the middle are both connected with an activation function batch normalization and a leaky linear rectifier function, and the last convolutional layer is connected with an S-shaped activation function; S3: estimating the human posture by the trained model.

2. The method of human pose estimation fusing vision and pressure as claimed in claim 1, wherein: The number of residual blocks contained in each of the five groups of residual modules in the image feature extraction module is three, four, six, three, and one respectively, and each residual block contains two 1×1 convolutional kernels and one convolutional kernel with a size of 3×3 and a padding of 1.

3. The method of fusing vision and pressure for human pose estimation according to claim 1, wherein: In the first convolutional layer of the image feature extraction module, the kernel size of the convolutional kernel is 7×7, the step is set to 2, the maximum pooling layer size is 3×3, and the step is set to 2, and the kernel size of the convolutional kernel in the second convolutional layer is 3×4.

4. The method of fusing vision and pressure for human pose estimation according to claim 1, wherein: In the seven convolutional layers of the pressure feature extraction module, the first five layers and the seventh layer use a kernel with a size of 3×3 and a padding with a size of 1×1, the sixth layer uses a kernel with a size of 7×7 without padding; the maximum pooling layer size applied is 2×2.

5. The method of fusing vision and pressure for human pose estimation according to claim 1, wherein: In the five convolutional layers of the multi-modal feature fusion module, the kernel size of each layer is 3×3×3, and the padding is 1×1×1; the kernel size of the middle layer is 2×2×2, and the step is set to 2.

6. The method of fusing vision and pressure for human pose estimation according to claim 1, wherein: Loss function of the model is: wherein N represents the number of human key points, i represents the serial number of the human key points, N-1 is the number of bones in the human skeleton, H i represents a true value of a 3D key point confidence map, represents a predicted value of a 3D key point confidence map, represents a deviation value, K i is a bone length calculated according to a model prediction result, and respectively represent a minimum value and a maximum value of a preset bone length, ||.|| represents an L2 norm.

7. The method of fusing vision and pressure for human pose estimation according to claim 1, wherein: The way of acquiring the pressure distribution data of the to-be-tested region by the pressure sensor is to arrange a plurality of pressure sensors in the to-be-tested region, and to splice the data collected by all the pressure sensors to form the pressure distribution data.

8. The method of fusing vision and pressure human pose estimation according to claim 7, characterized in that: It also includes aligning the data for splicing by a linear regression method and performing noise reduction processing on the spliced pressure distribution data.

9. A human pose estimation terminal device fusing vision and pressure, characterized by: The computer program is executed by the processor to implement the steps of the method of any one of claims 1-8.

10. A computer-readable storage medium, the computer-readable storage medium storing a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the method of any one of claims 1-8.

Images