Human body posture recognition method and device

Abstract

The invention relates to the technical field of image recognition, in particular to a human body posture recognition method and device. According to the method, an OpenPose skeleton point algorithm model is constructed, and the OpenPose skeleton point algorithm model comprises a MobileNet V3 network and two parallel multi-stage cyclic branches; wherein each stage of each branch comprises a plurality of distributed shift convolutions, a transition layer and a CA attention mechanism module which are sequentially connected along a forward propagation direction; the transition layer comprises a normalization layer, an activation function layer and a 1 * 1 convolution layer, inputting a to-be-detected image into the MobileNet V3 network, and extracting image features; inputting image features into the two parallel multi-stage cyclic branches, and respectively extracting a key point confidence map and a component affinity field; obtaining coordinate data of the human body key points based on the key point confidence map and the component affinity field; and inputting the coordinate data of the key points of the human body into a multi-layer perceptron, and outputting a posture prediction result. According to the method, an OpenPose skeleton point algorithm model is improved, the model is lightened, and the accuracy of posture recognition is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and in particular to a method and device for human body posture recognition. Background Art

[0002] Pose Estimation is an important research direction in the field of computer vision, which aims to detect and identify human postures in images or videos. In practical applications, posture recognition technology is widely used in intelligent monitoring, human-computer interaction, virtual reality, motion analysis and other fields. Today, posture recognition technology has evolved from early manual feature extraction and support vector machines to deep learning-driven convolutional neural networks. With the development of technology, human posture recognition methods have shifted from complex two-stage models to more efficient single-stage models, which significantly improve the real-time performance of human posture recognition. At present, the research focus is on balancing real-time performance and accuracy, exploring multimodal fusion, self-supervised learning and privacy protection to meet the growing needs of practical applications.

[0003] At present, the posture recognition method uses the OpenPose skeleton point algorithm to detect the key points of the human skeleton. The application of OpenPose's skeleton point algorithm in posture recognition can provide high precision, multi-person posture recognition detection, real-time performance and rich information, which makes it an ideal choice in many fields that require accurate posture recognition.

[0004] However, in complex backgrounds or occlusions, the OpenPose skeleton point algorithm often has difficulty accurately locating the key points of the human body, which may lead to unstable or erroneous detection of the key points of the human body, thereby reducing the accuracy of the posture recognition algorithm. In addition, the OpenPose skeleton point algorithm runs inefficiently on edge devices with limited computing resources, making it difficult to achieve real-time posture recognition. On the other hand, the OpenPose skeleton point algorithm exhibits poor robustness and generalization capabilities when affected by changes in lighting, posture, or different shooting devices. These factors may cause the performance of the model to fluctuate in different scenarios or conditions, reducing the reliability of the OpenPose skeleton point algorithm in practical applications. In addition, the edge system has limited computing resources, requiring the posture recognition algorithm to run and process data efficiently in a resource-constrained environment, while existing posture recognition algorithms are usually designed to run on high-performance central processing units and are difficult to deploy directly to edge systems. Summary of the invention

[0005] To this end, the technical problem to be solved by the present invention is to overcome the technical problems that the OpenPose skeleton point algorithm in the prior art has poor accuracy in detecting key points of the human body in complex scenes, has low operating efficiency on edge devices, and exhibits poor robustness and generalization ability when affected by changes in lighting, posture or different shooting devices. In addition, the problem that the existing posture recognition algorithm is difficult to deploy directly to the edge system is also solved.

[0006] To solve the above technical problems, the present invention provides a method for human body posture recognition, comprising the following steps:

[0007] Construct an OpenPose skeleton point algorithm model, the OpenPose skeleton point algorithm model includes: a MobileNetV3 network, two parallel multi-stage loop branches; wherein each stage of each branch of the two parallel multi-stage loop branches includes: a plurality of distributed shift convolutions, transition layers, and CA attention mechanism modules sequentially connected along the forward propagation direction; the transition layer includes: a normalization layer, an activation function layer, and a 1×1 convolution layer;

[0008] Inputting the image to be tested into the MobileNet V3 network, extracting features from the image to be tested, and obtaining image features;

[0009] Inputting the image features into the two parallel multi-stage loop branches to extract the key point confidence map and the component affinity field respectively;

[0010] Based on the key point confidence map and the component affinity field, the coordinate data of the key points of the human body are obtained;

[0011] The coordinate data of the key points of the human body are input into the multi-layer perceptron and the posture prediction results are output.

[0012] Preferably, the image features are input into two parallel multi-stage loop branches, and each stage extracts the key point confidence map and component affinity field of the input features of the current stage, including the following steps:

[0013] Input the input features into the two branches of the current stage respectively, and extract the initial key point confidence map and component affinity field of the current stage respectively through multiple distribution shift convolutions of the two branches;

[0014] The initial key point confidence map and component affinity field of the current stage are respectively reduced in dimension through the transition layer of the corresponding branch to obtain the initial key point confidence map and component affinity field of the current stage after dimension reduction;

[0015] The initial key point confidence map and component affinity field of the current stage after dimensionality reduction are respectively adjusted through the CA attention mechanism module of the corresponding branch to adjust the weight of each channel to obtain the key point confidence map and component affinity field of the input features of the current stage.

[0016] Preferably, the CA attention mechanism module includes: a coordinate information embedding module and a CA generation module.

[0017] Preferably, the activation function of the depthwise separable convolution in the MobileNet V3 network is a PReLU activation function.

[0018] Preferably, the training process of the OpenPose skeleton point algorithm model is: obtaining a posture recognition data set, preprocessing the posture recognition data set, and dividing the preprocessed posture recognition data set into a training set, a validation set, and a test set;

[0019] The OpenPose skeleton point algorithm model is trained using the training set, and the transfer learning method is used to prioritize the training of the parameters of the PReLU activation function of the depthwise separable convolution in the MobileNet V3 network. After the parameters of the PReLU activation function are determined, the overall training of the OpenPose skeleton point algorithm model is performed. The trained OpenPose skeleton point algorithm model is verified using the validation set to obtain the target OpenPose skeleton point algorithm model.

[0020] Preferably, the posture recognition dataset is a PoseTrack dataset, and diverse samples and corresponding time series data are added for preset application scenarios.

[0021] Preferably, the loss function of each stage of the two parallel multi-stage loop branches in the OpenPose skeleton point algorithm model is a cross entropy loss function.

[0022] Preferably, the acquiring the coordinate data of the key points of the human body based on the key point confidence map and the component affinity field comprises: calculating the coordinate data of the key points of the human body by a pair matching method in graph theory based on the key point confidence map and the component affinity field.

[0023] Preferably, the activation function of the multi-layer perceptron output layer is a tanh function.

[0024] The present invention also provides a human body posture recognition device, comprising:

[0025] The image acquisition device is used to acquire the image to be tested in real time and transmit the image to be tested to the edge computing device; the edge computing device is used to obtain the posture prediction result based on the image to be tested, connect to the cloud platform through the 5G communication module, and upload the posture prediction result to the user terminal, including:

[0026] A lightweight model building module, used to build an OpenPose skeleton point algorithm model, wherein the OpenPose skeleton point algorithm model includes: a MobileNet V3 network and two parallel multi-stage loop branches;

[0027] Each stage of each branch of the two parallel multi-stage loop branches includes: multiple distribution shift convolutions, transition layers, and CA attention mechanism modules connected in sequence along the forward propagation direction;

[0028] The transition layer includes: normalization layer, activation function layer and 1×1 convolution layer;

[0029] An image extraction module is used to input the image to be tested into the MobileNet V3 network, extract features from the image to be tested, and obtain image features;

[0030] A key point confidence map and component affinity field acquisition module, used for inputting image features into the two parallel multi-stage loop branches to extract key point confidence maps and component affinity fields respectively;

[0031] A coordinate data acquisition module is used to acquire coordinate data of key points of the human body based on the key point confidence map and the component affinity field;

[0032] The recognition module is used to input the coordinate data of the key points of the human body into the multi-layer perceptron and output the posture prediction results;

[0033] Cloud platform: used to store human body posture prediction results and support uploading and downloading of the human body posture prediction results;

[0034] User terminal: used for users to receive and view posture prediction results in real time and provide human-computer interaction.

[0035] The above technical solution of the present invention has the following beneficial effects compared with the prior art:

[0036] A human posture recognition method and device according to the present invention constructs an OpenPose skeleton point algorithm model, wherein the OpenPose skeleton point algorithm model comprises: a MobileNet V3 network, two parallel multi-stage loop branches, wherein each stage of each branch of the two parallel multi-stage loop branches comprises: a plurality of distribution shift convolutions, transition layers, and CA attention mechanism modules sequentially connected along the forward propagation direction; the transition layer comprises: a normalization layer, an activation function layer, and a 1×1 convolution layer; by using a lightweight MobileNet V3 network to extract image features, the computational efficiency and real-time performance of the OpenPose skeleton point algorithm model are improved, the image features are input into two parallel multi-stage loop branches, and key point confidence maps and component affinity fields are extracted respectively, and by using distribution shift convolution, not only the output accuracy can be maintained the same as the original convolution, but also the memory of the OpenPose skeleton point algorithm model is reduced, the calculation speed of the OpenPose skeleton point algorithm model is improved, and the operation is more efficient in scenarios where embedded system resources are limited. The introduction of the transition layer and the CA attention mechanism module enhances the model's ability to focus on spatial and channel features, thereby improving the accuracy of posture recognition, especially in complex posture and occlusion scenarios. By inputting the coordinate data of the key points of the human body into the multi-layer perceptron and outputting the posture prediction results, the multi-layer perceptron has strong nonlinear modeling capabilities, which enables it to perform well in processing complex data patterns and features, and can also complete accurate recognition in multi-person scenarios.

[0037] In addition, the cross entropy loss function is used as the loss function of each stage of the two branches of the OpenPose skeleton point algorithm model. By optimizing the loss function during training, the accuracy and robustness of the key point positioning of the OpenPose skeleton point algorithm model are significantly improved, and the accuracy of human posture recognition is improved, especially in application scenarios with changing lighting, partial occlusion and complex postures. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below according to specific embodiments of the present invention in conjunction with the accompanying drawings, wherein

[0039] Figure 1 The present invention is a flowchart of the steps of a method for recognizing human posture.

[0040] Figure 2 It is a flow chart of the MobileNet V3 network in a method and device for human posture recognition of the present invention.

[0041] Figure 3 This is a convolutional layer structure diagram of the MobileNet V3 deep separable convolution in a human posture recognition method and device of the present invention.

[0042] Figure 4 This is a structural diagram of the first stage of two branches in the OpenPose skeleton point algorithm model in a human posture recognition method and device of the present invention.

[0043] Figure 5 This is a structural diagram of the second stage of two branches in the OpenPose skeleton point algorithm model in a human posture recognition method and device of the present invention.

[0044] Figure 6 This is a structural diagram of the CA attention mechanism module in a human posture recognition method and device of the present invention.

[0045] Figure 7 This is a structural diagram of a multi-layer perceptron in a human posture recognition method and device of the present invention.

[0046] Figure 8 This is a schematic diagram of data flow on the Alibaba Cloud platform in a human posture recognition method and device of the present invention. DETAILED DESCRIPTION

[0047] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it, but the embodiments are not intended to limit the present invention.

[0048] Reference Figure 1 As shown, an embodiment of the present invention provides a method for human body posture recognition, comprising the following steps:

[0049] Step S1: constructing an OpenPose skeleton point algorithm model, the OpenPose skeleton point algorithm model comprising: a MobileNet V3 network, two parallel multi-stage loop branches; wherein each stage of each branch of the two parallel multi-stage loop branches comprises: a plurality of distributed shift convolutions, transition layers, and CA attention mechanism modules sequentially connected along the forward propagation direction; the transition layer comprises: a normalization layer, an activation function layer, and a 1×1 convolution layer;

[0050] Except for the first stage, before each subsequent stage, the key point confidence map and component affinity field of the current stage output by the two branches of the previous stage are connected with the image feature F as the input of the next stage.

[0051] The first branch extracts the key point confidence map (Part Confidence Maps), which is used to represent the position of each part in the image, that is, the key point position information. The second branch extracts the part affinity field (Part Affinity Fields), which is used to represent the relationship between each part, that is, the key point connection information. Each branch is an iterative prediction architecture.

[0052] like Figure 4 As shown, Figure 4 The first stage structure diagram of the two branches of the OpenPose skeleton point algorithm model includes three distributed shift convolution, transition layer, and CA attention mechanism modules connected in sequence along the forward propagation direction. Figure 5 As shown, Figure 5 This is the second-stage structure diagram of the two branches of the OpenPose skeleton point algorithm model. The first stage of both branches includes five distributed shift convolutions, transition layers, and CA attention mechanism modules connected in sequence along the forward propagation direction.

[0053] Distribution Shift Convolution (DSConv), DSConv is an innovative convolution operation designed to improve the memory efficiency and computational speed of convolutional neural networks (CNNs) while maintaining or improving the accuracy of the model. Distribution Shift Convolution includes three key technologies: first, weight decomposition, which divides the weight tensor into blocks along the depth dimension, each block contains a low-order integer component (VQK) and a high-precision floating-point scaling factor (kernel distribution offset, KDS) to significantly reduce memory usage and computational complexity; second, the block floating-point method, which helps maintain model performance by dividing the activation tensor into blocks and letting each block share an exponent to reduce accuracy loss and maintain the distribution of activation values; finally, probability distribution maintenance, through precise weight decomposition and distribution offset strategies, maintains the probability distribution of weights and activations between the quantized model and the original model, thereby maintaining the accuracy of the model. It can be seen that DSConv runs more efficiently in resource-constrained scenarios of embedded systems by reducing memory usage and improving computational speed. It is able to maintain the same output accuracy as the original convolution, which is just in line with the use of edge systems.

[0054] MobileNet V3 is a lightweight convolutional neural network pre-trained on a large-scale dataset. By loading these pre-trained weights, you can take advantage of the features learned on general image data to get a good starting point on specific tasks.

[0055] The OpenPose skeleton point algorithm model of the present invention adds a CA attention mechanism on the original basis. Since the distribution shift convolution will cause the number of channels of the final output feature map to be large, in order to reduce the number of channel layers, a transition layer is used for dimensionality reduction, and the input and output sizes of the CA attention mechanism are adjusted to match the structure of the MobileNet V3 network. The CA attention mechanism pays attention to both spatial and channel features to better capture important information in the feature map. This dual focus can enhance the model's ability to understand complex scenes. By introducing coordinate information, the model can be richer in feature expression and enhance the ability to distinguish targets, thereby improving the accuracy of posture recognition. In addition, the CA attention mechanism is lighter and suitable for edge deployment needs.

[0056] In this embodiment, preferably, the training process of the OpenPose skeleton point algorithm model is: obtaining a posture recognition data set, preprocessing the posture recognition data set, and dividing the preprocessed posture recognition data set into a training set, a validation set, and a test set;

[0057] The OpenPose skeleton point algorithm model is trained using the training set, and the transfer learning method is used to prioritize the training of the parameters of the PReLU activation function of the deep separable convolution in the MobileNet V3 network. After the parameters of the PReLU activation function are determined, the overall training of the OpenPose skeleton point algorithm model is performed. When setting the training parameters, the initial learning rate is selected as 0.001 as the starting point. As the training progresses, the learning rate decay strategy (exponential decay) is used. The performance of the OpenPose skeleton point algorithm model is evaluated using the validation set, and indicators such as accuracy and recall are calculated to monitor whether the model is overfitting. According to the validation results, hyperparameters such as learning rate and batch size are adjusted in time. The early stopping strategy is used to monitor the validation loss. When the loss no longer decreases, training is stopped to prevent overfitting. Fine-tune specific layers as needed to further improve the performance of the OpenPose skeleton point algorithm model and obtain the target OpenPose skeleton point algorithm model.

[0058] The posture recognition dataset is the PoseTrack dataset, and a variety of samples and corresponding time series data are added for preset application scenarios. PoseTrack provides rich dynamic scene data, which is suitable for detecting and tracking human postures. The dataset contains annotations of various postures and key points, which is helpful for model training and evaluation. At the same time, this dataset is particularly suitable for training using OpenPose to improve the performance of the model in different scenarios. This training selects the skeletal point position information in PoseTrack, that is, the key point annotation information. By expanding the dataset for specific application scenarios and adding time series data, the adaptability and generalization ability of the model in specific tasks are enhanced, and it can be recognized more accurately.

[0059] For example, in a smart home scenario, we can supplement indoor posture samples and corresponding time series data, as well as posture samples and corresponding time series data of dangerous actions such as falling. Human posture recognition can improve user experience, increase safety and enhance home automation level in applications.

[0060] The posture recognition dataset is preprocessed and divided into training set, validation set and test set in a ratio of 8:1:1. The preprocessing includes:

[0061] Data enhancement is achieved by randomly rotating images, adjusting the size of images, flipping images horizontally and vertically, and performing operations on image brightness and contrast.

[0062] The posture recognition dataset images are normalized and the pixel values ​​are scaled to the following range: [0, 1], and the pixel values ​​are scaled by dividing them by 255. The formula is as follows.

[0063]

[0064] Among them, X is the original pixel value, X norm is the normalized value.

[0065] Standardize the key point position coordinate information in the posture recognition data set: make the average value of all key point position coordinate information 0 and the variance 1, so as to eliminate the dimension difference and value range difference between different key point position coordinate information. The specific calculation formula is as follows:

[0066]

[0067] Among them, X is the original key point position coordinate information, μ is the mean of the key point position coordinate information, σ is the standard deviation of the key point position coordinate information, X std is the normalized key point position coordinate information. Normalize the key point position coordinate information in the gesture recognition dataset: scale the key point position coordinate information to a small specific interval so that all key point position coordinate information are in the same numerical range. The specific calculation formula is as follows:

[0068]

[0069] Among them, X is the original key point position coordinate information, X max and X min They are the maximum and minimum values ​​of the key point position coordinate information, X ndh It is the normalized key point position coordinate information.

[0070] Data standardization and normalization are part of data preprocessing, which are used to ensure that the data has a consistent numerical range during the processing process to avoid excessive influence of certain key point position coordinate information on the model. Step S2: Input the image to be tested into the MobileNet V3 network, extract features from the image to be tested, and obtain image features;

[0071] like Figure 2 , 3 As shown, Figure 2 This is the MobileNet V3 network flow chart. Figure 3 This is the convolutional layer structure diagram of the MobileNet V3 depthwise separable convolution. The MobileNet V3 network is divided into two processes: first, depthwise convolution, and then pointwise convolution. In the pointwise convolution part, 1x1 convolution is used for channel compression, and the PReLU activation function is used when applying batch normalization and activation function.

[0072] In this embodiment, preferably, the activation function of the depthwise separable convolution in the MobileNet V3 network is a PReLU activation function, and a transfer learning method is adopted to preferentially train the parameters of the PReLU activation function. After the parameters of the PReLU activation function are determined, the overall training of the OpenPose skeleton point algorithm model is performed.

[0073] The PReLU activation function formula is:

[0074]

[0075] Among them, x i represents the value of the ith input to the PReLU activation function, a i The corresponding parameter representing the value of the i-th input to the PReLU activation function.

[0076] The PReLU activation function can better reflect the characteristics of negative samples than the ordinary ReLU activation function, so PReLU is used to enhance the learning of negative samples, especially in application scenarios with lighting changes, partial occlusion and complex postures. The use of PReLU activation function will enhance the robustness of the OpenPose skeleton point algorithm model.

[0077] The OpenPose skeleton point algorithm model uses the MobileNet V3 network as the feature extraction model, making the overall architecture lightweight and suitable for running on edge devices, significantly reducing the demand for computing resources and improving the real-time performance of the model.

[0078] Step S3: inputting the image features into the two parallel multi-stage loop branches to extract the key point confidence map and the component affinity field respectively;

[0079] In this embodiment, preferably, the loss function of each stage of the two parallel multi-stage loop branches in the OpenPose skeleton point algorithm model is a cross entropy loss function.

[0080] Specifically, the image features are input into two parallel multi-stage loop branches, and each stage extracts the key point confidence map and component affinity field of the input features of the current stage, including the following steps:

[0081] Input the input features into the two branches of the current stage respectively, and extract the initial key point confidence map and component affinity field of the current stage respectively through multiple distribution shift convolutions of the two branches;

[0082] The initial key point confidence map and component affinity field of the current stage are respectively reduced in dimension through the transition layer of the corresponding branch to obtain the initial key point confidence map and component affinity field of the current stage after dimension reduction;

[0083] The initial key point confidence map and component affinity field of the current stage after dimensionality reduction are respectively adjusted through the CA attention mechanism module of the corresponding branch to adjust the weight of each channel to obtain the key point confidence map and component affinity field of the input features of the current stage.

[0084] The key point confidence map and component affinity field in the first stage are: S 1 =ρ 1 (F) The key point confidence map and component affinity field at the tth stage are:

[0085] S t =ρ t (F,S t-1 ,L t-1 ),

[0086]

[0087] Among them, S 1 represents the confidence map of the key points in the first stage, L 1 represents the affinity field of the first stage components, F represents the image features of the image to be tested, S t represents the confidence map of the key points at stage t, L t represents the affinity field of the component at stage t, S t-1 represents the confidence map of the key points in stage t-1, L t-1 represents the affinity field of the component at stage t-1, ρ 1 (.)and They are the network structures of the key point confidence map and component affinity field predicted in the first stage, ρ t (.)and They are the network structures for predicting key point confidence maps and component affinity fields at the t-th stage, respectively, where t represents the stage.

[0088] In this embodiment, preferably, Figure 6 As shown, Figure 6 This is the structure diagram of the CA attention mechanism module. The CA attention mechanism module includes: a coordinate information embedding module and a CA generation module.

[0089] The input feature map of the CA attention mechanism module is reduced in dimension through the transition layer. The coordinate information embedding module uses global pooling to decompose the X and Y coordinates into two one-dimensional features. The X direction is encoded using the average pooling layer of (H*1) to generate z h , the Y direction is encoded using the average pooling layer of (1*W) to generate z w The CA generation module will transform it using 1*1 convolution to adjust the input and output size of the CA attention mechanism to match the structure of MobileNet V3V3. The formula is:

[0090] f=δ(F 1 ([z h ,z w ])),

[0091] Where f represents the intermediate feature map of spatial information in the horizontal and vertical directions, [.,.] represents the cascade operation along the spatial dimension, r represents the scaling factor, and z h is the result of encoding the X coordinate by the vertical pooling layer, z w is the result of encoding the Y coordinate by the horizontal pooling layer, F n Indicates z h and z w A 1*1 convolution operation is performed, δ represents nonlinear activation (sigmoid function), C represents the number of channels of the input feature map of the CA attention mechanism module, H represents the vertical dimension of the input feature map of the CA attention mechanism module, and W represents the horizontal dimension of the input feature map of the CA attention mechanism module.

[0092] The sigmoid activation function is used to generate the final attention weight, and the generated weight is multiplied by the CA input feature map to implement the attention weighted operation. The formula is:

[0093]

[0094] Then the input feature map of the CA attention mechanism module is updated, g h Output in X direction, g w is the output in the Y direction, f h represents the input feature map of the CA attention mechanism module in the X direction, fw Represents the input feature map of the CA attention mechanism module in the Y direction.

[0095] The loss function of each stage of the two parallel multi-stage loop branches in the OpenPose skeleton point algorithm model is a cross entropy loss function.

[0096] The cross entropy loss function formula is:

[0097]

[0098] in, is the loss function of the key point position information of the first branch at stage t, is the loss function of the key point connection information of the second branch t stage, W(P) = 0 is the unlabeled character joint point, W(P) = 1 is the labeled character joint point and non-joint point, Indicates the label map information of the jth key point position, Labeled graph information representing the connection information of the c-th keypoint.

[0099] By introducing the CA attention mechanism module and optimizing the loss function in the skeleton point algorithm, the model's ability to focus on spatial and channel features is enhanced, thereby improving the accuracy of posture recognition, especially in complex postures and occlusion scenes.

[0100] Step S4: based on the key point confidence map and the component affinity field, obtaining the coordinate data of the key points of the human body;

[0101] In this embodiment, preferably, the obtaining of coordinate data of key points of the human body based on the key point confidence map and the component affinity field includes: calculating the coordinate data of key points of the human body through the even matching method in graph theory based on the key point confidence map and the component affinity field. Even matching can ensure that each key point is uniquely and accurately paired to its corresponding component. In this way, the coordinate data of key points of the human body can be more accurately located, and errors caused by overlapping or interference of components can be reduced. The even matching method can handle complex postures and occlusion situations because it can consider the affinity and relative position relationship between components. Even in the case of partial occlusion or overlap of components, the graph theory method can still find a reasonable pairing through global optimization.

[0102] Step S5: Input the coordinate data of the key points of the human body into the multi-layer perceptron and output the posture prediction result.

[0103] like Figure 7 As shown, Figure 7The figure is a diagram of the structure of a multi-layer perceptron. The multi-layer perceptron is mainly composed of an input layer, a hidden layer, and an output layer. These layers are interconnected through neurons (or nodes) to form a deep feedforward neural network. The input layer receives the coordinate data of the key points of the human body and passes it to the hidden layer. The formula is as follows:

[0104] z (1) =w (1) ·x+b (1)

[0105] h (1) =σ(z (1) ),

[0106] Among them, z (1) is the output of the first hidden layer, w (1) is the weight of the first hidden layer, b (1) is the weight bias of the first hidden layer, h (1) Normalize the output of the first hidden layer.

[0107] Each neuron in the hidden layer is connected to all neurons in the previous layer, and each connection has a weight that determines how the signal propagates. The purpose of the hidden layer is to learn the feature representations in the input data. Finally, the hidden layer passes its output to the output layer, which produces the final prediction or classification result, as follows:

[0108] z (2) =w (2) ·h (1) +b (2) ,

[0109] Among them, z (2) is the output of the second output layer, w (2) is the weight of the second output layer, b (2) is the weight bias of the second output layer.

[0110] The present invention changes the second layer output layer function to a tanh function, and the formula is as follows:

[0111] h (2) =tanh(z (2) ),

[0112] h (2) Normalize the output of the second output layer.

[0113] In this embodiment, preferably, the activation function of the output layer of the multi-layer perceptron is a tanh function. Because the output range of the tanh function is between [-1, 1], it alleviates the problem of gradient vanishing and gradient exploding to a certain extent compared with the original Sigmod function. Because its positive and negative gradients can offset some of the effects during back propagation, it can reduce the oscillation in the gradient descent process, speed up its model training speed, and improve the accuracy of human posture recognition.

[0114] Among them, the training process of the multi-layer perceptron is: select the PoseTrack dataset as the basis, and this training selects the skeleton point position information and posture information in PoseTrack, that is, the key point annotation information and the corresponding posture information.

[0115] The data set is preprocessed and divided into training set, validation set and test set in a ratio of 8:1:1. The coordinate data is normalized using the following formula:

[0116]

[0117] Among them, x i ′ represents the position of the i-th coordinate after the coordinate data is normalized, x i represents the initial position of the i-th coordinate, μ 1 is the mean of the initial positions of the coordinates, σ 1 is the standard deviation of the initial position of each coordinate. Coordinate data normalization ensures that all coordinate data are mapped to the specified range, so that when training the multi-layer perceptron, different data points have similar dimensions and numerical ranges, which helps the training and convergence of the model.

[0118] The training methods of multi-layer perceptron include forward propagation and back propagation. Forward propagation is the process of calculating the output result from the input data through the neural network, and back propagation is to adjust the weights and biases between neurons according to the error between the output result and the true value. During the training process, the output result gradually approaches the true value by iteratively updating the weights and biases.

[0119] Use the validation set to evaluate model performance, calculate indicators such as accuracy and recall, monitor whether the model is overfitting, and adjust hyperparameters such as learning rate and batch size in a timely manner based on the validation results.

[0120] The embodiment of the present invention further provides a human body posture recognition device, comprising:

[0121] An image acquisition device is used to acquire the image to be tested in real time and transmit the image to be tested to the edge computing device through the data transmission module;

[0122] The edge computing device is used to obtain the posture prediction results based on the image to be tested, connect to the cloud platform through the 5G communication module, and upload the posture prediction results to the user terminal, including:

[0123] A lightweight model building module, used to build an OpenPose skeleton point algorithm model, wherein the OpenPose skeleton point algorithm model includes: a MobileNet V3 network and two parallel multi-stage loop branches;

[0124] Each stage of each branch of the two parallel multi-stage loop branches includes: multiple distribution shift convolutions, transition layers, and CA attention mechanism modules connected in sequence along the forward propagation direction;

[0125] The transition layer includes: normalization layer, activation function layer and 1×1 convolution layer;

[0126] An image extraction module is used to input the image to be tested into the MobileNet V3 network, extract features from the image to be tested, and obtain image features;

[0127] A key point confidence map and component affinity field acquisition module, used for inputting image features into the two parallel multi-stage loop branches to extract key point confidence maps and component affinity fields respectively;

[0128] A coordinate data acquisition module is used to acquire coordinate data of key points of the human body based on the key point confidence map and the component affinity field;

[0129] The recognition module is used to input the coordinate data of the key points of the human body into the multi-layer perceptron and output the posture prediction results;

[0130] Cloud platform: used to store human body posture prediction results and support uploading and downloading of the human body posture prediction results;

[0131] User terminal: used for users to receive and view posture prediction results in real time and provide human-computer interaction.

[0132] Traditional technical solutions often do not provide good data transmission and data management. The data transmission method may have high latency, and cannot realize instant warning and recording functions, or even data loss. Our technical solution adopts cutting-edge 5G communication modules, which greatly improves the efficiency of data transmission. The 5G communication module provides ultra-high-speed network connection, which can realize real-time data upload and remote access, improve the response speed of the system, and its high bandwidth capacity can support larger-scale data transmission, facilitate the upload and cloud processing of complex data. In addition, it has a wide coverage and stable connection, which is suitable for use in a variety of environments to ensure the reliability of the system.

[0133] In this embodiment, SolidWorks software is optionally used for three-dimensional modeling to ensure the accuracy and feasibility of the design. Through the powerful functions of SolidWorks, the modeling of each component of the device, including the shell, internal structure and key components, is completed to ensure that the model can reflect the actual engineering needs.

[0134] The trained OpenPose skeleton point algorithm model and multi-layer perceptron are converted into ONNX models, and then converted into a runnable RKNN model through rktoolkit2 of the RK3399Pro edge device. Finally, we load the above model on the RK3399Pro and implement real-time inference.

[0135] The main reasons why RK3399Pro is suitable for edge intelligence include its integrated neural network processor unit (NPU), which enables it to provide significant acceleration when processing AI and deep learning tasks. The following are the applicability and advantages of RK3399Pro in the field of edge intelligence:

[0136] Neural Network Processor Unit (NPU) acceleration: RK3399Pro integrates an NPU, which is specially designed to efficiently perform neural network inference tasks. This enables the device to quickly process complex deep learning models such as face detection, speech recognition, etc., which usually require real-time processing on edge devices. Multiple deep learning framework support: RK3399Pro supports multiple popular deep learning frameworks, including PyTorch and Caffe, which enables developers to choose the tools and libraries that best suit their needs for model development and optimization.

[0137] Complete model training and deployment tools: In addition to supporting model inference, RK3399Pro also provides complete model training tools and network training model instances. This means that the development team can perform end-to-end AI development on edge devices, from model training to deployment, all of which can be completed on RK3399Pro.

[0138] High-performance graphics processor and expansion interface: In addition to NPU, RK3399Pro also integrates a high-performance graphics processor (GPU) that can be used to process vision-related tasks. In addition, it also provides a rich expansion interface to support the connection of multiple sensors and external devices, providing greater flexibility and scalability for edge computing applications.

[0139] Applicable to a variety of edge intelligence applications: Due to its comprehensive functions and performance characteristics, RK3399Pro can be widely used in multiple fields such as mobile edge computing, smart home, industrial automation, and intelligent monitoring systems. Whether it is on mobile devices that require high-performance AI processing or in intelligent perception systems that need to deploy complex AI models, RK3399Pro can play an important role.

[0140] In summary, RK3399Pro is an ideal choice due to its powerful AI processing capabilities, diverse deep learning framework support, and rich expansion interfaces, especially suitable for scenarios that require complex AI and machine learning task processing on edge devices.

[0141] Specifically, a high-resolution camera is used to obtain the image to be tested in real time, and the image to be tested is transmitted to the edge computing device through the data transmission module to ensure the accuracy and integrity of the data. Real-time detection of human posture is achieved through the edge computing device. According to the recognized posture information, whether there is a dangerous posture, such as falling, imbalance, etc., is judged in real time to ensure timely response. The recognition results are uploaded to the Alibaba Cloud platform through the onboard Qualcomm RedCapX35 5G communication module through the RedCap protocol to ensure efficient data transmission and low latency. The cloud platform receives and stores the data, and supports the upload and download of the human posture prediction results. On the edge intelligent device RK3399Pro, the SYN6288 voice module is controlled through serial communication to play the prompt voice, which serves as an early warning and alarm for dangerous behaviors.

[0142] Through investigation and research, we found that many human posture recognition systems lack humanized user interface and mini-program design, which reduces the user experience. The technical solution we proposed designs a special WeChat mini-program for users, which makes it easier for users to view relevant information and easier to handle when abnormal behavior is detected.

[0143] The cloud product flow function of Alibaba Cloud platform message forwarding can forward the data in Topic 1 to Topic 2. The data flow diagram is as follows Figure 8 As shown. Includes:

[0144] Data source: Topic type messages supported by Topic1, see Data Format (Non-Cloud Gateway Products and Devices), Custom Topic (MQTT Cloud Gateway), Message Forwarding Topic (GB / T 32960 Cloud Gateway), Message Forwarding Topic (JT / T808 Cloud Gateway), Message Forwarding Topic (SL 651 Cloud Gateway).

[0145] Data destination: Topic2 that receives messages from Topic1.

[0146] Cloud product flow can forward the Topic1 message of device 1 to Topic2 of another device 2 with subscription permission. Device 2 subscribes to Topic2 and can receive the Topic1 message of device 1, thus realizing communication between different devices. In the specific implementation operation, it is necessary to create a data destination on the Alibaba Cloud platform, and then configure and start the parser.

[0147] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0148] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0151] Obviously, the above embodiments are merely examples for the purpose of clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the present invention.

Claims

1. A human body posture recognition method, characterized in that: The following steps are involved: Construct an OpenPose skeleton point algorithm model, the OpenPose skeleton point algorithm model includes: a MobileNet V3 network, two parallel multi-stage loop branches; wherein each stage of each branch of the two parallel multi-stage loop branches includes: a plurality of distributed shift convolutions, a transition layer, and a CA attention mechanism module sequentially connected along the forward propagation direction; the transition layer includes: a normalization layer, an activation function layer, and a 1×1 convolution layer; Inputting the image to be tested into the MobileNet V3 network, extracting features from the image to be tested, and obtaining image features; Inputting the image features into the two parallel multi-stage loop branches to extract the key point confidence map and the component affinity field respectively; Based on the key point confidence map and the component affinity field, the coordinate data of the key points of the human body are obtained; The coordinate data of the key points of the human body are input into the multi-layer perceptron and the posture prediction results are output.

2. A human body posture recognition method according to claim 1, characterized in that: The image features are input into two parallel multi-stage loop branches, and each stage extracts the key point confidence map and component affinity field of the input features of the current stage, including the following steps: Input the input features into the two branches of the current stage respectively, and extract the initial key point confidence map and component affinity field of the current stage respectively through multiple distribution shift convolutions of the two branches; The initial key point confidence map and component affinity field of the current stage are respectively reduced in dimension through the transition layer of the corresponding branch to obtain the initial key point confidence map and component affinity field of the current stage after dimension reduction; The initial key point confidence map and component affinity field of the current stage after dimensionality reduction are respectively adjusted through the CA attention mechanism module of the corresponding branch to adjust the weight of each channel to obtain the key point confidence map and component affinity field of the input features of the current stage.

3. A human body posture recognition method according to claim 1, characterized in that: The CA attention mechanism module includes: a coordinate information embedding module and a CA generation module.

4. A human body posture recognition method according to claim 1, characterized in that: The activation function of the depthwise separable convolution in the MobileNet V3 network is the PReLU activation function.

5. A human body posture recognition method according to claim 4, characterized in that: The training process of the OpenPose skeleton point algorithm model is as follows: obtain a posture recognition dataset, preprocess the posture recognition dataset, and divide the preprocessed posture recognition dataset into a training set, a validation set, and a test set; use the training set to train the OpenPose skeleton point algorithm model, and use the transfer learning method to prioritize the training of the parameters of the PReLU activation function of the deep separable convolution in the MobileNet V3 network. After the parameters of the PReLU activation function are determined, the overall training of the OpenPose skeleton point algorithm model is performed, and the trained OpenPose skeleton point algorithm model is verified by the validation set to obtain the target OpenPose skeleton point algorithm model.

6. A human body posture recognition method according to claim 5, characterized in that: The posture recognition dataset is the PoseTrack dataset, and adds diverse samples and corresponding time series data for preset application scenarios.

7. A human body posture recognition method according to claim 1, characterized in that: The loss function of each stage of the two parallel multi-stage loop branches in the OpenPose skeleton point algorithm model is a cross entropy loss function.

8. A human body posture recognition method according to claim 1, characterized in that: The method of obtaining the coordinate data of the key points of the human body based on the key point confidence map and the component affinity field includes: calculating the coordinate data of the key points of the human body through a pair matching method in graph theory based on the key point confidence map and the component affinity field.

9. A human body posture recognition method according to claim 1, characterized in that: The activation function of the multi-layer perceptron output layer is the tanh function.

10. A human body posture recognition device, characterized in that: include: An image acquisition device, used to acquire the image to be tested in real time and transmit the image to be tested to the edge computing device; The edge computing device is used to obtain the posture prediction results based on the image to be tested, connect to the cloud platform through the 5G communication module, and upload the posture prediction results to the user terminal, including: A lightweight model building module, used to build an OpenPose skeleton point algorithm model, wherein the OpenPose skeleton point algorithm model includes: a MobileNet V3 network and two parallel multi-stage loop branches; Each stage of each branch of the two parallel multi-stage loop branches includes: multiple distributed shift convolutions, transition layers, and CA attention mechanism modules connected in sequence along the forward propagation direction; the transition layer includes: a normalization layer, an activation function layer, and a 1×1 convolution layer; An image extraction module is used to input the image to be tested into the MobileNet V3 network, extract features from the image to be tested, and obtain image features; A key point confidence map and component affinity field acquisition module, used for inputting image features into the two parallel multi-stage loop branches to extract key point confidence maps and component affinity fields respectively; A coordinate data acquisition module is used to acquire coordinate data of key points of the human body based on the key point confidence map and the component affinity field; The recognition module is used to input the coordinate data of the key points of the human body into the multi-layer perceptron and output the posture prediction results; Cloud platform: used to store human body posture prediction results and support uploading and downloading of the human body posture prediction results; User terminal: used for users to receive and view posture prediction results in real time and provide human-computer interaction.

Images