Pose recognition method and system combining environmental adaptation with estimation and classification, and medium

Abstract

The present invention relates to a pose recognition method and system combining environmental adaptation with estimation and classification, and a medium, the method comprising: on the basis of an application scenario, obtaining a human pose estimation model; inputting image data to be tested into the human pose estimation model to output human pose keypoints; performing data enhancement and normalization on the human pose keypoints; inputting the human pose keypoints that have undergone data enhancement and normalization and high-order pose features into a pose classification model to obtain a pose recognition result, the high-order pose features comprising bone vectors, the mean of bone vector moduli, relative lengths of bones, and an included angle between the line connecting each human pose keypoint and a trunk midpoint and the horizontal direction. Thus, according to the present invention, pose recognition is divided into two processes of pose estimation and pose classification, which facilitates algorithm updating and iteration being performed when there is better estimation and classification performance, and the structure is flexible.

Description

Gesture recognition method, system and medium combining environmental adaptation and estimation classification Technical Field

[0001] The present invention relates to a posture recognition method, system and medium combining environmental adaptation with estimation and classification, and relates to the field of computer image processing. Background Art

[0002] Human posture recognition has been extensively studied in the field of computer vision. It estimates the positions of human body parts in sensor-captured input data (images and video data) and uses this information to comprehensively determine the current posture of the human body. Human posture recognition technology, which leverages the body's geometric and motion information, has been widely applied in fields such as human-computer interaction, motion analysis, augmented reality (AR), virtual reality, and healthcare.

[0003] In recent years, various deep learning solutions have rapidly developed and have been proven in various tasks such as image classification, semantic segmentation, and object detection. Similarly, deep learning technology has made significant progress and achieved remarkable results in human pose recognition. However, deep learning-based human pose recognition methods still face challenges such as occlusion, insufficient training data, and ambiguity that need to be overcome.

[0004] SUMMARY OF THE INVENTION

[0005] Due to the difficulty of performing posture classification on incomplete posture estimation results in complex scenarios such as occlusion or blurring of human body parts, posture recognition faces difficulties. The present invention provides a posture recognition method, system and medium that combines environmental adaptation and estimation classification, including: through posture estimation (key point detection) and posture classification, introducing high-order posture features such as bone length and angle based on the coordinate input of the posture key points of the classification network, modeling the corresponding relationship between posture features and posture categories at different levels and scales such as posture point differential coordinates, relative bone length, posture point-center connection line and horizontal angle, and adding data processing to the output of the fine-tuned posture estimation model, integrating some manual annotations, achieving the purpose of data cleaning and data augmentation, making the input features of the classification network fit the actual application scenario, and reducing the negative impact of the incomplete estimated skeleton map caused by occlusion, light and shadow on the subsequent posture recognition classification task. Therefore, the advantages of recognition based on posture feature input compared with RGB pixel-level input are less information redundancy, lower operation and training threshold, stronger interpretability, and better environmental adaptability. In addition, the present invention is not limited to the coordinate feature input of key points. In order to avoid the influence of the position offset of the human body in the picture, high-order posture features such as relative bone length and the angle between the key point-torso line and the horizontal direction are introduced. These features not only contain more posture information but also can well solve the negative impact caused by the uneven distribution of the human body in the picture. Technical issues

[0006] Currently, many high-performance implementations of 2D pose estimation exist for simple scenes. However, research on classifying different pose categories based on different estimation results is relatively limited. In complex scenes, estimation algorithms may not be able to provide a complete sequence of pose keypoints, that is, the complete human skeleton structure. Filling missing keypoints with zeros for overall classification will inevitably negatively impact pose recognition. In summary, pose classification based on incomplete pose estimation results in complex scenes, such as those with occlusion or blurring of human parts, is a major challenge for pose recognition. Technical Solutions

[0007] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0008] In a first aspect, the present invention provides a posture recognition method combining environmental adaptation with estimation and classification, comprising:

[0009] Obtain a human posture estimation model based on the application scenario;

[0010] Input the image data to be detected into the human posture estimation model to output the key points of human posture;

[0011] Perform data enhancement and normalization on key points of human posture;

[0012] The human posture key points and high-order posture features after data enhancement and normalization are input into the posture classification model to obtain the posture recognition results. Among them, the high-order posture features include bone vectors, the mean of the bone vector modulus, the relative length of each bone, and the angle between the line connecting the human posture key points and the midpoint of the torso and the horizontal direction.

[0013] Furthermore, a human posture estimation model is obtained based on the application scenario, including:

[0014] The human pose estimation model is pre-trained using the keypoints portion of the open source pose dataset COCO. The human pose estimation model uses the Openpose network model, which consists of a backbone network and two branches: the Heatmap branch and the Paf branch. These two branches are used to extract image features, obtain the location information of key points, and the attribution of each key point to the human body.

[0015] Taking the human pose estimation pre-trained model trained on the open source pose dataset as the starting point, the human pose estimation model is retrained using specific application scenario data and manual annotation of pose key points to achieve fine-tuning of the human pose estimation model.

[0016] Furthermore, the human body posture key points are subjected to data enhancement and normalization, including:

[0017] The key points of human posture are screened for valid points according to the preset integrity conditions, samples that do not meet the integrity conditions are manually labeled, and key points that meet the integrity conditions but are not detected are filled with 0;

[0018] The horizontal and vertical coordinates of each key point after screening are normalized.

[0019] Furthermore, the human body posture key points include 13 key points, namely: head (Head) key point, neck (Neck) key point, left shoulder (L_Shoulder) key point, right shoulder (R_Shoulder) key point, left elbow (L_Elbow) key point, right elbow (R_Elbow) key point, left wrist (L_Wrist) key point, right wrist (R_Wrist) key point, trunk (Spine) key point, left knee (L_Knee) key point, right knee (R_Knee) key point, left ankle (L_Ankle) key point and right ankle (R_Ankle) key point.

[0020] Furthermore, the human body posture key points and high-order posture features after data enhancement and normalization are input into the posture classification model to obtain posture recognition results, including:

[0021] The human body posture key points after data enhancement and normalization are differentiated, and high-order posture features are extracted based on the human body skeleton graph;

[0022] The human body posture coordinate point differences and high-order posture features are uniformly padded with zeros to form a matrix, and then concat-inputted into the trained posture classification model in the channel dimension to obtain the posture recognition result.

[0023] Furthermore, the input features of the posture classification model are:

[0024] Among them, padding represents the supplementary matrix operation, concat represents splicing, and the calculation formulas of the high-order posture features in the above formula are:

[0025] Where Joint is the difference between the coordinates of each posture key point and the coordinates of the torso center point, w i , u are the key points of any posture, center is the spine key point, x and y refer to the horizontal and vertical coordinates of the key point respectively; represents the bone vector; scale is the mean of the bone vector modulus; bone represents the normalized bone feature, i.e., the relative length of the bone; angle represents the angle feature, i.e., the angle between the line connecting the posture key point and the midpoint of the torso and the horizontal direction.

[0026] Furthermore, the posture classification model uses Attention_ResUnet for feature extraction, adds a classification head after the final_conv layer, selects Adam as the optimizer, uses multi-classification cross entropy as the loss function, and introduces the attention mechanism ResUnet into the posture classification model.

[0027] In a second aspect, the present invention further provides a posture recognition system that combines environmental adaptation with estimation and classification, the system comprising:

[0028] A model acquisition unit is configured to obtain a human posture estimation model based on an application scenario;

[0029] A posture key point acquisition unit is configured to input the image data to be detected into the human posture estimation model and output the human posture key points;

[0030] A data processing unit is configured to perform data enhancement and normalization processing on key points of human posture;

[0031] The posture classification unit inputs the human posture key points and high-order posture features after data enhancement and normalization into the posture classification model to obtain the posture recognition results.

[0032] In a third aspect, the present invention also provides an electronic device comprising: one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the methods.

[0033] In a fourth aspect, the present invention also provides a computer-readable storage medium storing one or more programs, characterized in that the one or more programs include instructions that, when executed by a computing device, enable the computing device to perform any of the methods. Beneficial effects

[0034] The present invention adopts the above technical solution, which has the following characteristics:

[0035] 1. The present invention does not have strict integrity requirements for the skeleton structure output by posture estimation, which makes it more adaptable to recognition in complex scenes such as occlusion. It can also enhance the performance in specific scenes by fine-tuning the model in different application scenarios.

[0036] 2. The present invention splits posture recognition into two processes: posture estimation and posture classification, which facilitates algorithm update and iteration when better estimation and classification are available, and has a flexible structure.

[0037] 3. Since the present invention identifies and classifies the posture features output by posture estimation, it can effectively avoid the adverse effects of complex scenes such as light and shadow changes on RGB pixel-level algorithms. In addition, the posture data is tightly distributed without a large amount of redundancy, which is more friendly to the algorithm's computing power and resource consumption, and has strong robustness.

[0038] 4. The key point coordinates of the present invention only provide information on joint positions, while the relative bone lengths and line slopes can provide the proportions and relative positional relationships between joint points. This makes the posture description more comprehensive and can capture the relative extension and bending degrees between different skeletal parts, thereby expressing the posture more accurately.

[0039] 5. The human posture estimation model adopted in the present invention is based on the Openpose network. It is a preprocessing method for the data between the estimation network output and the classification network input in posture recognition that adapts to different application scenarios and demand environments. Unlike the traditional coordinate input of posture point sequence, the present invention adopts a classification feature extraction method that integrates relative coordinates and high-order features such as bone length and angle to solve data imbalance problems such as posture point position bias.

[0040] 6. The present invention adds data augmentation, cleaning and normalization between posture estimation and posture classification. In addition to using the posture features output by posture estimation as the training input of the posture classification network, it also manually labels a part of the posture data, and masks a certain proportion of the originally complete manually labeled skeleton map, filtering out the part with too few detectable points in the posture estimation network output, thereby adapting to the situation where the posture estimation features are incomplete in actual applications, and eliminating the influence of the unreliable skeleton map generated by the estimation network on the classification training.

[0041] In summary, the present invention can derive the posture of a human body based on pictures or video frames, and can be widely applied to bus stations, railways and other related scenes to realize the recognition of human posture characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. Throughout the drawings, the same reference numerals are used to denote the same components. In the drawings:

[0043] FIG1 is a flowchart of estimation-inter-classification data preprocessing according to an embodiment of the present invention.

[0044] FIG2 is a flow chart of data processing and feature extraction according to an embodiment of the present invention.

[0045] Figure 3(a) is a schematic diagram of the human skeleton of the traditional Openpose algorithm.

[0046] FIG3( b ) is a schematic diagram of human skeleton reconstruction according to an embodiment of the present invention.

[0047] FIG4 is a flow chart of the posture classification stage according to an embodiment of the present invention.

[0048] FIG5( a ) is a sample picture of a gesture recognition system according to an embodiment of the present invention.

[0049] FIG5( b ) is a sample picture of a gesture recognition system according to another embodiment of the present invention.

[0050] FIG6 is a schematic diagram of an electronic device according to an embodiment of the present invention. Modes for Carrying Out the Invention

[0051] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the present invention to those skilled in the art.

[0052] Embodiment 1: This embodiment provides a posture recognition method that combines environmental adaptation with estimation and classification, including:

[0053] S1. In this embodiment, as shown in FIG1 , an open source posture dataset is used to perform posture estimation to obtain key points of human posture, including:

[0054] S11. Use the keypoints part of the posture open source dataset (COCO dataset, as shown in Figure 5(a) and Figure 5(b)) to pre-train the human posture estimation model.

[0055] As shown in Figure 3(a), the human posture estimation model can predict 18 posture key points of the human body. The 18 posture key points of the human body include 5 facial key points and 13 torso key points. The facial key points are: left eye, right eye, left ear, right ear and nose key points; the torso key points are: left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, right ankle and neck key points.

[0056] Similar to the strategy adopted by Openpose, this embodiment artificially sets the midpoint between the left and right shoulders as the "Neck" key point. The COCO keypoints human key point dataset is processed to add the "Neck" key point, which is expressed as follows:

[0057] Among them, xL_Shoulder, xR_Shoulder, yL_Shoulder, and yR_Shoulder are the coordinate values ​​of the left shoulder key point and the right shoulder key point respectively.

[0058] The human body posture key point dataset and the "Neck" key point are input into the human body posture estimation model (CNN Based), and the human body posture estimation model (2D) is obtained after training.

[0059] Furthermore, the human pose estimation module uses the Openpose model, whose network consists of two branches: the Heatmap branch and the Paf branch, which respectively model the location information of key points and the relationship between each point and the human body. Since the network is a cascaded network of t stages in series, each stage corresponds to the output of two branches. These two branches share weights, and parameters can be optimized simultaneously during gradient backpropagation. Therefore, the loss function of stage_t is as follows:

[0060] The ground truth value, W, is a binary mask where W(p) = 0 when an annotation is missing at image position p. This mask is used to avoid penalizing true predictions during training. Intermediate supervision at each stage solves the vanishing gradient problem by periodically replenishing the gradient. The overall goal is to:

[0061] The goal is to minimize this f. The present invention uses the pre-trained human pose estimation model obtained by the Openpose model on the COCO dataset, which can be fine-tuned according to the application scenario.

[0062] Furthermore, the OpenPose network structure is a multi-stage convolutional neural network (CNN) that extracts information about the human body posture in the image through step-by-step processing, including:

[0063] Backbone Network: The backbone network used by OpenPose is a variant of ResNet or VGGNet. This backbone network is responsible for extracting features from the input image and gradually reducing the size of the feature map through layered convolution and pooling operations.

[0064] PAFs branch (Part Affinity Fields Branch): Based on the backbone network, OpenPose introduces a PAFs branch to predict the connection information between joints. PAFs is a vector field that represents the relationship between joints and is used to determine the connection relationship between body parts. This branch consists of a series of convolutional layers to predict the connection between each pair of joint points from the feature map.

[0065] Heatmaps branch: In addition to the PAFs branch, OpenPose also includes a Heatmaps branch to predict the position probability distribution of each joint point. Heatmaps are two-dimensional heat maps corresponding to each joint point, which are used to determine the precise location of the joint point. The Heatmaps branch also consists of a series of convolutional layers to generate a position probability map for each joint point from the feature map.

[0066] The loss function of OpenPose mainly consists of two parts, namely PAFs loss and Heatmaps loss: PAFs loss Heatmaps loss is used to measure the difference between the predicted joint connection vector and the true connection vector. This loss function calculates the Euclidean distance between the predicted PAFs and the true PAFs, as well as the direction cosine similarity between the predicted PAFs and the true PAFs. By minimizing these two differences, the model can learn the ability to accurately predict the connection relationship between joints. Used to measure the error between the predicted joint point location probability map and the true location probability map, this loss function calculates the mean square error (MSE) between the predicted heatmaps and the true heatmaps. By minimizing this error, the model can learn the ability to accurately predict joint point locations.

[0067] During training, the weighted sum of the PAFs loss and the Heatmaps loss is used as the overall loss, as shown in Equation (2). Since this two-branch structure is a multi-layer cascade with more than one layer, each layer is called a stage, represented by t in Equation (2). The final loss should be the sum of the loss functions of each stage, and the backpropagation algorithm and gradient descent optimization method are used to update the network parameters to reduce the overall loss. Depending on the task requirements and the characteristics of the dataset, different losses can be weighted to better adjust the model's emphasis on connection relationships and joint point locations.

[0068] In summary, the OpenPose loss function consists of the PAFs loss and the Heatmaps loss, which measure the difference between the predicted joint connection vectors and position probability maps and the ground truth, respectively. Through backpropagation and gradient descent optimization, the model gradually learns to accurately predict human poses. It's also important to note that in the OpenPose algorithm's loss function, w(p) is a weight parameter used to balance the importance of different joint positions. Specifically, w(p) represents the weight assigned to each joint position. In real applications, different joints may have different importance. For example, in human pose estimation, the accuracy of joints such as the knee and torso may be more important than other parts such as the wrist or face. To account for this difference in importance, the weight parameter w(p) is used to adjust the contribution of each joint position to the loss. Generally, w(p) can be set based on task requirements and dataset characteristics. During training, adjusting the value of w(p) can help the model focus more on important joint positions, thereby improving their accuracy. This value can be adjusted based on the actual scenario.

[0069] S12. In order to better adapt to the application scenario, the trained human posture estimation model is fine-tuned in the application scenario to obtain the final human posture estimation model.

[0070] In this embodiment, a batch of images were manually collected to annotate the posture data. The key point distribution was consistent with the 18 key points preset by Openpose. The human posture estimation model was fine-tuned based on the collected application scenario data and manual annotation of posture key points to obtain the final human posture estimation model.

[0071] Furthermore, fine-tuning refers to retraining a pre-trained human pose estimation model using image data collected from specific scenarios to adapt the model to new tasks or domains. During fine-tuning, a human pose estimation model pre-trained on a publicly available dataset can be used as a starting point, and then retrained using image data collected from the new application scenario. The goal is to retain the human pose estimation model while making it more adaptable to the specific scenario that the current task needs to solve, and to improve the performance of the human pose estimation model in that scenario.

[0072] Furthermore, the fine-tuning process of the human pose estimation model includes: collecting and labeling data for specific application scenarios, adjusting the parameters of the pre-trained human pose estimation fine-tuning model (such as learning rate, etc.) to adapt to the new data; training and optimizing the model based on the new data. In this way, the performance of the human pose estimation model can be improved using new data, and convergence can usually be reached faster.

[0073] S2. Perform data enhancement and normalization on the output data of the human posture estimation fine-tuning model.

[0074] In this embodiment, after the image data is input into the final human pose estimation model, the coordinates of the human pose key points are obtained, thereby determining the human skeletal structure. Due to complex environmental factors such as occlusion, not all key points can be detected. Therefore, data augmentation processing is performed on the output data of the human pose estimation model, including data cleaning and partial manual labeling, to ensure that the augmented human pose data is accurate and highly distinguishable.

[0075] As shown in Figure 2, after obtaining the pose estimation output, valid points of human pose key points are screened. Parts with too few detected key points are filtered out using the set integrity condition 1. After meeting integrity condition 1, the original image undergoes data augmentation, such as rotation, scaling, and / or translation. Parts that do not meet integrity condition 1 are manually annotated. Manual annotation involves sending video frames that do not meet integrity condition 1 to the manual annotation process and annotating them according to the Openpose human model format. The augmented image is then subjected to integrity condition 2 for another round of valid point screening. Integrity condition 2 is more stringent than integrity condition 1 and can be adjusted based on actual needs. Samples that do not meet integrity condition 2 are manually annotated, and key points that were not detected by integrity condition 2 are padded with zeros. The horizontal and vertical coordinates of each filtered key point are then normalized. Integrity conditions 1 and 2 can be set based on specific scenarios and business needs, typically thresholds for the number of valid points, but are not specified here.

[0076] S3. Posture Classification

[0077] In this embodiment, for the feature input of the posture classification network, unlike previous studies that only use the coordinates of the key points of the human posture as input, the present invention takes into account that if the position distribution of the human body in the picture is uneven when making different postures, the network will associate the position bias of the coordinates with the posture category, thereby adversely affecting the measured scene. In order to expand the ability of the neural network to extract body structure information, the present invention proposes a high-order representation in the form of bone length and angle. The proposed posture features comprehensively capture the relative motion between different parts of the body while maintaining robustness to changes in the target human body. Therefore, the present invention is able to distinguish challenging postures with similar motion trajectories (such as sitting and squatting), which is a difficult problem in previous studies. Experimental results show that these high-order features are complementary to existing coordinate features.

[0078] In this embodiment, the posture estimation network in the trained Openpose format is used to reconstruct the skeleton diagram in order to avoid introducing irrelevant information when calculating high-order features such as bone length and angle. The previous facial key points are omitted, and the head is represented by the Head node and the Neck node. The Spine node is added to calculate the angle with the horizontal direction with other estimated key points as the angle feature. Therefore, after calling the human posture estimation model to obtain the coordinates of the key points, the coordinates of the center point of the torso are subtracted from all coordinates to ensure that the posture features are in a relatively uniform position when input into the classification network, thereby weakening the influence of position bias on classification.

[0079] Furthermore, the topological structure of the skeleton graph was calculated for each bone (defined as the geometric distance between two key points) and the angle between the horizontal line connecting each bone and the torso center. It was discovered that the original Openpose skeleton model did not contain all key points that were useful for distinguishing different postures (especially those involving standing, sitting, squatting, and lying down). Therefore, the original estimated 18-keypoint skeleton was converted to the current 13-keypoint skeleton, as shown in Figure 4. Compared with the previous skeleton structure, as shown in Figure 3(a), the new skeleton omits facial keypoint details and adds a description of the torso (spine, head, neck). Experiments confirmed that the reorganized skeleton structure is beneficial for distinguishing different posture features.

[0080] In this embodiment, as shown in FIG4 , the specific process of posture classification includes:

[0081] S31. After completing the normalization processing of the human body posture key points, perform posture coordinate point difference, and calculate the bone vectors in the skeleton diagram, the mean of the bone vector modulus, the relative length of each bone, and the angle between the line connecting the posture key point and the midpoint of the torso and the horizontal direction;

[0082] S32. The coordinate features, relative bone length features, and angle features of the posture key points are uniformly padded with zeros to form an 8*8 matrix. Then, the matrix is ​​concat- ored in the channel dimension and input into the trained posture classification model to obtain posture recognition results. This improves the classification accuracy while maintaining a low computational cost, thereby supporting real-time posture recognition on edge devices.

[0083] In this embodiment, the 8*8 matrix is ​​used to solve the problem that the existing eigenvalues ​​cannot form a square and be calculated in a deep neural network. As for the value, the optimal solution in this case is obtained by manually adjusting the hyperparameters. This is taken as an example and is not limited to this.

[0084] In this embodiment, the Chinese meaning of "channel" is "channel". Since it is not appropriate to directly connect information at different levels for convolution processing, it is necessary to expand a new dimension (i.e., 8*8*1) from the two-dimensional (8*8) and place them side by side on the newly created dimension (i.e., the feature dimension after splicing becomes 8*8*3). After convolution, feature information at different semantic levels can be extracted.

[0085] In this embodiment, the present invention introduces the relative skeleton length and the horizontal angle between the center line of key points in the high-order posture features. This makes the key point information no longer an isolated value, and has a significant effect on specifically distinguishing between sitting, squatting, lying, and other actions. The calculation formula of the high-order posture features is as follows:

[0086] In the formula, joint represents the key point of posture, w i , u is the arbitrary posture key point part, center refers to the central posture key point after the previous reconstruction of the human skeleton diagram (corresponding to the spine posture key point in the new skeleton diagram), x and y refer to the horizontal and vertical coordinates of the key point respectively; The arrow above indicates that it is a directed vector, which is obtained by subtracting the directed vectors of the line connecting the two end posture points and the center point; scale represents the scaling factor of the bone length. In order to offset the negative bias effect of the human body's distance from the camera on this feature, it is essentially The average value of the vector modulus; bone represents the normalized bone feature. At this time, the arrow above is gone, which means it is a scalar with only size information. The calculation method is each The modulus value is divided by scale; angle represents the angle feature, which is specifically described as the angle between the line connecting the current posture key point and the center posture key point and the horizontal direction, where Represents the slope, and taking arctan(·) of the slope yields the angle between the line and the horizontal direction; padding represents the aforementioned peripheral padding operation of 0 to 8*8 matrix, padding the posture features (coordinates, bones, angles) of each level with 0, then expanding the dimension to (8*8*1), concat (splicing) them together, and the dimension is (8*8*3). Among them, formula (3) is the difference between the coordinates of each posture key point and the coordinates of the torso center point to weaken the position bias, formula (4) is the calculation of the bone vector in the skeleton graph, formula (5) is the calculation of the mean value of the bone vector modulus, formula (6) is the calculation of the relative length of each bone, formula (7) is the calculation of the angle between the line connecting the posture key point and the torso midpoint and the horizontal direction, and formula (8) is the input feature of the posture classification network. The coordinate features, relative bone length features, and angle features of the posture key points are uniformly padded with zeros to an 8*8 matrix, and then spliced ​​in the channel dimension and input into the posture classification network.

[0087] Furthermore, the filtered posture data is input into the posture classification network for training to obtain the posture classification model. Considering that posture features not only contain information about some parts but also certain overall information, and the correlation between key points in the skeleton needs to be well modeled, the posture classification model chooses ResUnet with the introduction of the attention mechanism. This supports multi-scale extraction of posture features while learning the dependencies between posture key points. This is the model structure of Attention_ResUnet.

[0088] For the posture classification network, this embodiment uses Attention_ResUnet for feature extraction, adds a classification head after the final_conv layer, selects Adam as the optimizer, and uses multi-classification cross entropy as the loss function. The formula is as follows:

[0089] Among them, p ic is the posterior probability that sample i is classified into category c. In order to make the sum of the posterior probabilities equal to 1, a Softmax layer is applied to the output. The formula is as follows: ic =Softmax(y ic W T +b)

[0090] Convolution can be abstracted as a linear change: ic =y ic W T +b.

[0091] Among them, i is the ordinal number of the sample, c represents the category, and applying softmax ensures that the resulting distribution will not be affected by the maximum and minimum values, and ensures that the sum is 1, which conforms to the definition of probability. In deep neural networks, the last convolution layer is usually used to extract features and generate a vector of corresponding outputs. However, these outputs are not restricted to a specific range, nor are they normalized to a probability distribution. Therefore, after the last layer of convolution, a softmax function is usually used to convert the output into a probability distribution. The softmax function can convert the elements in a vector into non-negative numbers and ensure that the sum of all elements is equal to 1. It does this by calculating the exponent of each element and then dividing it by the sum of all elements. In this way, the output of the last layer can be interpreted as the probability of each category.

[0092] Example 2: The above-mentioned Example 1 provides a gesture recognition method that combines environmental adaptation with estimation and classification. Correspondingly, this embodiment provides a gesture recognition system that combines environmental adaptation with estimation and classification. The system provided in this embodiment can implement the gesture recognition method that combines environmental adaptation with estimation and classification of Example 1. The system can be implemented through software, hardware, or a combination of software and hardware. For ease of description, this embodiment is described separately based on its functions in various units. Of course, during implementation, the functions of each unit can be implemented in the same or multiple software and / or hardware. For example, the system can include integrated or separate functional modules or functional units to perform the corresponding steps of each method in Example 1. Because the system of this embodiment is basically similar to the method embodiment, the description of this embodiment is relatively simple. For relevant details, please refer to the partial description of Example 1. The embodiment of the gesture recognition system that combines environmental adaptation with estimation and classification provided by the present invention is merely illustrative.

[0093] Specifically, this embodiment provides a posture recognition system that combines environmental adaptation with estimation and classification, and the system includes:

[0094] A model acquisition unit is configured to obtain a human posture estimation model based on an application scenario;

[0095] A posture key point acquisition unit is configured to input the image data to be detected into the human posture estimation model and output the human posture key points;

[0096] A data processing unit is configured to perform data enhancement and normalization processing on key points of human posture;

[0097] The posture classification unit inputs the human posture key points and high-order posture features after data enhancement and normalization into the posture classification model to obtain the posture recognition results.

[0098] Embodiment 3: This embodiment provides an electronic device corresponding to the posture recognition method combined with environmental adaptation and estimation classification provided in this embodiment 1. The electronic device can be an electronic device used for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the method of embodiment 1.

[0099] As shown in Figure 6, the electronic device includes a processor, a memory, a communication interface and a bus, and the processor, the memory and the communication interface are connected by a bus to complete communication with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Component (EISA) bus, etc. A computer program that can be run on the processor is stored in the memory, and the processor executes the method of embodiment 1 when running the computer program. Its implementation principle and technical effect are similar to those of embodiment 1 and are not described in detail here. It will be understood by those skilled in the art that the structure shown in Figure 6 is only a block diagram of a partial structure related to the present application solution, and does not constitute a limitation on the computing device to which the present application solution is applied. The specific computing device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0100] In a preferred embodiment, the logic instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), optical disk and other media that can store program code.

[0101] In a preferred embodiment, the processor may be a central processing unit (CPU), a digital signal processor (DSP), or other general-purpose processors of various types, which are not limited herein.

[0102] Embodiment 4: This embodiment provides a computer program product, which may include a computer program stored on a computer-readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the method provided in the above-mentioned embodiment 1. Its implementation principle and technical effects are similar to those of embodiment 1 and will not be repeated here.

[0103] In a preferred embodiment, a computer-readable storage medium may be a tangible device that retains and stores instructions used by an instruction execution device, such as, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any combination thereof. The computer-readable storage medium stores computer program instructions that cause a computer to execute the method provided in the first embodiment.

[0104] CROSS-REFERENCE TO RELATED APPLICATIONS

[0105] This application claims priority to the Chinese patent application (application number: 202311412624.X) filed on October 27, 2023, and the entire contents of that patent application are incorporated herein by reference. Industrial Applicability

[0106] The present invention can derive the posture of a human body based on pictures or video frames, and can be widely applied to bus stations, railways and other related scenes to realize the recognition of human posture characteristics.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A posture recognition method combining environmental adaptation and estimation classification, characterized in that: include: Obtain a human posture estimation model based on the application scenario; Input the image data to be detected into the human body posture estimation model to output the key points of human body posture; Perform data enhancement and normalization on key points of human posture; The human body posture key points and high-order posture features after data enhancement and normalization are input into the posture classification model to obtain the posture recognition results. The high-order posture features include bone vectors, the mean of bone vector modulus, the relative length of each bone, and the angle between the line connecting the human body posture key points and the midpoint of the torso and the horizontal direction.

2. The posture recognition method combining environmental adaptation and estimation classification according to claim 1, characterized in that: Obtain a human posture estimation model based on the application scenario, including: The keypoints part of the open source posture dataset COCO is used to pre-train the human posture estimation model. The human posture estimation model adopts the Openpose network model. The Openpose network model consists of a backbone network and two branches. The two branches are the Heatmap branch and the Paf branch, which are used to extract features from the image, obtain the location information of key points, and the attribution relationship of each key point to the human body. The human posture estimation pre-trained model trained on the open source posture dataset is used as the starting point, and the human posture estimation model is retrained using specific application scenario data and manual annotation of posture key points to achieve fine-tuning of the human posture estimation model.

3. The posture recognition method combining environmental adaptation and estimation classification according to claim 1, characterized in that: The key points of human body posture are enhanced and normalized, including: The key points of human posture are screened for valid points through the preset completeness condition, samples that do not meet the completeness condition are manually labeled, and key points that meet the completeness condition but are not detected are filled with 0; The horizontal and vertical coordinates of each key point after screening are normalized.

4. The posture recognition method combining environmental adaptation and estimation classification according to claim 1, characterized in that: The human body posture key points include 13 key points, namely: head key point, neck key point, left shoulder key point, right shoulder key point, left elbow key point, right elbow key point, left wrist key point, right wrist key point, torso key point, left knee key point, right knee key point, left ankle key point and right ankle key point.

5. The posture recognition method combining environmental adaptation and estimation classification according to claim 4 is characterized in that: The human body posture key points and high-order posture features after data enhancement and normalization are input into the posture classification model to obtain the posture recognition results, including: The human body posture key points after data enhancement and normalization are differentiated by posture coordinate points, and high-order posture features are extracted based on the human body skeleton graph; The relative coordinate features of the key points of human posture and the high-order posture features are uniformly padded with zeros to form a matrix, and then concat in the channel dimension and input into the trained posture classification model to obtain the posture recognition result.

6. The posture recognition method combining environmental adaptation and estimation classification according to claim 5, characterized in that: The input features of the posture classification model are: Among them, padding represents the supplementary matrix operation, concat represents concatenation, and the calculation formulas of the high-order posture features in the above formula are: Where Joint is the difference between the coordinates of each posture key point and the coordinates of the trunk center point, and w i , u are the key points of any posture, center is the spine key point, x and y refer to the horizontal and vertical coordinates of the key point respectively; represents the bone vector; scale is the mean value of the bone vector modulus; bone represents the normalized bone feature, that is, the relative length of the bone; angle represents the angle feature, that is, the angle between the line connecting the posture key point and the midpoint of the torso and the horizontal direction.

7. The posture recognition method combining environmental adaptation and estimation classification according to claim 1, characterized in that: The posture classification model chooses to introduce the attention mechanism ResUnet. The posture classification model uses Attention_ResUnet for feature extraction, adds a classification head after the final_conv layer, selects Adam as the optimizer, and the loss function is multi-classification cross entropy.

8. A posture recognition system combining environmental adaptation and estimation classification, characterized in that: The system includes: A model acquisition unit, configured to obtain a human posture estimation model based on an application scenario; A posture key point acquisition unit is configured to input the image data to be detected into the human posture estimation model to output the human posture key points; A data processing unit is configured to perform data enhancement and normalization processing on key points of human body posture; The posture classification unit inputs the human body posture key points and high-order posture features after data enhancement and normalization into the posture classification model to obtain the posture recognition result.

9. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any one of the methods described in claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 7.

Images