A human action recognition method based on posture-enhanced relational features

Abstract

The present invention discloses a method for human action recognition based on posture-enhanced relationship features, comprising the following steps: S1: obtaining human posture key point data and surrounding object bounding boxes, and converting the posture key point data into a bounding box format; S2: using a basic network to extract image features, and further obtaining basic features corresponding to the human body, posture, and object through ROIPooling; S3: using the basic features and the bounding boxes to respectively calculate two groups of human-posture relationship features and two groups of human-object relationship features, and fusing the relationship features with the basic features according to specific weights to obtain enhanced features of the human body, posture, and object; S4: sending the three enhanced features into a classifier, and fusing the classification results to obtain scores corresponding to each action, thereby obtaining a final recognition result. Compared with the prior art, the present invention takes into account different situations in action recognition and has the advantages of high recognition accuracy and a wide range of applications.

Description

Technical Field

[0001] The present invention relates to the technical field of human motion recognition, and in particular to a human motion recognition method based on posture-enhanced relationship features. Background Art

[0002] The primary task of action recognition is to enable computers to identify the movements of a person in a given image or video. This topic has long been a hot topic in computer vision research and has found widespread application in fields such as security monitoring, human-computer interaction, and information retrieval. With the continuous advancement of software and hardware technologies, the application scenarios of action recognition are constantly enriching and expanding.

[0003] Although the final output of action recognition is an action label, it differs significantly from general image classification tasks. A notable characteristic of action recognition is that it revolves around the human body. Researchers are primarily interested in the human body and its motion-related elements. Therefore, the problem is not limited to simple classification and may intersect with other topics such as object detection and pose estimation. Furthermore, aspects of the image, such as the background, that are less relevant to human motion, are less important in action recognition.

[0004] Like most computer vision research topics, action recognition has undergone a transition from traditional methods to deep learning approaches, with deep learning methods currently firmly occupying the top of the accuracy rankings. The primary difference between the two approaches is that traditional methods require manual feature design, while deep learning methods utilize convolutional neural networks to automatically extract features. However, both traditional and deep learning approaches recognize the crucial role of contextual features, such as pose and object, in action recognition. In traditional approaches, contextual features are typically designed based on image characteristics and then combined with machine learning methods. Hoai et al. used pose information as key auxiliary information, designing pose feature descriptors to represent specific local body parts, which were then fed into a support vector machine (SVM) classifier for classification. Prest et al., starting from spatial concepts such as distance and position, developed a mathematical model to describe the interactions between people and objects. In deep learning approaches, convolutional neural networks can transform contextual information into more expressive features, leading to a shift in research focus towards identifying and utilizing action-related parts of an image. Zhao et al. also paid attention to the important role of posture information in action recognition, and established the correspondence between posture and local action by designing a suitable mapping method, and finally used the overall action score and the local action score to complete the final classification; Yan et al. used multi-level features from the whole image, the human body to the context candidate area, and combined the attention mechanism to process the features, so that these features have a higher correlation with the action; Zhang et al. believe that although context features are important, the method of providing additional context information is relatively cumbersome, so they try to directly use images and action labels to generate action masks containing action information as context features.

[0005] After analyzing existing image action recognition methods, we believe that several challenges remain. First, they underutilize contextual information. Most methods utilize only one type of contextual information as an aid, which prevents them from effectively covering all scenarios. For example, considering only object information results in poor recognition of actions unrelated to the object; considering only human posture information results in poor recognition of actions involving interaction with objects. Second, they lack consideration of the connection between contextual and global features. Existing methods often process global and contextual features in separate branches and then simply fuse them without considering the relationship between the two. This can lead to misleading contextual information. For example, in some cases, the posture of body parts may appear consistent, but the actual actions may be completely different. This explains why some methods, despite utilizing both posture and object information, still struggle in certain recognition scenarios. Furthermore, these two issues also result in limited scalability. If new contextual features emerge, they typically require a separate branch to process them, preventing the reuse of existing processing methods.

[0006] Therefore, it is of great practical significance to develop an image action recognition method that effectively utilizes contextual information and considers its relationship with global information. Summary of the Invention

[0007] Due to the above-mentioned defects in the prior art, the present invention provides an image action recognition method that effectively utilizes contextual information (especially posture information) and considers its relationship with global information, namely, a human action recognition method based on posture-enhanced relational features, which overcomes the defects of existing image action recognition methods in insufficient utilization of contextual information and lack of consideration of the relationship between contextual features and global features.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] A method for human action recognition based on posture-enhanced relational features comprises the following steps:

[0010] S1: Obtain the human body posture key point data and the surrounding object bounding box, and convert the posture key point data into the bounding box format;

[0011] S2: Use the basic network to extract image features, and further obtain the basic features corresponding to the human body, posture, and object through ROI Pooling;

[0012] S3: Calculate two sets of human-pose relationship features and two sets of human-object relationship features using the basic features and the bounding box, and fuse the relationship features with the basic features according to specific weights to obtain enhanced features of human body, pose, and object;

[0013] S4: The three enhanced features are fed into the classifier, and the classification results are fused to obtain the scores corresponding to each action, thereby obtaining the final recognition result.

[0014] Furthermore, the step S1 specifically includes:

[0015] S11: Detect the object in the image using a convolutional neural network-based target detection method and obtain its bounding box bo;

[0016] S12: Detect human posture using a posture estimation method based on a convolutional neural network to obtain key points of human posture;

[0017] S13: Combine and calculate the key points of the human body posture according to a specific conversion rule, map them to local body parts, scale the calculated parts according to the scaling factor k and the body part scaling rule, and output the posture area b in the bounding box format p .

[0018] Furthermore, the conversion rules of the human posture key points to local body parts are defined as: the nose, left ear, and right ear key points are converted to the head area; the left shoulder, right shoulder, left hip, and right hip key points are converted to the body area; the left wrist and left elbow key points are converted to the left hand area; the right wrist and right elbow key points are converted to the right hand area; the left wrist, left elbow, and left shoulder key points are converted to the left elbow area; the right wrist, right elbow, and right shoulder key points are converted to the right elbow area; the left knee and left ankle key points are converted to the left foot area; the right knee and right ankle key points are converted to the right foot area; the left knee, left ankle, and left hip key points are converted to the left knee area; and the right knee, right ankle, and right hip key points are converted to the right knee area.

[0019] Furthermore, the body part scaling rule is as follows: let the upper left coordinate of a body part be (x0, y0), the length be w0, the width be h0, and the scaling factor k be a decimal that satisfies 0.50≤k≤1.50. Then the final output posture region can be described as:

[0020] Upper left corner horizontal coordinate

[0021] Upper left corner vertical coordinate

[0022] Length w = kw0

[0023] Width h = kh0

[0024] Furthermore, the step S3 specifically includes:

[0025] S31: Basic characteristics of the human body h and the basic features of posture f p Calculate the relationship weight and get the weight w p; Basic characteristics of the human body h and the basic features of the object f o Calculate the relationship weight and get the weight w o .

[0026] S32: Encode the spatial position of the human body bounding box and the posture bounding box to obtain the weight w ploc ; Encode the spatial position of the human body bounding box and the object bounding box to obtain the weight w oloc .

[0027] S33: According to the posture-related weight w p and w ploc And the basic features of human body and posture are used to calculate the relationship features f between two groups of posture enhancement hp and f ph , according to the object-related weight w o and w oloc And the basic features of the human body and objects are used to calculate the enhanced relationship features f between the two groups of objects ho and f oh .

[0028] S34: Assign specific weights to the four groups of relationship features, and fuse the two relationship features dominated by the human body to output three relationship features.

[0029] Furthermore, the posture relationship weight w p The calculation method is:

[0030]

[0031] Among them, the F function is a feature dimension conversion function, which converts the two basic features into the same linear space through the fully connected layer for easy processing. k The dimension of the basic feature after the conversion is completed.

[0032] Calculate the object relationship weight w o Just replace the pose basic features with the object basic features.

[0033] Furthermore, the pose space position encoding weight w ploc The calculation method is:

[0034] w ploc =F·L(b h ,b p )

[0035] Among them, the F function is a dimension conversion function, which can be in the form of a fully connected layer or a one-dimensional convolution; the L function is a spatial position encoding function in the form of a trigonometric function, which encodes the coordinates of the human body bounding box and the posture bounding box.

[0036] Calculate the object space position encoding weight w oloc Just replace the pose bounding box with the object bounding box.

[0037] Furthermore, the posture-enhanced relational feature f hp The calculation method is:

[0038] f hp =F(f p ·Σ(w ploc w p ))

[0039] Among them, the F function is a dimension conversion function, which can be in the form of a fully connected layer or a one-dimensional convolution. To calculate another set of posture-enhanced relationship features f ph , just change the weight w p Perform transposition.

[0040] Calculate the object enhanced relation feature f ho With f oh Just replace the weights and features with those corresponding to the object.

[0041] Furthermore, for the obtained sets of relationship features, weighting and fusion rules are defined.

[0042] The weighting rules for the four groups of relationship features are:

[0043] f hp :fh o: f ph :f oh =k1:k2:k3:k4

[0044] Among them, k1, k2, k3, and k4 satisfy:

[0045] 0 <k1,k2<k3,k4

[0046] The weighted relational features are fused with the corresponding basic features through addition according to the following rules to output enhanced relational features:

[0047] f hr =f h +f hp +f ho

[0048] f pr =f p +f ph

[0049] f or =f o +f oh

[0050] That is, the final output is 3 enhanced features fhr 、f pr and f or .

[0051] Furthermore, the step S4 specifically includes:

[0052] S41: The three enhanced features f hr 、f pr and f or Send to the classifier for classification;

[0053] S42: Fusing the classification results corresponding to each enhanced feature to obtain the overall classification result, i.e., the scores of various actions;

[0054] S43: Use the softmax function to convert the action scores into probabilities. The action with the highest probability is the final recognition result. At the same time, the cross entropy function is used to calculate the loss for backpropagation.

[0055] Furthermore, the classification result fusion rule is defined as follows:

[0056] Let f hr 、f pr and f or The corresponding classification result is Cls h 、Cls p and Cls o , then the classification score of action a after the classification results are fused is:

[0057] Cis final (a)=Cls h (a)+maxCls p (a)+maxCls o (a)

[0058] Among them, max selects the maximum value of the classification scores of multiple posture areas or objects for action a.

[0059] Compared with the prior art, the present invention has the following advantages:

[0060] 1) The present invention designs a conversion rule for converting human posture key points into posture regions. Each region is generated by calculating multiple key points, so that the generated regions can reflect actual movements. This realizes the process of converting posture key points that cannot be directly used into easy-to-use and information-rich posture regions.

[0061] 2) The present invention uses both posture and object contextual information to assist in human action recognition, and also considers the relationship between these two types of information and higher-level human body information. Multiple contextual information allows different situations of action recognition to be taken into account. Learning the relationship between contextual features and human body features suppresses potentially misleading contextual information and compensates for the shortcomings that multiple contextual information may introduce. The combination of these two enables more accurate action recognition.

[0062] 3) The present invention takes into account the scalability of the solution, and adopts a similar solution to process the two types of contextual information, posture and object. If new contextual information needs to be introduced, it only needs to be converted into a bounding box format, and the original processing solution can be reused. Due to the introduction of multiple groups of contextual features, the present invention adds a relationship feature weighting step. The user can assign the weights of the relationship features corresponding to each group of contextual features as needed, making the solution more universal. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The following detailed description of non-limiting embodiments is provided with reference to the accompanying drawings.

[0064] Figure 1 Schematic diagram of the workflow of the human action recognition method based on posture-enhanced relationship features of the present invention;

[0065] Figure 2 It is a framework diagram of the human action recognition method based on posture-enhanced relational features of the present invention;

[0066] Figure 3 It is a flow chart of the transformation of posture key points into posture regions;

[0067] Figure 4 It is a flowchart of relational feature calculation;

[0068] Figure 5 It is a visualization diagram for calculating the pose relationship weights. DETAILED DESCRIPTION

[0069] The structure of the present invention is further described below with reference to the accompanying drawings and specific embodiments, but is not intended to limit the present invention.

[0070] Example

[0071] like Figure 1 As shown, the human action recognition method based on posture-enhanced relational features provided by the present invention mainly includes the following four steps:

[0072] 1) Obtain human body posture key point data and surrounding object bounding boxes, and convert the posture key point data into a bounding box format that contains action information and is easy to process;

[0073] 2) Use a mature and reliable basic network to extract image features and map bounding boxes to features through ROI Pooling to further obtain basic features corresponding to the human body, posture, and object;

[0074] 3) Using the basic features and bounding boxes, two sets of human-pose relationship features and two sets of human-object relationship features are calculated respectively. The relationship features are fused with the basic features according to specific weights to obtain enhanced features of human body, pose, and object.

[0075] 4) The three enhanced features are fed into the classifier, and the classification results are fused to obtain the scores corresponding to each action, thereby obtaining the final recognition result.

[0076] Figure 2 based on Figure 1 The workflow shown in the figure shows the basic framework of the entire scheme. Next, this manual will explain the method in detail according to the above four steps.

[0077] The specific method of step 1) is:

[0078] 11) Select target detection and pose estimation methods to process the image to be recognized, obtaining bounding boxes of surrounding objects and pose keypoint data. The main criterion for selection is accuracy. In this embodiment, Faster R-CNN is selected as the target detection method, and AlphaPose is selected as the pose estimation method.

[0079] 12) Generate posture area from posture key points, the process is as follows Figure 3 As shown in the figure, there may be multiple people in the image. To avoid recognizing unnecessary human poses, this solution takes two measures: first, during pose estimation, the required people are cropped and fed into the pose estimation network, directly excluding most unnecessary people; then, if multiple people's poses are still recognized, only the people's poses with high confidence are used to calculate the pose area.

[0080] The obtained pose keypoints are converted according to the following rules: nose, left ear, and right ear keypoints are converted to the head region; left shoulder, right shoulder, left hip, and right hip keypoints are converted to the body region; left wrist and left elbow keypoints are converted to the left hand region; right wrist and right elbow keypoints are converted to the right hand region; left wrist, left elbow, and left shoulder keypoints are converted to the left elbow region; right wrist, right elbow, and right shoulder keypoints are converted to the right elbow region; left knee and left ankle keypoints are converted to the left foot region; right knee and right ankle keypoints are converted to the right foot region; left knee, left ankle, and left hip keypoints are converted to the left knee region; right knee, right ankle, and right hip keypoints are converted to the right knee region. Since actions are generally not completed by a single joint, this multi-keypoint conversion method allows the generated pose regions to better reflect the action.

[0081] The specific calculation method can be adjusted according to different situations. In this embodiment, each posture area is calculated as follows:

[0082] Head: The length and width are the distance between the two ears, with the center point being the nose;

[0083] Body: Length is the greater of the lengths of the shoulders or hips; width is the greater of the lengths of the left shoulder-hip line or the right shoulder-hip line; center is the midpoint of the line between the midpoints of the shoulders and hips;

[0084] Left hand: The left wrist is the center, and the length of the left forearm (from left wrist to left elbow) is the length and width;

[0085] Right hand: The right wrist is the center, and the length of the right forearm (from right wrist to right elbow) is the length and width;

[0086] Left elbow: With the left elbow as the center, the length and width are the sum of the left upper arm (from the left shoulder to the left elbow) and the left forearm;

[0087] Right elbow: With the right elbow as the center, the length and width are the sum of the right upper arm (from right shoulder to right elbow) and the right forearm;

[0088] Left foot: The left ankle is the center, and the length of the left calf (from the left knee to the left ankle) is the length and width;

[0089] Right foot: The right ankle is the center, and the length of the right calf (from the right knee to the right ankle) is the length and width;

[0090] Left knee: With the left knee as the center, add up the length and width of the left thigh (from left hip to left knee) and half of the left calf;

[0091] Right knee: With the right knee as the center, the length and width are the sum of half of the right thigh (from right hip to right knee) and half of the right calf.

[0092] After calculating the preliminary posture area, the posture area can be scaled according to the actual situation. The scaling rules are:

[0093] Assume that the upper left coordinate of a body part is (x0, y0), the length is w0, the width is h0, and the scaling factor k is a decimal that satisfies 0.50≤k≤1.50. The final output posture area can be described as:

[0094] Upper left corner horizontal coordinate

[0095] Upper left corner vertical coordinate

[0096] Length w = kw0

[0097] Width h = kh0

[0098] In this embodiment, the scaling factor k=1.00. The final output is the posture area in the bounding box format, such as Figure 3Shown on the right.

[0099] The specific method of step 2) is:

[0100] 21) The image is fed into a base network to extract features. The base network is selected to balance accuracy and efficiency. In this example, a ResNet-50 network pre-trained on the ImageNet dataset is selected, and its convolution blocks 1 to 4 are used to extract full-image features.

[0101] 22) Use ROIPooling to map the human body bounding box (provided by the dataset), posture area, and object bounding box to the features, and further obtain the basic features f corresponding to the human body, posture, and object h 、f p and f o .

[0102] like Figure 4 As shown, the specific method of step 3) is:

[0103] 31) Basic characteristics of the human body h and the basic features of posture f p Calculate the relationship weight and get the weight w p ; Basic characteristics of the human body h and the basic features of the object f o Calculate the relationship weight and get the weight w o .

[0104] Posture relationship weight w p The calculation method is:

[0105]

[0106] Among them, the F function is a feature dimension conversion function, which converts the two basic features into the same linear space through the fully connected layer for easy processing. k The dimension of the basic feature after the conversion is completed.

[0107] Calculate the object relationship weight w o Just replace the pose basic features with the object basic features.

[0108] 32) Encode the spatial position of the human body bounding box and the posture bounding box to obtain the weight w ploc ; Encode the spatial position of the human body bounding box and the object bounding box to obtain the weight w oloc .

[0109] Pose space position encoding weight w ploc The calculation method is:

[0110] w ploc =F·L(b h ,bp )

[0111] Among them, the F function is a dimensionality conversion function, which can be in the form of a fully connected layer or a one-dimensional convolution. This embodiment uses a fully connected layer; the L function is a spatial position encoding function in the form of a trigonometric function. This embodiment uses a sine / cosine function to encode the coordinates of the human body bounding box and the posture bounding box.

[0112] Calculate the object space position encoding weight w oloc Just replace the pose bounding box with the object bounding box.

[0113] 33) According to the posture-related weight w p and w ploc And the basic features of human body and posture are used to calculate the relationship features f between two groups of posture enhancement hp and f ph , according to the object-related weight w o and w oloc And the basic features of the human body and objects are used to calculate the enhanced relationship features f between the two groups of objects ho and f oh .

[0114] Posture-enhanced relational features fh p的 The calculation method is:

[0115] f hp =F(f p ·∑(w ploc w p ))

[0116] Among them, the F function is a dimension conversion function, which can be in the form of a fully connected layer or a one-dimensional convolution. In this embodiment, a fully connected layer is selected. If another set of posture-enhanced relationship features f is to be calculated ph , just change the weight w p Perform transposition.

[0117] Calculate the object enhanced relation feature f ho With f oh Just replace the weights and features with those corresponding to the object.

[0118] 34) Assign specific weights to the four groups of relationship features, and fuse the two relationship features dominated by the human body to output three enhanced relationship features.

[0119] The weighting rules for the four groups of relationship features are:

[0120] f hp :f ho :f ph :f oh =k1:k2:k3:k4

[0121] Among them, k1, k2, k3, and k4 satisfy:

[0122] 0 <k1,k2<k3,k4

[0123] The relational features output after weighting are fused with the basic features according to the following rules to output enhanced relational features:

[0124] f hr =f h +f hp +f ho

[0125] f pr =f p +f ph

[0126] f or =f o +f oh

[0127] That is, the final output is 3 enhanced features f hr 、f pr and f or .

[0128] like Figure 4 As shown, in this embodiment, from the perspective of strengthening the activation of context information, setting

[0129] f hp :f ho :f ph :f oh =1:1:2:2.

[0130] The specific method of step 4) is:

[0131] 41) The three enhanced features f hr 、f pr and f or Send it to the classifier for classification, and the corresponding classification result is Cls h 、Cls p and Cls o ,In this embodiment, the fully connected layer is selected as the classifier.

[0132] 42) The classification results corresponding to each enhanced feature are fused to obtain the overall classification result, that is, the scores of various actions. The classification result of action a after fusion is:

[0133] Cls final (a)=Cls h (a)+maxCls o (a)+maxCls o (a)

[0134] Among them, max selects the maximum value of the classification scores of multiple posture areas or objects for action a.

[0135] 43) The softmax function is used to convert the action scores into probabilities. The action with the highest probability is the final recognition result. At the same time, the cross entropy function is used to calculate the loss for back propagation.

[0136] Experimental results of human action recognition method based on posture-enhanced relational features:

[0137] To verify the actual performance of this invention in action recognition tasks, experiments were conducted on two well-known public datasets: Pascal VOC 2012Action and Stanford 40Action. Pascal VOC 2012Action provides 4588 images, with more than 2,000 images in the training set and more than 2,000 images in the validation set, covering a total of 11 action categories (including the "other" category). Stanford40Action provides 9532 images, with 4,000 images in the training set and the remaining images in the test set, covering a total of 40 action categories. Both datasets provide human body bounding box annotation data. Consistent with most methods, this solution uses mAP as the evaluation metric.

[0138] Table 1 shows the comparison of the mAP index between the present invention and other methods on the PascalVOC 2012Action validation set.

[0139] Table 1 mAP (%) on the Pascal VOC 2012 Action validation set

[0140]

[0141] According to Table 1, it can be seen that the present invention has achieved the best accuracy on this dataset, and has achieved a certain improvement when the accuracy of this dataset reaches above 90%. It is worth noting that for the action of jumping, this method achieved an AP accuracy of up to 92.3%, which is 2.7 percentage points higher than the original best method. The characteristic of the jumping action is that there is no object to assist, and the local posture is easily confused with actions such as clapping. This shows that for some scenarios where only human posture can be used to assist in action recognition, the present invention can achieve effective utilization of posture information and achieve better results.

[0142] Table 2 shows the comparison of the mAP index between the present invention and other methods on the Stanford 40Action test set.

[0143] Table 2 mAP (%) on the Stanford 40Action test set

[0144]

[0145]

[0146] According to Table 2, we can see that the present invention has achieved the best accuracy on this dataset, and also achieved an improvement when the accuracy of this dataset reached more than 90%. In order to clarify the role of the relational features of posture enhancement on this dataset, Figure 5 A visualization was performed, listing the pose regions with the highest and lowest relationship weights in the selected image. The results show that the present invention can concentrate relationship weights on pose regions most relevant to the action, while irrelevant pose regions receive very low weights. This result indicates that the pose regions generated by pose keypoints contain action information, and that the learning of pose relationship weights activates these pose regions containing action information. This process is similar to how humans visually recognize actions in images, demonstrating that this method can help computers further understand images.

[0147] Those skilled in the art should understand that they can implement variations by combining the prior art with the above embodiments, which will not be described in detail here. Such variations do not affect the essence of the present invention and will not be described in detail here.

[0148] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the devices and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-mentioned disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention that do not depart from the content of the technical solutions of the present invention are still within the scope of protection of the technical solutions of the present invention.

Claims

1. A method for human action recognition based on posture-enhanced relational features, characterized by: The following steps are involved: S1: Obtain the human body posture key point data and the surrounding object bounding box, and convert the posture key point data into the bounding box format; S2: Use the basic network to extract image features, and further obtain the basic features corresponding to the human body, posture, and object through ROIPooling; S3: Use the basic features and bounding box to calculate two groups of human-posture relationship features and two groups of human-object relationship features respectively, and fuse the relationship features with the basic features according to specific weights to obtain the human body, posture, and object enhanced features f hr 、f pr and f or ; S4: The three enhanced features are fed into the classifier and the classification results are fused to obtain the scores corresponding to each action, thereby obtaining the final recognition result; The step S1 specifically includes: S11: Use the target detection method based on convolutional neural network to detect the object in the image and obtain its bounding box b o ; S12: Detect human posture using a posture estimation method based on a convolutional neural network to obtain key points of human posture; S13: Combine and calculate the key points of the human body posture according to a specific conversion rule, map them to local body parts, scale the calculated parts according to the scaling factor k and the body part scaling rule, and output the posture area b in the bounding box format p ; The conversion rules of the human posture key points to local body parts are defined as: the nose, left ear, and right ear key points are converted to the head area; the left shoulder, right shoulder, left hip, and right hip key points are converted to the body area; the left wrist and left elbow key points are converted to the left hand area; the right wrist and right elbow key points are converted to the right hand area; the left wrist, left elbow, and left shoulder key points are converted to the left elbow area; the right wrist, right elbow, and right shoulder key points are converted to the right elbow area; the left knee and left ankle key points are converted to the left foot area; the right knee and right ankle key points are converted to the right foot area; the left knee, left ankle, and left hip key points are converted to the left knee area; the right knee, right ankle, and right hip key points are converted to the right knee area; The body part scaling rule is: let the upper left coordinate of a body part be (x0, y0), the length be w0, the width be h0, and the scaling factor k be a decimal satisfying 0.50≤k≤1.

50. The final output posture region description is: Upper left corner horizontal coordinate Upper left corner vertical coordinate Length w = kw0; Width h = kh0; The step S3 specifically includes: S31: Basic characteristics of the human body h and the basic features of posture f p Calculate the relationship weight and get the weight w p ; Basic characteristics of the human body h and the basic features of the object f o Calculate the relationship weight and get the weight w o ; S32: Encode the spatial position of the human body bounding box and the posture bounding box to obtain the weight w ploc ; Encode the spatial position of the human body bounding box and the object bounding box to obtain the weight w oloc ; S33: According to the posture-related weight w p and w ploc And the basic features of human body and posture are used to calculate the relationship features f between two groups of posture enhancement hp and f ph , according to the object-related weight w o and w oloc And the basic features of the human body and objects are used to calculate the enhanced relationship features f between the two groups of objects ho and f oh ; S34: assign specific weights to the four groups of relationship features, and fuse the two relationship features dominated by the human body to output three relationship features; Posture relationship weight w p The calculation method is: Among them, the F function is the feature dimension conversion function, which converts the two basic features into the same linear space through the fully connected layer for easy processing. k The dimension of the basic feature after the conversion is completed; Object relationship weight w o The calculation method is: Pose space position encoding weight w ploc The calculation method is: w ploc =F·L(b h ,b p ) Among them, the F function is the feature dimension conversion function, which is in the form of a fully connected layer or a one-dimensional convolution; the L function is a spatial position encoding function in the form of a trigonometric function, which is used to encode the coordinates of the human body bounding box b. h and the pose bounding box coordinates b p Encoding; Object space position encoding weight w oloc The calculation method is: w oloc =F·L(b h ,b o ) Among them, b o is the coordinate of the object bounding box; Posture-enhanced relational features f hp The calculation method is: f hp =F(f p ·∑(w ploc w p )) Among them, the F function is a dimension conversion function in the form of a fully connected layer or a one-dimensional convolution; Calculate another set of posture-enhanced relationship features f ph , just change the weight w p Perform transposition; Calculate the object enhanced relation feature f ho With f oh Just change f hp With f ph The posture-related weight w in the calculation formula p and w ploc Replaced by the object-related weight w o and w oloc , and the basic posture feature f p Replaced with the object's basic feature f o ; The weighting rules for the four groups of relationship features are: f hp :f ho :f ph :f oh =k1:k2:k3:k4 Among them, k1, k2, k3, and k4 satisfy: 0 <k1,k2<k3,k4 The relational features output after weighting are fused with the basic features according to the following rules to output enhanced relational features: f hr =f h +f hp +f ho f pr =f p +f ph f or =f o +f oh That is, the final output is 3 enhanced features f hr 、f pr and f or .

2. The method for human motion recognition based on posture-enhanced relational features according to claim 1, characterized in that: The step S4 specifically includes: S41: The three enhanced features f hr 、f pr and f or Send to the classifier for classification; S42: Fusing the classification results corresponding to each enhanced feature to obtain the overall classification result, i.e., the scores of various actions; S43: Use the softmax function to convert the action scores into probabilities. The action with the highest probability is the final recognition result. At the same time, the cross entropy function is used to calculate the loss for backpropagation.

3. The method for human motion recognition based on posture-enhanced relational features according to claim 2, characterized in that: The fusion rules of the classification results are defined as follows: Let f hr 、f pr and f or The corresponding classification result is Cls h 、Cls p and Cls o , then the classification result of action a after fusion is: Cls final (a)=Cls h (a)+maxCls p (a)+maxCls o (a) Among them, max selects the maximum value of the classification scores of multiple posture areas or objects for action a.

Images