Method for recognizing behavior based on combined geometric features of skeleton joints

Abstract

This application relates to the field of video data processing technology and provides a behavior recognition method based on the geometric features of skeleton joint combinations, aiming to improve the accuracy and efficiency of behavior recognition based on the geometric features of skeleton joint combinations. The method mainly includes: dividing the skeleton into X joint combinations, where X is a positive integer greater than 1; describing the spatial position features of each joint combination; describing the relative geometric features of each joint combination; combining the spatial position features and relative geometric features of each joint combination to describe the geometric features of each joint combination; inputting the geometric features of each joint combination into a trained deep convolutional network for feature extraction and classification, and outputting the possible probabilities of several behaviors; comprehensively analyzing the possible probabilities of several behaviors according to a preset fusion function, and outputting the behavior with the highest possible probability to obtain the target behavior.

Description

Technical Field

[0001] This application belongs to the field of video data processing technology, and in particular relates to a behavior recognition method based on the geometric features of skeleton joint combinations. Background Technology

[0002] Existing skeleton sequence-based behavior recognition technologies primarily rely on the spatial features of all joints. Currently, all joints are arranged according to their joint numbers to form a whole, serving as a description of individual behavior features for behavior classification. The skeleton sequence obtained from video data contains joints from multiple parts of the head, hands, legs, and spine in each frame, with each joint numbered starting from 1. Current skeleton sequence-based behavior recognition methods sequentially sort all joints according to their joint numbers, convert the skeleton sequence data into color RGB images as feature descriptions of the behavior, and then use deep networks for feature extraction to achieve behavior classification and recognition.

[0003] However, in existing skeleton sequence-based behavior recognition technologies, the applicant found that methods that directly arrange all joints and describe behavioral features suffer from low recognition efficiency and insufficient accuracy. For example, in the action of "clapping," only the joints of the forelimbs change, while the spatial positions of other joints remain unchanged, contributing little to the feature description. Summary of the Invention

[0004] The purpose of this application is to provide a behavior recognition method based on the geometric features of skeleton joint combinations, which aims to improve the accuracy and efficiency of behavior recognition based on the geometric features of skeleton joint combinations.

[0005] This application provides a behavior recognition method based on the geometric features of skeleton joint combinations, including:

[0006] The skeleton is divided into X joint combinations, where X is a positive integer greater than 1;

[0007] Describe the spatial positional characteristics of each of the joint combinations;

[0008] Describe the relative geometric features of each of the joint combinations;

[0009] The spatial position features and relative geometric features of each joint combination are combined to describe the geometric features of each joint combination.

[0010] The geometric features of each joint combination are input into a trained deep convolutional network for feature extraction and classification, and the possible probabilities of several behaviors are output respectively.

[0011] The possible probabilities of the several behaviors are comprehensively analyzed based on a preset fusion function, and the behavior with the highest possible probability is output to obtain the target behavior.

[0012] Optionally, X is 3; the skeleton is a human skeleton; dividing the skeleton into X joint combinations includes:

[0013] According to preset rules, invalid joints in the human skeleton are removed to obtain the remaining joints;

[0014] The remaining joints are divided into three joint combinations P1, P2, and P3, wherein P1 includes the joints of the left and right arms among the remaining joints, P2 includes the joints of the left arm and right leg among the remaining joints, and P3 includes the joints of the right arm and left leg among the remaining joints.

[0015] Optionally, each of the joint combinations includes a set of joint numbers, a spatial connection relationship between the joints, and a spatial connection order of the joints;

[0016] P1, P2, and P3 are respectively described as follows:

[0017]

[0018]

[0019]

[0020] Wherein H represents the set of joint numbers of the left arm and the right arm;

[0021] LG represents the set of joint numbers for the left arm and the right leg;

[0022] RG represents the set of joint numbers for the right arm and the left leg;

[0023] The J m This indicates the joint with serial number m;

[0024] The J n This represents the joint with index n;

[0025] The J k This indicates the joint with the serial number k.

[0026] Optionally, the spatial positional features describing each of the joint combinations include:

[0027] The spatial positional characteristics of P1, P2, and P3 are described as follows:

[0028]

[0029]

[0030]

[0031] The This represents the three-dimensional coordinates of the joint with index i in P1 at time t, where T represents the duration;

[0032] The This represents the three-dimensional coordinates of the joint with index j in P2 at time t;

[0033] The This represents the three-dimensional coordinates of the joint with index k in P3 at time t.

[0034] Optionally, after describing the spatial location features of P1, P2, and P3 respectively, the method further includes:

[0035] Regarding the spatial location features of P1, the Normalize the first X value to a value between 0 and 255;

[0036] The first X value is represented as a 224×112 image using linear interpolation.

[0037] Regarding the spatial location features of P1, the Normalize the first Y value to a value between 0 and 255;

[0038] The first Y value is represented as a 224×112 image using linear interpolation.

[0039] Regarding the spatial location features of P1, the Normalize to a first Z value between 0 and 255;

[0040] The first Z value is represented as a 224×112 image using linear interpolation.

[0041] The first RGB image is obtained by treating the first X value as the Red, Green, and Blue channels of the first RGB image, the first Y value as the first 224×112 image, and the first Z value as the Red, Green, and Blue channels of the first RGB image, respectively.

[0042] The spatial location features of P1 are described using the first RGB image;

[0043] Regarding the spatial location features of P2, the Normalize to a second X value between 0 and 255;

[0044] The second X value is represented as a 224×112 image using linear interpolation.

[0045] Regarding the spatial location features of P2, the Normalize the result to a second Y value between 0 and 255;

[0046] The second Y value is represented as a 224×112 image using linear interpolation.

[0047] Regarding the spatial location features of P2, the Normalize to a second Z value between 0 and 255;

[0048] The second Z value is represented as a 224×112 image using linear interpolation.

[0049] The second RGB image is obtained by treating the 224×112 image represented by the second X value, the 224×112 image represented by the second Y value, and the 224×112 image represented by the second Z value as the Red, Green, and Blue channels of the second RGB image, respectively.

[0050] The second RGB image is used to describe the spatial location features of P2;

[0051] Regarding the spatial location features of P3, the Normalize the result to a third X value between 0 and 255;

[0052] The third X value is represented as a 224×112 image using linear interpolation.

[0053] Regarding the spatial location features of P3, the Normalize the result to a third Y value between 0 and 255;

[0054] The third Y value is represented as a 224×112 image using linear interpolation.

[0055] Regarding the spatial location features of P3, the Normalize to a third Z value between 0 and 255;

[0056] The third Z value is represented as a 224×112 image using linear interpolation.

[0057] The 224×112 image represented by the third X value, the 224×112 image represented by the third Y value, and the 224×112 image represented by the third Z value are respectively regarded as the Red, Green, and Blue channels of the third RGB image to obtain the third RGB image.

[0058] The spatial location features of P3 are described using the third RGB image.

[0059] Optionally, the description of the relative geometric features of each of the joint combinations includes:

[0060] The relative geometric features of P1 are described using the relative positional features of P1:

[0061]

[0062] The above J represents the arm joint i In the three-dimensional coordinates at time t, the arm joint includes the left arm joint and the right arm joint;

[0063] The Indicates the arm joint J i The three-dimensional coordinates at time t-1;

[0064] The Indicates the arm joint J i The relative positional characteristics at time t;

[0065] The Considered as the relative geometric feature of P1;

[0066] Calculate the relative distance d21 between the left arm joints in P2, the relative distance d22 between the left arm joint and the right leg joint, and the relative distance d23 between the left arm joint and the right leg joint relative to the preset origin.

[0067] The relative distances d21, d22, and d23 are considered as the relative geometric features of P2;

[0068] Calculate the relative distance d31 between the right arm joints in P3, the relative distance d32 between the right arm joint and the left leg joint, and the relative distance d33 between the right arm joint and the left leg joint relative to the preset origin.

[0069] The relative distances d31, d32, and d33 are considered as the relative geometric features of P3.

[0070] Optionally, after describing the relative geometric features of P1, P2, and P3 respectively, the method further includes:

[0071] The relative geometric features of P1 Normalize to a first relative value of X between 0 and 255;

[0072] The first relative value of X is represented as a 224×112 image using linear interpolation.

[0073] Regarding the spatial location features of P1, the Normalize to a first relative value of Y between 0 and 255;

[0074] The first relative value of Y is represented as a 224×112 image using linear interpolation.

[0075] Regarding the spatial location features of P1, the Normalize to the first Z-relative value between 0 and 255;

[0076] The first Z relative value is represented as a 224×112 image using linear interpolation.

[0077] The first 224×112 image represented by the first X relative value, the first 224×112 image represented by the first Y relative value, and the first 224×112 image represented by the first Z relative value are respectively regarded as the Red, Green, and Blue channels of the fourth RGB image to obtain the fourth RGB image.

[0078] The fourth RGB image is used to describe the relative geometric features of P1;

[0079] The relative distance d21 in the relative geometric features of P2 is normalized to a first d21 value between 0 and 255;

[0080] The first d21 value is represented as a 224×112 image using linear interpolation.

[0081] The relative distance d22 in the relative geometric features of P2 is normalized to a first d22 value between 0 and 255;

[0082] The first d22 value is represented as a 224×112 image using linear interpolation.

[0083] The relative distance d23 in the relative geometric features of P2 is normalized to a first d23 value between 0 and 255;

[0084] The first d23 value is represented as a 224×112 image using linear interpolation.

[0085] The first d21 value is used to represent a 224×112 image, the first d22 value is used to represent a 224×112 image, and the first d23 value is used to represent a 224×112 image, which are respectively regarded as the Red, Green, and Blue channels of the fifth RGB image, to obtain the fifth RGB image.

[0086] The fifth RGB image is used to describe the relative geometric features of P2;

[0087] The relative distance d31 in the relative geometric features of P3 is normalized to a first d31 value between 0 and 255;

[0088] The first d31 value is represented as a 224×112 image using linear interpolation.

[0089] The relative distance d32 in the relative geometric features of P3 is normalized to a first d32 value between 0 and 255;

[0090] The first d32 value is represented as a 224×112 image using linear interpolation.

[0091] The relative distance d33 in the relative geometric features of P3 is normalized to a first d33 value between 0 and 255;

[0092] The first d33 value is represented as a 224×112 image using linear interpolation.

[0093] The first d31 value is used to represent a 224×112 image, the first d32 value is used to represent a 224×112 image, and the first d33 value is used to represent a 224×112 image, which are respectively regarded as the Red, Green, and Blue channels of the sixth RGB image, to obtain the sixth RGB image.

[0094] The sixth RGB image is used to describe the relative geometric features of P3.

[0095] Optionally, describing the spatial position features of each joint combination in conjunction with the relative geometric features as geometric features includes:

[0096] By combining the first RGB image and the fourth RGB image, a first geometric feature RGB image with a length vector of 224×224 corresponding to P1 is formed;

[0097] By combining the second RGB image and the fifth RGB image, a second geometric feature RGB image with a length vector of 224×224 corresponding to P2 is formed;

[0098] By combining the third RGB image and the sixth RGB image, a third geometric feature RGB image corresponding to the 224×224 length vector of P3 is formed.

[0099] Optionally, the geometric features of each joint combination are input into a trained deep convolutional network for feature extraction and classification, and the network outputs the possible probabilities of several behaviors, including:

[0100] The first geometric feature RGB image, the second geometric feature RGB image, and the third geometric feature RGB image are respectively input into a trained convolutional neural network ResNet-50 for feature extraction, and the possible probabilities of several behaviors are output respectively.

[0101] Optionally, the possible probabilities of the several behaviors include: the posterior probability b1 of the behavior extracted from the first geometric feature RGB image, the posterior probability b2 of the behavior extracted from the second geometric feature RGB image, and the posterior probability b3 of the behavior extracted from the third geometric feature RGB image.

[0102] The step of comprehensively analyzing the possible probabilities of the several behaviors based on a preset fusion function, and outputting the behavior with the highest possible probability to obtain the target behavior, includes:

[0103] P(L|S)=η1b1+η2b2+η3b3

[0104] η1, η2, and η3 are preset weights;

[0105] The posterior probability vector P(L|S) represents the probability that the skeleton belongs to action label category L.

[0106] label = Find(max(P(L|S)))

[0107] The label represents the target behavior.

[0108] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0109] As can be seen, the behavior recognition method based on the geometric features of skeleton joint combinations in this application divides the skeleton into X joint combinations, unlike existing technologies that treat all joints of the skeleton as a whole for recognition. Dividing it into X joint combinations improves the classification accuracy of behaviors and determines the geometric feature basis for the recognition and classification process of each joint combination, reducing the amount of data to be analyzed. A trained deep convolutional network is used for feature extraction and classification to obtain the possible probabilities of several behaviors. Then, the possible probabilities of several behaviors are comprehensively analyzed according to a preset fusion function, and the behavior with the highest possible probability is output to obtain the target behavior. Since the amount of data for a joint combination is smaller than that for all joints of the entire skeleton, less computational resources are required, thereby improving the efficiency of behavior recognition based on the geometric features of skeleton joint combinations. Furthermore, by describing the spatial position features and relative geometric features of each joint combination, the accuracy of behavior recognition based on the geometric features of skeleton joint combinations is improved. Attached Figure Description

[0110] Figure 1 This is a schematic flowchart of an embodiment of the behavior recognition method based on the geometric features of skeleton joint combinations in this application;

[0111] Figure 2 This is a schematic diagram of an embodiment of the present application that divides the joints of the human skeleton into different joint combinations;

[0112] Figure 3 This is a schematic diagram of an embodiment of the relative geometric features in the joint assembly of this application;

[0113] Figure 4 This is a schematic diagram of an embodiment of the deep convolutional network of this application for feature extraction and classification of three joint combinations P1, P2, and P3;

[0114] Figure 5 This is a schematic diagram of an embodiment of the behavior recognition device based on the geometric features of human skeletal joints in this application;

[0115] Figure 6 This is a schematic diagram of the structure of one embodiment of the computer device of this application. Detailed Implementation

[0116] In order to clearly identify the various behaviors expressed by the various joint states of the skeleton, the applicant analyzed the existing behavior recognition methods based on the human skeleton and found that the current methods of directly arranging all joints of the skeleton and describing their behavioral characteristics based on joint numbers have at least the following problems and defects:

[0117] 1. Joints with adjacent serial numbers do not have spatial correlation in the human skeleton. For example, the joint representing the head and the joint representing the hand have adjacent serial numbers, but these two joints are not adjacent in spatial position, which is not conducive to the identification of behavioral characteristics. Therefore, the applicant believes that joints with spatial correlation should be grouped according to their spatial correlation in the skeleton.

[0118] 2. Existing techniques describe all joints by feature. This approach fails to focus on the local joints where behavioral changes actually occur. For example, in the action of "clapping," only the joints in the hands change, while the spatial positions of other joints in the legs and head remain unchanged. Therefore, it does not contribute much to feature description. The occurrence of behavioral features is mainly represented by certain joints of the skeleton. Therefore, the features representing behavior should be concentrated on some key joints of the skeleton.

[0119] 3. Existing technical methods do not consider the relative spatial relationship between symmetrical joints. Most human movements are performed by hand or leg joints, such as "running," "clapping," "kicking," and "making gestures." The relative spatial relationship between the left hand or leg joint and the right hand or leg joint is crucial for the differentiation of behavioral characteristics and should be considered in the description of behavioral characteristics.

[0120] Based on the above understanding, the following embodiments will only use human behavior recognition as an example for illustration. It is worth noting that the behavior recognition method based on the geometric features of skeletal joint combinations in this application is not only applicable to human behavior recognition, but also, under specific conditions, can be used to recognize the behavior of other objects, especially animals, and output the corresponding recognized target behavior.

[0121] Please see Figure 1 An embodiment of the behavior recognition method based on the geometric features of skeleton joint combinations in this application includes:

[0122] 101. Divide the skeleton into X joint combinations, where X is a positive integer greater than 1.

[0123] First, this application requires knowledge of the state of all joints in the skeleton. These joints can be simply considered as articulation points. This step requires obtaining the spatial coordinates of all articulation points in the skeleton, the connection relationships of all articulation points, and the spatial connection order of all articulation points. Based on this, this step divides these articulation points of the skeleton into X joint combinations, where X is a positive integer greater than 1.

[0124] Specifically, the skeleton in question is the human skeleton. This step first removes invalid joints from the human skeleton according to preset rules, obtaining the remaining joints. Invalid joints here refer to joints irrelevant to the recognition of a specific behavior. For example, to recognize behaviors such as "running," "clapping," "kicking," and "making gestures," these behaviors are mainly formed by the movement of limb joints. In this case, spinal joints, lumbar joints, etc., can be considered invalid joints. It should be noted that the specific joint objects of invalid joints may differ depending on the behavior being recognized. Therefore, the specific joint objects of invalid joints are not limited here. Since the specific joint objects of invalid joints differ, the preset rules for removing invalid joints also differ, and these preset rules are not limited here either. This step of removing invalid joints reduces the computational load on joints, increases computational speed, and improves the recognition efficiency of the behavior recognition method based on the geometric features of skeleton joint combinations in this application.

[0125] In one embodiment, please refer to Figure 2 ,in Figure 2 The left-middle diagram shows a simplified schematic of the human skeletal joints. This skeletal structure mainly includes: left shoulder joint 1, right shoulder joint 2, spinal joint 3, lumbar joint 4, left hip joint 5, right hip joint 6, coccyx joint 7, left elbow joint 8, right elbow joint 9, left wrist joint 10, right wrist joint 11, left finger joint 12, right finger joint 13, left knee joint 14, right knee joint 15, left ankle joint 16, right ankle joint 17, left toe joint 18, right toe joint 19, and head joint 20. To identify behaviors such as "running," "clapping," "kicking," and "gesturing," this scheme considers spinal joint 3, lumbar joint 4, coccyx joint 7, and head joint 20 as invalid joints. These joints are subtracted to obtain the remaining joints, as detailed in the diagram. Figure 2 As shown in the middle right figure. Figure 2 The middle right figure divides the human skeleton into three joint combinations: P1, P2, and P3. Joint combination P1 includes the joints of the remaining left arm (left finger joint 12, left wrist joint 10, left elbow joint 8, left shoulder joint 1) and right arm (right finger joint 13, right wrist joint 11, right elbow joint 9, right shoulder joint 2); joint combination P2 includes the joints of the remaining left arm (left finger joint 12, left wrist joint 10, left elbow joint 8, left shoulder joint 1) and right leg (right hip joint 6, right knee joint 15, right ankle joint 17, right toe joint 19); joint combination P3 includes the joints of the remaining right arm (right finger joint 13, right wrist joint 11, right elbow joint 9, right shoulder joint 2) and left leg (left hip joint 5, left knee joint 14, left ankle joint 16, left toe joint 18).

[0126] Furthermore, the above three joint combinations can also include a set of joint numbers, spatial connection relationships between joints, and spatial connection order of joints, with P1, P2, and P3 described as follows:

[0127]

[0128]

[0129]

[0130] Where H represents the set of joint numbers for the left and right arms; LG represents the set of joint numbers for the left and right legs; RG represents the set of joint numbers for the right and left legs; J m Indicates the joint with serial number m; J n J represents the joint with index n; k This represents the joint with the sequence number k. The above formula can more accurately express the combination state of each joint in the above three joint combinations.

[0131] 102. Describe the spatial positional characteristics of each joint combination.

[0132] The spatial position features of the joint combinations divided in step 101 are described. These features are described by the three-dimensional spatial coordinates of each joint in the combination, for example, by the three-dimensional spatial coordinates of all joints at different times. Therefore, the set of three-dimensional coordinates of each joint in the combination can be described as the spatial position feature of the joint combination. Based on this, the spatial position features of all joints in the three joint combinations P1, P2, and P3 are described as follows:

[0133]

[0134]

[0135]

[0136] in Let T represent the three-dimensional coordinates of joint number i in joint combination P1 at time t, where T represents the duration. This represents the three-dimensional coordinates of the joint with index j in P2 at time t; This represents the three-dimensional coordinates of the joint with index k in P3 at time t.

[0137] Furthermore, the spatial position features of joint combination P1, joint combination P2, and joint combination P3 can be converted into RGB images for description. RGB images are more suitable for recognition and classification by neural network models.

[0138] For joint assembly P1 The three coordinates are normalized to correspond to the Red, Green, and Blue channels of the first RGB image. This includes the spatial position features of joint combination P1. The first X value is normalized to between 0 and 255. Taking the normalized x-axis coordinate of joint assembly P1 as an example:

[0139]

[0140] in This represents the normalized x-axis coordinate, i.e., the first X value; then, using linear interpolation, the first X value is represented as a 224×112 image. Similarly, for the spatial position features of joint assembly P1... Normalize to the first Y value between 0 and 255, specifically for... The normalization process can also refer to the above. The formula is normalized to obtain the corresponding y-axis coordinate, i.e., the first Y value. Then, the first Y value is represented as a 224×112 image using linear interpolation. Similarly, the spatial position features of joint assembly P1 are normalized. Normalize to the first Z value between 0 and 255, specifically for... The normalization process can also refer to the above. The formula is normalized to obtain the corresponding z-axis coordinate, i.e., the first Z value. Then, the first Z value is represented as a 224×112 image by linear interpolation. Finally, the 224×112 images represented by the first X value, the first Y value, and the first Z value are regarded as the Red, Green, and Blue channels of the first RGB image, respectively, to obtain a first RGB image. This realizes the use of the first RGB image to describe the spatial position features of the joint assembly P1.

[0141] For joint assembly P2 The three coordinates are normalized to correspond to the Red, Green, and Blue channels of the second RGB image. This includes the spatial position features of joint assembly P2. Normalize to a second X value between 0 and 255, specifically for... The normalization process can also refer to the above. The formula is normalized to obtain the corresponding x-axis coordinate, i.e., the second X value. Then, the second X value is represented as a 224×112 image using linear interpolation. Similarly, the spatial position features of joint assembly P2 are... Normalize the second Y value to a value between 0 and 255, specifically for... The normalization process can also refer to the above. The formula is normalized to obtain the corresponding y-axis coordinate, i.e., the second Y value. Then, the second Y value is represented as a 224×112 image using linear interpolation. Similarly, the spatial position features of joint assembly P2 are normalized. Normalize to a second Z value between 0 and 255, specifically for... The normalization process can also refer to the above. The formula is normalized to obtain the corresponding z-axis coordinate, i.e., the second Z value. Then, the second Z value is represented as a 224×112 image by linear interpolation. Finally, the 224×112 images represented by the second X value, the second Y value, and the second Z value are regarded as the Red, Green, and Blue channels of the second RGB image, respectively, to obtain a second RGB image. This realizes the use of the second RGB image to describe the spatial position features of the joint assembly P2.

[0142] For joint assembly P3 The three coordinates are normalized to correspond to the Red, Green, and Blue channels of the third RGB image. This includes the spatial position features of joint assembly P3. Normalize to a third X value between 0 and 255, specifically for... The normalization process can also refer to the above. The formula is normalized to obtain the corresponding x-axis coordinate, i.e., the third X value. Then, the third X value is represented as a 224×112 image using linear interpolation. Similarly, the spatial position features of joint assembly P3 are... Normalize the third Y value to a value between 0 and 255, specifically for... The normalization process can also refer to the above. The formula is normalized to obtain the corresponding y-axis coordinate, i.e., the third Y value. Then, the third Y value is represented as a 224×112 image using linear interpolation. Similarly, the spatial position features of joint assembly P3 are normalized. Normalize to a third Z value between 0 and 255, specifically for... The normalization process can also refer to the above. The formula is normalized to obtain the corresponding z-axis coordinate, i.e., the third Z value. Then, the third Z value is represented as a 224×112 image by linear interpolation. Finally, the 224×112 images represented by the third X value, the third Y value, and the third Z value are regarded as the Red, Green, and Blue channels of the third RGB image, respectively, to obtain a third RGB image. This realizes the use of the third RGB image to describe the spatial position features of the joint assembly P3.

[0143] 103. Describe the relative geometric features of each joint combination.

[0144] The relative geometric features of the joint combinations segmented in step 101 are described. Considering the spatial position features of the joints, the relative geometric features between different joint combinations are also crucial for behavior recognition. The relative geometric features of different joints are significant for representing behaviors. For example, in the "clapping" behavior, the relative distance between the symmetrical joints of the left and right arms becomes very small; similarly, in the "picking up" behavior, the relative distance between the arm joints (including the left and right arms) and the leg joints also becomes very small. These relative distances are important for distinguishing different behaviors. Furthermore, the relative position features of joints at adjacent time points are also important for behavior representation, especially for actions with inconsistent amplitudes, such as "walking" and "running," where the amplitude of change in the relative position of joints differs between adjacent time points. Please refer to [link to relevant documentation]. Figure 3 This application describes two relative geometric features. The first is the relative distance feature between different joint combinations, such as... Figure 3 As shown in the left figure; the second type is the relative positional feature of the same joint combination, such as... Figure 3 As shown in the image on the right.

[0145] For joint assembly P1, this embodiment uses the relative positional features of joint assembly P1 to describe the relative geometric features of joint assembly P1:

[0146]

[0147] in J represents the arm joint i The three-dimensional coordinates at time t, where the arm joints include the left arm joint and the right arm joint; J represents the arm joint i The three-dimensional coordinates at time t-1; J represents the arm joint i The relative position characteristics at time t; Considered as the relative geometric features of joint assembly P1;

[0148] For joint combinations P2 and P3, refer to the following calculation formulas:

[0149]

[0150]

[0151]

[0152] Where d1 represents the relative distance between different arm joints, d2 represents the relative distance between the arm and leg joints, and d3 represents the relative distance between the joint and the origin.

[0153] For joint assembly P2, the relative distances between the joints in joint assembly P2 are considered as the relative geometric features of joint assembly P2. The relative distances d21 between the left arm joints, d22 between the left arm joint and the right leg joint, and d23 between the left arm joint and the right leg joint relative to a preset origin are calculated. Specifically, the relative distance d21 between the left arm joints in joint assembly P2 is calculated using the formula for d1 above; the relative distance d22 between the left arm joint and the right leg joint is calculated using the formula for d2 above; and the relative distance d23 between the left arm joint and the right leg joint relative to the preset origin is calculated using the formula for d3 above. Relative distances d21, d22, and d23 are considered as the relative geometric features of joint assembly P2.

[0154] For joint assembly P3, the relative distances between the joints in joint assembly P3 are considered as the relative geometric features of joint assembly P3. The relative distances d31 between the right arm joints, d32 between the right arm joint and the left leg joint, and d33 between the right arm joint and the left leg joint relative to a preset origin are calculated. Specifically, the relative distance d31 between the right arm joints in joint assembly P3 is calculated using the formula for d1 above; the relative distance d32 between the right arm joint and the left leg joint is calculated using the formula for d2 above; and the relative distance d33 between the right arm joint and the left leg joint relative to the preset origin is calculated using the formula for d3 above. Relative distances d31, d32, and d33 are considered as the relative geometric features of joint assembly P3.

[0155] Furthermore, the relative geometric features of joint combination P1, joint combination P2, and joint combination P3 can be converted into RGB images for description. RGB images are more suitable for recognition and classification by neural network models.

[0156] For the relative geometric features of joint assembly P1 The three coordinates are normalized to correspond to the Red, Green, and Blue channels of the fourth RGB image. This includes the relative geometric features of joint combination P1. Normalize to the first relative value of X between 0 and 255, specifically for... The normalization process can also refer to the above. The formula is normalized to obtain the corresponding x-axis coordinate, i.e., the first relative X value. Then, the first relative X value is represented as a 224×112 image using linear interpolation. Similarly, the relative geometric features of joint assembly P1 are normalized. Normalize to the first relative value of Y between 0 and 255, specifically for... The normalization process can also refer to the above. The formula is normalized to obtain the corresponding y-axis coordinate, i.e., the second Y value. Then, the second Y value is represented as a 224×112 image using linear interpolation. Similarly, the relative geometric features of joint assembly P1 are normalized. Normalize to the first Z-relative value between 0 and 255, specifically for The normalization process can also refer to the above. The formula is normalized to obtain the corresponding z-axis coordinate, i.e., the first Z relative value. Then, the first Z relative value is represented as a 224×112 image by linear interpolation. Finally, the 224×112 images represented by the first X relative value, the first Y relative value, and the first Z relative value are regarded as the Red, Green, and Blue channels of the fourth RGB image, respectively, to obtain a fourth RGB image. This realizes the use of the fourth RGB image to describe the relative geometric features of the joint combination P1.

[0157] For the relative geometric features of joint assembly P2, the relative distance d21 in the relative geometric features of joint assembly P2 is normalized to a first d21 value between 0 and 255. The specific normalization process for the relative distance d21 can be referred to the above. The formula is normalized to obtain the corresponding first d21 value, and then the first d21 value is represented as a 224×112 image using linear interpolation. The relative distance d22 in the relative geometric features of joint assembly P2 is normalized to a first d22 value between 0 and 255. The specific normalization process of the relative distance d22 can be referred to the above. The formula is normalized to obtain the corresponding first d22 value, and then the first d22 value is represented as a 224×112 image using linear interpolation. The relative distance d23 in the relative geometric features of P2 is normalized to a first d23 value between 0 and 255. The specific normalization process of the relative distance d23 can be referred to the above. The formula is normalized to obtain the corresponding first d23 value. Then, the first d23 value is represented as a 224×112 image by linear interpolation. The 224×112 images represented by the first d21 value, the first d22 value, and the first d23 value are respectively regarded as the Red, Green, and Blue channels of the fifth RGB image to obtain a fifth RGB image. This realizes the use of the fifth RGB image to describe the relative geometric features of the joint combination P2.

[0158] For the relative geometric features of joint assembly P3, the relative distance d31 in the relative geometric features of joint assembly P3 is normalized to a first d31 value between 0 and 255. The specific normalization process for the relative distance d31 can be referred to the above. The formula is normalized to obtain the corresponding first d31 value, and then the first d31 value is represented as a 224×112 image using linear interpolation. The relative distance d32 in the relative geometric features of joint assembly P3 is normalized to a first d32 value between 0 and 255. The specific normalization process of the relative distance d32 can be referred to the above. The formula is normalized to obtain the corresponding first d32 value, and then the first d32 value is represented as a 224×112 image using linear interpolation. The relative distance d33 in the relative geometric features of joint assembly P3 is normalized to a first d33 value between 0 and 255. The specific normalization process of the relative distance d33 can be referred to the above. The formula is normalized to obtain the corresponding first d33 value. Then, the first d33 value is represented as a 224×112 image by linear interpolation. The 224×112 images represented by the first d31 value, the first d32 value, and the first d33 value are respectively regarded as the Red, Green, and Blue channels of the sixth RGB image to obtain a sixth RGB image. This realizes the use of the sixth RGB image to describe the relative geometric features of the joint assembly P3.

[0159] In another embodiment, for joint assembly P1, the relative distances between the joints of joint assembly P1 can also be used as the relative geometric features of joint assembly P1. The relative distance d11 formed between the arm joints, the relative distance d12 between the arm joints and the leg joints (left and right leg joints), and the relative distance d13 between the arm joints and a preset origin are calculated. Specifically, the relative distance d11 formed between the arm joints in joint assembly P1 is calculated using the formula for d1 above; the relative distance d12 between the arm joints and the leg joints is calculated using the formula for d2 above; the relative distance d13 between the arm joints and the preset origin is calculated using the formula for d3 above; and the relative distances d11, d12, and d13 are considered as the relative geometric features of joint assembly P1.

[0160] Furthermore, for the relative geometric features of joint assembly P1, the relative distance d11 in the relative geometric features of joint assembly P1 is normalized to a first d31 value between 0 and 255. The specific normalization process for the relative distance d11 can be referred to the above. The formula is normalized to obtain the corresponding first d11 value, and then the first d11 value is represented as a 224×112 image using linear interpolation. The relative distance d12 in the relative geometric features of joint assembly P1 is normalized to a first d32 value between 0 and 255. The specific normalization process of the relative distance d12 can be referred to the above. The formula is normalized to obtain the corresponding first d12 value, and then the first d12 value is represented as a 224×112 image using linear interpolation. The relative distance d13 in the relative geometric features of joint assembly P1 is normalized to a first d13 value between 0 and 255. The specific normalization process of the relative distance d13 can be referred to the above. The formula is normalized to obtain the corresponding first d13 value. Then, the first d13 value is represented as a 224×112 image by linear interpolation. The 224×112 images represented by the first d11 value, the first d12 value, and the first d13 value are respectively regarded as the Red, Green, and Blue channels of the seventh RGB image to obtain a seventh RGB image. This realizes the use of the seventh RGB image to describe the relative geometric features of the joint combination P1.

[0161] 104. Combine the spatial position features and relative geometric features of each joint combination to describe the geometric features of each joint combination.

[0162] The spatial position features of each joint combination described in step 102 and the relative geometric features of each joint combination described in step 103 are combined to describe the geometric features of each joint combination.

[0163] For example, please see Figure 4 Combining the first and fourth RGB images, a first geometric feature RGB image with a length of 224×224 corresponding to the joint combination P1 is formed; combining the second and fifth RGB images, a second geometric feature RGB image with a length of 224×224 corresponding to the joint combination P2 is formed; combining the third and sixth RGB images, a third geometric feature RGB image with a length of 224×224 corresponding to the joint combination P3 is formed. Alternatively, the first and seventh RGB images can be combined to form another first geometric feature RGB image with a length of 224×224 corresponding to the joint combination P1.

[0164] 105. Input the geometric features of each joint combination into the trained deep convolutional network for feature extraction and classification, and output the possible probabilities of several behaviors respectively.

[0165] As is understandable, this embodiment uses a trained ResNet-50 convolutional neural network for feature extraction. The training sample images for this ResNet-50 network also require processing as described in steps 101, 102, 103, and 104 above. This ensures that the trained ResNet-50 network can recognize the spatial location features, relative geometric features, and geometric features of each joint combination. Combined with manual labeling of the behaviors expressed in the training sample images, this improves the efficiency and accuracy of ResNet-50 training, resulting in a trained ResNet-50 network capable of recognizing and classifying specific labeled behaviors. The training process for the ResNet-50 network is a relatively mature existing technology and will not be elaborated upon here.

[0166] Please see Figure 4 In this step, the first geometric feature RGB image, the second geometric feature RGB image, and the third geometric feature RGB image from step 104 are input into the trained convolutional neural network ResNet-50 for feature extraction, and the possible probabilities of several behaviors are output respectively.

[0167] 106. Based on the preset fusion function, perform a comprehensive analysis of the possible probabilities of several behaviors, output the behavior with the highest possible probability, and obtain the target behavior.

[0168] For example, the possible probabilities of several behaviors obtained in step 105 include: the posterior probability b1 of the behavior extracted from the first geometric feature RGB image, the posterior probability b2 of the behavior extracted from the second geometric feature RGB image, and the posterior probability b3 of the behavior extracted from the third geometric feature RGB image; then according to the following formula:

[0169] P(L|S)=η1b1+η2b2+η3b3

[0170] Where η1, η a Let η3 be the preset weights, and the posterior probability vector P(L|S) represent the probability that the skeleton belongs to the action label category L. Then, label = Find(max(P(L|S))), where label represents the target behavior. For example, the target behavior may be a label category such as "clap", "pick up", "walk", or "run".

[0171] Following the above description of the embodiments of this application, this embodiment also lists the recognition results of two other different skeleton-based behavior databases, NTU RGB+D 60 and Northwestern-UCLA. NTU RGB+D 60 contains 56,880 behavior samples, with 40 objects participating in the capture of 60 types of actions. Each object repeats the action facing three cameras at different angles, thus requiring two verification methods: cross-view (CV) and cross-subject (CS) verification. The Northwestern-UCLA (N-UCLA) dataset contains 1,494 actions across 10 action types, each action type being repeated 6 times by 10 different objects. This dataset contains three cameras forming multiple viewpoints, and cross-validation is performed based on cross-view (CV). The behavior recognition technology based on skeleton joint combination geometric features of this application embodiment is compared with the current behavior recognition technology based on the connection of all skeleton joint numbers on the two datasets, NTU RGB+D 60 and N-UCLA. The recognition results are shown in Table 1 below.

[0172]

[0173] Table 1

[0174] Table 1 above compares the accuracy of the behavior recognition method based on the geometric features of skeleton joints proposed in this application with that of the prior art. The accuracy results show that the behavior recognition method based on the geometric features of skeleton joints proposed in this application has a significant improvement in accuracy compared to the prior art, demonstrating the superiority of the method.

[0175] The behavior recognition method based on the geometric features of skeleton joints proposed in this application overcomes the following shortcomings and achieves its advantages compared with existing behavior recognition technologies based on all joint spatial features:

[0176] 1. Existing technologies describe behavioral features based on the spatial features of all joints, neglecting the spatial connection features between joints. This application connects joints sequentially based on their spatial connection characteristics, rather than according to the joint number, thus reflecting the spatial characteristics between different joints.

[0177] 2. Existing technologies describe behavioral features based on the spatial features of all joints, without highlighting key information about the behavior. The joint combination technique proposed in this application highlights the key parts and joints where the behavior occurs, removes joints that do not contribute much to the description of behavioral features, focuses the description of behavioral features on the arm and leg joints, and divides these joints into three joint combinations, comprehensively considering the impact of different joint combinations on behavioral differences.

[0178] 3. Existing technologies lack descriptions of the relative features of different joints, including relative spatial and relative temporal features, which are crucial for behavior representation. This application proposes relative distance features between different joints and relative temporal features of arm joints, which, together with joint spatial features, constitute joint combination geometric features. This expresses both the spatial and temporal characteristics of the human skeleton, significantly improving the accuracy of behavior recognition.

[0179] The above embodiments describe the behavior recognition method based on the geometric features of skeleton joint combinations according to this application. The following describes the behavior recognition device based on the geometric features of skeleton joint combinations according to this application. Please refer to [link to relevant documentation]. Figure 5 ,include:

[0180] The partitioning unit 501 is used to divide the skeleton into X joint combinations, where X is a positive integer greater than 1;

[0181] The first description unit 502 is used to describe the spatial position characteristics of each of the joint combinations;

[0182] The second description unit 503 is used to describe the relative geometric features of each of the joint combinations;

[0183] The description unit 504 is used to combine the spatial position features and the relative geometric features of each joint combination to describe the geometric features of each joint combination.

[0184] The classification output unit 505 is used to input the geometric features of each joint combination into a trained deep convolutional network for feature extraction and classification, and output the possible probabilities of several behaviors respectively.

[0185] The fusion analysis unit 506 is used to perform a comprehensive analysis of the possible probabilities of the several behaviors according to a preset fusion function, output the behavior with the highest possible probability, and obtain the target behavior.

[0186] Optionally, X is 3; the skeleton is a human skeleton; when dividing the skeleton into X joint combinations, the dividing unit 501 is specifically used for:

[0187] According to preset rules, invalid joints in the human skeleton are removed to obtain the remaining joints;

[0188] The remaining joints are divided into three joint combinations P1, P2, and P3, wherein P1 includes the joints of the left and right arms among the remaining joints, P2 includes the joints of the left arm and right leg among the remaining joints, and P3 includes the joints of the right arm and left leg among the remaining joints.

[0189] Optionally, each of the joint combinations includes a set of joint numbers, a spatial connection relationship between the joints, and a spatial connection order of the joints;

[0190] The partitioning unit 501 describes P1, P2, and P3 as follows:

[0191]

[0192]

[0193]

[0194] Wherein H represents the set of joint numbers of the left arm and the right arm;

[0195] LG represents the set of joint numbers for the left arm and the right leg;

[0196] RG represents the set of joint numbers for the right arm and the left leg;

[0197] The J m This indicates the joint with serial number m;

[0198] The J n This represents the joint with index n;

[0199] The J k This indicates the joint with the serial number k.

[0200] Optionally, when the first description unit 502 describes the spatial positional features of each joint assembly, it is specifically used for:

[0201] The spatial positional characteristics of P1, P2, and P3 are described as follows:

[0202]

[0203]

[0204]

[0205] The This represents the three-dimensional coordinates of the joint with index i in P1 at time t, where T represents the duration;

[0206] The This represents the three-dimensional coordinates of the joint with index j in P2 at time t;

[0207] The This represents the three-dimensional coordinates of the joint with index k in P3 at time t.

[0208] Optionally, the first description unit 502 is further configured to:

[0209] Regarding the spatial location features of P1, the Normalize the first X value to a value between 0 and 255;

[0210] The first X value is represented as a 224×112 image using linear interpolation.

[0211] Regarding the spatial location features of P1, the Normalize the first Y value to a value between 0 and 255;

[0212] The first Y value is represented as a 224×112 image using linear interpolation.

[0213] Regarding the spatial location features of P1, the Normalize to a first Z value between 0 and 255;

[0214] The first Z value is represented as a 224×112 image using linear interpolation.

[0215] The first RGB image is obtained by treating the first X value as the Red, Green, and Blue channels of the first RGB image, the first Y value as the first 224×112 image, and the first Z value as the Red, Green, and Blue channels of the first RGB image, respectively.

[0216] The spatial location features of P1 are described using the first RGB image;

[0217] Regarding the spatial location features of P2, the Normalize to a second X value between 0 and 255;

[0218] The second X value is represented as a 224×112 image using linear interpolation.

[0219] Regarding the spatial location features of P2, the Normalize the result to a second Y value between 0 and 255;

[0220] The second Y value is represented as a 224×112 image using linear interpolation.

[0221] Regarding the spatial location features of P2, the Normalize to a second Z value between 0 and 255;

[0222] The second Z value is represented as a 224×112 image using linear interpolation.

[0223] The second RGB image is obtained by treating the 224×112 image represented by the second X value, the 224×112 image represented by the second Y value, and the 224×112 image represented by the second Z value as the Red, Green, and Blue channels of the second RGB image, respectively.

[0224] The second RGB image is used to describe the spatial location features of P2;

[0225] Regarding the spatial location features of P3, the Normalize the result to a third X value between 0 and 255;

[0226] The third X value is represented as a 224×112 image using linear interpolation.

[0227] Regarding the spatial location features of P3, the Normalize the result to a third Y value between 0 and 255;

[0228] The third Y value is represented as a 224×112 image using linear interpolation.

[0229] Regarding the spatial location features of P3, the Normalize to a third Z value between 0 and 255;

[0230] The third Z value is represented as a 224×112 image using linear interpolation.

[0231] The 224×112 image represented by the third X value, the 224×112 image represented by the third Y value, and the 224×112 image represented by the third Z value are respectively regarded as the Red, Green, and Blue channels of the third RGB image to obtain the third RGB image.

[0232] The spatial location features of P3 are described using the third RGB image.

[0233] Optionally, the second description unit 503, when describing the relative geometric features of each of the joint combinations, is specifically used for:

[0234] The relative geometric features of P1 are described using the relative positional features of P1:

[0235]

[0236] The above J represents the arm joint i In the three-dimensional coordinates at time t, the arm joint includes the left arm joint and the right arm joint;

[0237] The Indicates the arm joint J i The three-dimensional coordinates at time t-1;

[0238] The Indicates the arm joint J i The relative positional characteristics at time t;

[0239] The Considered as the relative geometric feature of P1;

[0240] Calculate the relative distance d21 between the left arm joints in P2, the relative distance d22 between the left arm joint and the right leg joint, and the relative distance d23 between the left arm joint and the right leg joint relative to the preset origin.

[0241] The relative distances d21, d22, and d23 are considered as the relative geometric features of P2;

[0242] Calculate the relative distance d31 between the right arm joints in P3, the relative distance d32 between the right arm joint and the left leg joint, and the relative distance d33 between the right arm joint and the left leg joint relative to the preset origin.

[0243] The relative distances d31, d32, and d33 are considered as the relative geometric features of P3.

[0244] Optionally, the second description unit 503 is further configured to:

[0245] The relative geometric features of P1 Normalize to a first relative value of X between 0 and 255;

[0246] The first relative value of X is represented as a 224×112 image using linear interpolation.

[0247] Regarding the spatial location features of P1, the Normalize to a first relative value of Y between 0 and 255;

[0248] The first relative value of Y is represented as a 224×112 image using linear interpolation.

[0249] Regarding the spatial location features of P1, the Normalize to the first Z-relative value between 0 and 255;

[0250] The first Z relative value is represented as a 224×112 image using linear interpolation.

[0251] The first 224×112 image represented by the first X relative value, the first 224×112 image represented by the first Y relative value, and the first 224×112 image represented by the first Z relative value are respectively regarded as the Red, Green, and Blue channels of the fourth RGB image to obtain the fourth RGB image.

[0252] The fourth RGB image is used to describe the relative geometric features of P1;

[0253] The relative distance d21 in the relative geometric features of P2 is normalized to a first d21 value between 0 and 255;

[0254] The first d21 value is represented as a 224×112 image using linear interpolation.

[0255] The relative distance d22 in the relative geometric features of P2 is normalized to a first d22 value between 0 and 255;

[0256] The first d22 value is represented as a 224×112 image using linear interpolation.

[0257] The relative distance d23 in the relative geometric features of P2 is normalized to a first d23 value between 0 and 255;

[0258] The first d23 value is represented as a 224×112 image using linear interpolation.

[0259] The first d21 value is used to represent a 224×112 image, the first d22 value is used to represent a 224×112 image, and the first d23 value is used to represent a 224×112 image, which are respectively regarded as the Red, Green, and Blue channels of the fifth RGB image, to obtain the fifth RGB image.

[0260] The fifth RGB image is used to describe the relative geometric features of P2;

[0261] The relative distance d31 in the relative geometric features of P3 is normalized to a first d31 value between 0 and 255;

[0262] The first d31 value is represented as a 224×112 image using linear interpolation.

[0263] The relative distance d32 in the relative geometric features of P3 is normalized to a first d32 value between 0 and 255;

[0264] The first d32 value is represented as a 224×112 image using linear interpolation.

[0265] The relative distance d33 in the relative geometric features of P3 is normalized to a first d33 value between 0 and 255;

[0266] The first d33 value is represented as a 224×112 image using linear interpolation.

[0267] The first d31 value is used to represent a 224×112 image, the first d32 value is used to represent a 224×112 image, and the first d33 value is used to represent a 224×112 image, which are respectively regarded as the Red, Green, and Blue channels of the sixth RGB image, to obtain the sixth RGB image.

[0268] The sixth RGB image is used to describe the relative geometric features of P3.

[0269] Optionally, when the combination description unit 504 combines the spatial position features of each joint combination with the relative geometric features to describe the geometric features, it is specifically used for:

[0270] By combining the first RGB image and the fourth RGB image, a first geometric feature RGB image with a length vector of 224×224 corresponding to P1 is formed;

[0271] By combining the second RGB image and the fifth RGB image, a second geometric feature RGB image with a length vector of 224×224 corresponding to P2 is formed;

[0272] By combining the third RGB image and the sixth RGB image, a third geometric feature RGB image corresponding to the 224×224 length vector of P3 is formed.

[0273] Optionally, when the classification output unit 505 inputs the geometric features of each joint combination into a trained deep convolutional network for feature extraction and classification, and outputs the possible probabilities of several behaviors, it is specifically used for:

[0274] The first geometric feature RGB image, the second geometric feature RGB image, and the third geometric feature RGB image are respectively input into a trained convolutional neural network ResNet-50 for feature extraction, and the possible probabilities of several behaviors are output respectively.

[0275] Optionally, the possible probabilities of the several behaviors include: the posterior probability b1 of the behavior extracted from the first geometric feature RGB image, the posterior probability b2 of the behavior extracted from the second geometric feature RGB image, and the posterior probability b3 of the behavior extracted from the third geometric feature RGB image.

[0276] When the fusion analysis unit 506 performs a comprehensive analysis of the possible probabilities of the several behaviors according to a preset fusion function, outputs the behavior with the highest possible probability, and obtains the target behavior, it is specifically used for:

[0277] P(L|S)=η1b1+η2b2+η3b3

[0278] η1, η2, and η3 are preset weights;

[0279] The posterior probability vector P(L|S) represents the probability that the skeleton belongs to action label category L.

[0280] label = Find(max(P(L|S)))

[0281] The label represents the target behavior.

[0282] The operation performed by the behavior recognition device based on the geometric features of the skeleton joint assembly in this application embodiment is the same as that described above. Figure 1 The operations performed in the middle are similar, and will not be described in detail here.

[0283] The computer device in the embodiments of this application is described below. Please refer to [link / reference]. Figure 6 One embodiment of the computer device in this application includes: the computer device 600 may include one or more central processing units (CPUs) 801 and a memory 602, wherein the memory 602 stores one or more application programs or data. The memory 602 is volatile or persistent storage. The program stored in the memory 602 may include one or more modules, each module including a series of instruction operations on the computer device. Furthermore, the processor 601 may be configured to communicate with the memory 602 and execute the series of instruction operations in the memory 602 on the computer device 600. The computer device 600 may also include one or more wireless network interfaces 603, one or more input / output interfaces 604, and / or one or more operating systems, such as Windows Server, Mac OS, Unix, Linux, FreeBSD, etc. The processor 601 can execute the aforementioned... Figure 1 The specific operations performed in the illustrated embodiment will not be described in detail here.

[0284] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A behavior recognition method based on the geometric features of skeleton joint combinations, characterized in that, include: The skeleton is divided into X joint combinations, where X is a positive integer greater than 1; Describe the spatial positional characteristics of each of the joint combinations; Describe the relative geometric features of each of the joint combinations; The spatial position features and relative geometric features of each joint combination are combined to describe the geometric features of each joint combination. The geometric features of each joint combination are input into a trained deep convolutional network for feature extraction and classification, and the possible probabilities of several behaviors are output respectively. The possible probabilities of the several behaviors are comprehensively analyzed based on a preset fusion function, and the behavior with the highest possible probability is output to obtain the target behavior.

2. The behavior recognition method according to claim 1, characterized in that, X is 3; The skeleton is a human skeleton; the process of dividing the skeleton into X joint combinations includes: According to preset rules, invalid joints in the human skeleton are removed to obtain the remaining joints; The remaining joints are divided into three joint combinations P1, P2, and P3, wherein P1 includes the joints of the left and right arms among the remaining joints, P2 includes the joints of the left arm and right leg among the remaining joints, and P3 includes the joints of the right arm and left leg among the remaining joints.

3. The behavior recognition method according to claim 2, characterized in that, Each joint assembly includes a set of joint numbers, spatial connection relationships of the joints, and spatial connection order of the joints; P1, P2, and P3 are respectively described as follows: Wherein H represents the set of joint numbers of the left arm and the right arm; LG represents the set of joint numbers for the left arm and the right leg; RG represents the set of joint numbers for the right arm and the left leg; The J m This indicates the joint with serial number m; The J n This represents the joint with index n; The J k This indicates the joint with the serial number k.

4. The behavior recognition method according to claim 3, characterized in that, The spatial positional features describing each of the joint combinations include: The spatial positional characteristics of P1, P2, and P3 are described as follows: The This represents the three-dimensional coordinates of the joint with index i in P1 at time t, where T represents the duration; The This represents the three-dimensional coordinates of the joint with index j in P2 at time t; The This represents the three-dimensional coordinates of the joint with index k in P3 at time t.

5. The behavior recognition method according to claim 4, characterized in that, After describing the spatial location features of P1, P2, and P3 respectively, the method further includes: Regarding the spatial location features of P1, the Normalize the first X value to a value between 0 and 255; The first X value is represented as a 224×112 image using linear interpolation. Regarding the spatial location features of P1, the Normalize the first Y value to a value between 0 and 255; The first Y value is represented as a 224×112 image using linear interpolation. Regarding the spatial location features of P1, the Normalize to a first Z value between 0 and 255; The first Z value is represented as a 224×112 image using linear interpolation. The first RGB image is obtained by treating the first X value as the Red, Green, and Blue channels of the first RGB image, the first Y value as the first 224×112 image, and the first Z value as the Red, Green, and Blue channels of the first RGB image, respectively. The spatial location features of P1 are described using the first RGB image; Regarding the spatial location features of P2, the Normalize to a second X value between 0 and 255; The second X value is represented as a 224×112 image using linear interpolation. Regarding the spatial location features of P2, the Normalize the result to a second Y value between 0 and 255; The second Y value is represented as a 224×112 image using linear interpolation. Regarding the spatial location features of P2, the Normalize to a second Z value between 0 and 255; The second Z value is represented as a 224×112 image using linear interpolation. The second RGB image is obtained by treating the 224×112 image represented by the second X value, the 224×112 image represented by the second Y value, and the 224×112 image represented by the second Z value as the Red, Green, and Blue channels of the second RGB image, respectively. The second RGB image is used to describe the spatial location features of P2; Regarding the spatial location features of P3, the Normalize the result to a third X value between 0 and 255; The third X value is represented as a 224×112 image using linear interpolation. Regarding the spatial location features of P3, the Normalize the result to a third Y value between 0 and 255; The third Y value is represented as a 224×112 image using linear interpolation. Regarding the spatial location features of P3, the Normalize to a third Z value between 0 and 255; The third Z value is represented as a 224×112 image using linear interpolation. The 224×112 image represented by the third X value, the 224×112 image represented by the third Y value, and the 224×112 image represented by the third Z value are respectively regarded as the Red, Green, and Blue channels of the third RGB image to obtain the third RGB image. The spatial location features of P3 are described using the third RGB image.

6. The behavior recognition method according to claim 5, characterized in that, The relative geometric features describing each of the joint combinations include: The relative geometric features of P1 are described using the relative positional features of P1: The above J represents the arm joint i In the three-dimensional coordinates at time t, the arm joint includes the left arm joint and the right arm joint; The Indicates the arm joint J i The three-dimensional coordinates at time t-1; The Indicates the arm joint J i The relative positional characteristics at time t; The Considered as the relative geometric feature of P1; Calculate the relative distance d21 between the left arm joints in P2, the relative distance d22 between the left arm joint and the right leg joint, and the relative distance d23 between the left arm joint and the right leg joint relative to the preset origin. The relative distances d21, d22, and d23 are considered as the relative geometric features of P2; Calculate the relative distance d31 between the right arm joints in P3, the relative distance d32 between the right arm joint and the left leg joint, and the relative distance d33 between the right arm joint and the left leg joint relative to the preset origin. The relative distances d31, d32, and d33 are considered as the relative geometric features of P3.

7. The behavior recognition method according to claim 6, characterized in that, After describing the relative geometric features of P1, P2, and P3 respectively, the method further includes: The relative geometric features of P1 Normalize to a first relative value of X between 0 and 255; The first relative value of X is represented as a 224×112 image using linear interpolation. Regarding the spatial location features of P1, the Normalize to a first relative value of Y between 0 and 255; The first relative value of Y is represented as a 224×112 image using linear interpolation. Regarding the spatial location features of P1, the Normalize to the first Z-relative value between 0 and 255; The first Z relative value is represented as a 224×112 image using linear interpolation. The first 224×112 image represented by the first X relative value, the first 224×112 image represented by the first Y relative value, and the first 224×112 image represented by the first Z relative value are respectively regarded as the Red, Green, and Blue channels of the fourth RGB image to obtain the fourth RGB image. The fourth RGB image is used to describe the relative geometric features of P1; The relative distance d21 in the relative geometric features of P2 is normalized to a first d21 value between 0 and 255; The first d21 value is represented as a 224×112 image using linear interpolation. The relative distance d22 in the relative geometric features of P2 is normalized to a first d22 value between 0 and 255; The first d22 value is represented as a 224×112 image using linear interpolation. The relative distance d23 in the relative geometric features of P2 is normalized to a first d23 value between 0 and 255; The first d23 value is represented as a 224×112 image using linear interpolation. The first d21 value is used to represent a 224×112 image, the first d22 value is used to represent a 224×112 image, and the first d23 value is used to represent a 224×112 image, which are respectively regarded as the Red, Green, and Blue channels of the fifth RGB image, to obtain the fifth RGB image. The fifth RGB image is used to describe the relative geometric features of P2; The relative distance d31 in the relative geometric features of P3 is normalized to a first d31 value between 0 and 255; The first d31 value is represented as a 224×112 image using linear interpolation. The relative distance d32 in the relative geometric features of P3 is normalized to a first d32 value between 0 and 255; The first d32 value is represented as a 224×112 image using linear interpolation. The relative distance d33 in the relative geometric features of P3 is normalized to a first d33 value between 0 and 255; The first d33 value is represented as a 224×112 image using linear interpolation. The first d31 value is used to represent a 224×112 image, the first d32 value is used to represent a 224×112 image, and the first d33 value is used to represent a 224×112 image, which are respectively regarded as the Red, Green, and Blue channels of the sixth RGB image, to obtain the sixth RGB image. The sixth RGB image is used to describe the relative geometric features of P3.

8. The behavior recognition method according to claim 7, characterized in that, The spatial position features of each joint combination combined with the relative geometric features are described as geometric features including: By combining the first RGB image and the fourth RGB image, a first geometric feature RGB image with a length vector of 224×224 corresponding to P1 is formed; By combining the second RGB image and the fifth RGB image, a second geometric feature RGB image with a length vector of 224×224 corresponding to P2 is formed; By combining the third RGB image and the sixth RGB image, a third geometric feature RGB image corresponding to the 224×224 length vector of P3 is formed.

9. The behavior recognition method according to claim 8, characterized in that, The geometric features of each joint combination are input into a trained deep convolutional network for feature extraction and classification, and the network outputs the possible probabilities of several behaviors, including: The first geometric feature RGB image, the second geometric feature RGB image, and the third geometric feature RGB image are respectively input into a trained convolutional neural network ResNet-50 for feature extraction, and the possible probabilities of several behaviors are output respectively.

10. The behavior recognition method according to claim 9, characterized in that, The possible probabilities of the several behaviors include: the posterior probability b1 of the behavior extracted from the first geometric feature RGB image, the posterior probability b2 of the behavior extracted from the second geometric feature RGB image, and the posterior probability b3 of the behavior extracted from the third geometric feature RGB image. The step of comprehensively analyzing the possible probabilities of the several behaviors based on a preset fusion function, and outputting the behavior with the highest possible probability to obtain the target behavior, includes: P(L|S)=η1b1+η2b2+η3b3 η1, η2, and η3 are preset weights; The posterior probability vector P(L|S) represents the probability that the skeleton belongs to action label category L. label = Find(max(P(L|S))) The label represents the target behavior.

Images