Human action recognition method based on complementary fusion of multi-granularity skeleton spatiotemporal features

Abstract

In the disclosed embodiment, a method for human motion recognition based on complementary fusion of multi-granularity skeleton spatiotemporal features is provided, which belongs to the field of image recognition technology, and specifically includes: for each frame of human motion to be recognized, a multi-granularity human skeleton topology map is generated, and at each granularity, the human skeleton is fused with the end joint point as the starting point and in the direction of the spine center, and the trunk joint point information is retained; for the human motion sequence in the human skeleton topology map of each granularity, a human skeleton adjacency matrix of the corresponding granularity is constructed, and graph convolution operations and time series convolution operations are performed; based on the complementary analysis between features of different granularities, a multi-granularity channel attention mechanism and an improved attention fusion unit are designed to complementarily improve the spatiotemporal dynamic features of different granularities, and multi-granularity fusion features are obtained and input into the algorithm decision maker to obtain the motion recognition result. Through the scheme disclosed in the present invention, the recognition accuracy and adaptability are improved.

Description

Technical Field

[0001] The disclosed embodiments relate to the field of image recognition technology, and in particular to a method for human motion recognition based on complementary fusion of multi-granularity skeleton spatiotemporal features. Background Art

[0002] At present, human skeleton motion recognition technology uses advanced technologies in computer vision and deep learning to detect and track key points of the human body, thus realizing automatic recognition and analysis of human motion. This technology has broad application prospects in the fields of intelligent monitoring, virtual reality, sports training, medical rehabilitation, etc.

[0003] Current cutting-edge recognition technologies, especially multimodal fusion technologies based on deep learning, have become a research hotspot in the field of human skeleton action recognition. The algorithms in these technologies can effectively capture the temporal and spatial information in action sequences by performing convolution operations in time and space, thereby achieving accurate classification and recognition of different actions. The development of this type of technology has brought new breakthroughs and progress to human action recognition.

[0004] In reality, human action recognition technology still faces some major challenges and limitations. Due to the diversity and complexity of human actions, the same action may show different characteristics in different scenes and environments, making action recognition more complicated and difficult. When human actions are occluded by objects, even partial occlusion will significantly affect the accuracy of recognition, and current technology is not ideal in dealing with these occlusion problems. In addition, there are significant differences in the body structure and movement characteristics of different individuals, and existing general models often cannot adapt well to these individual differences. This model has poor robustness when facing diverse human actions and is difficult to provide accurate recognition results.

[0005] It can be seen that there is an urgent need for a human action recognition method based on the complementary fusion of multi-granularity skeleton spatiotemporal features with high recognition accuracy and adaptability. Summary of the invention

[0006] In view of this, the embodiments of the present disclosure provide a method for human motion recognition based on the complementary fusion of multi-granularity skeleton spatiotemporal features, which at least partially solves the problems of poor recognition accuracy and adaptability in the prior art.

[0007] The present disclosure provides a method for human action recognition based on complementary fusion of multi-granularity skeleton spatiotemporal features, including:

[0008] Step 1: for each frame of the human action to be recognized, a multi-granularity human skeleton topology map is generated, and at each granularity, the human skeleton is fused in the direction from the end joint point to the center of the spine, and the main joint point information is retained;

[0009] Step 2: construct a human skeleton adjacency matrix of the corresponding granularity for the human motion sequence in the human skeleton topology graph of each granularity, and perform graph convolution operations and temporal convolution operations to extract the spatial features and temporal features of each frame of human motion, and finally obtain the spatiotemporal dynamic features of the human motion sequence of each granularity;

[0010] Step 3: Based on the complementarity analysis between features of different granularities, a multi-granularity channel attention mechanism and an improved attention fusion unit are designed to complementarily improve the spatiotemporal dynamic features of different granularities, obtain multi-granularity fusion features and input them into the algorithm decision maker to obtain the action recognition results.

[0011] According to a specific implementation of the embodiment of the present disclosure, step 1 specifically includes:

[0012] Step 1.1, take granularity 0 as the initial granularity, at which the skeleton topology remains in its original state, each joint point is retained, and the initial human skeleton is retained;

[0013] Step 1.2: for each granularity 1 to granularity N, a step-by-step fusion method is adopted to gradually fuse the position of the limb end nodes and the adjacent joint points to obtain N+1 types of human skeleton topology maps of different granularities;

[0014] Step 1.3: According to the human skeleton topology map of each granularity and the matching alignment formula, skeleton compression is performed on the human body posture of each frame in the sequence.

[0015] According to a specific implementation of the embodiment of the present disclosure, the matching alignment formula is:

[0016]

[0017] Where D is the input skeleton sequence, k is the number of compression rounds, parts is the fused limb part, p is the current fused limb, cat is the fusion operation, that is, the adjacent nodes at the end of each limb in the third dimension of the sequence are fused, and X k is the obtained fused skeleton sequence, T represents the operation of saving the current skeleton sequence, dim represents the dimension of the fused skeleton sequence, Represents the fusion operation of limb p during the k-th compression process.

[0018] According to a specific implementation of the embodiment of the present disclosure, step 2 specifically includes:

[0019] Step 2.1, for each granularity of human action skeleton sequence, construct a human skeleton adjacency matrix of the corresponding granularity;

[0020] Step 2.2, according to the spatiotemporal graph convolution formula, based on the constructed human skeleton adjacency matrix, filtering operation is performed to capture the local spatial information between nodes, and spatial features of different granularities of human posture in each frame are obtained. In addition, for the spatial features of different granularities of human posture in T frames, features are extracted in the time dimension using a sliding window method to obtain the temporal features of the entire action sequence, and the spatiotemporal dynamic features of the human action sequence of each granularity are obtained by combining the spatial features and the temporal features.

[0021] According to a specific implementation method of the embodiment of the present disclosure, the spatiotemporal graph convolution formula is:

[0022]

[0023] Among them, D max is the predefined maximum distance, F in and F out are the input and output feature maps, A d represents the adjacency matrix of the graph distance d, is a normalized diagonal matrix, Indicates A d The k-th row element, α is set to a small value to avoid Λ d For each adjacency matrix, set a learnable matrix M d , to indicate the importance of each edge.

[0024] According to a specific implementation of the embodiment of the present disclosure, step 3 specifically includes:

[0025] Step 3.1, the spatiotemporal dynamic features corresponding to granularity 0 are complementarily improved with the spatiotemporal dynamic features corresponding to granularity 1 to granularity N, and N improved features are obtained, wherein the expression of the complementary improvement is:

[0026] F' 0n =[(Q0*K0*F n )*(Q0*K0*F n ) T ]⊙F0

[0027] Among them, Q0 represents the query matrix generated by linear mapping the feature map F0 corresponding to the granularity 0 of each frame action sequence, K0 represents the key map matrix, and F n Represents the spatiotemporal dynamic characteristics corresponding to any granularity from 1 to N, where n ranges from 1 to N;

[0028] Step 3.2: Based on the multi-granularity channel attention mechanism, the N improved features are fused to obtain multi-granularity fused features.

[0029] In step 3.3, the multi-granularity fusion features are input into the algorithm decision maker to obtain the action recognition result.

[0030] According to a specific implementation of the embodiment of the present disclosure, step 3.2 specifically includes:

[0031] Step 3.2.1, input N improved features X(F' 01 , F' 02 , ..., F' 0n ), the dimension is (C, H, W), the N improved features X are globally averaged pooled to generate a C×1×1 feature map, and after two layers of point-by-point convolution, a normalization layer and an activation function, the data enhanced matrix X1 is obtained, where the expression of the data enhanced matrix X1 is

[0032] X1=Pool(BN(Conv(ReLU(BN(Conv(X))))))

[0033] Among them, Pool() is the pooling layer, BN() is the data normalization calculation, Conv() is the convolution operation, and ReLU() is the activation function;

[0034] Step 3.2.2, the N improved features X are subjected to two layers of point-by-point convolution, a normalization layer, and an activation function to obtain the data-enhanced matrix X2, where the expression of the data-enhanced matrix X2 is

[0035] X2 = BN(Conv(ReLU(BN(Conv(X))))))

[0036] Among them, BN() is the data normalization calculation, Conv() is the convolution operation, and ReLU() is the activation function;

[0037] Step 3.2.3, perform matrix addition on X1 and X2 to obtain a matrix X with dimensions (C, H, W) 12 , X 12 The attention weight E is generated by the Sigmoid activation function, and the attention weight E is multiplied element by element with the N improved features X to finally obtain the multi-granularity fusion feature F.

[0038] The human motion recognition scheme based on complementary fusion of multi-granularity skeleton spatiotemporal features in the disclosed embodiment includes: step 1, for each frame of the human motion to be recognized, a multi-granularity human skeleton topology map is generated, and at each granularity, the human skeleton is fused with the end joint point as the starting point and in the direction of the spine center, and the trunk joint point information is retained; step 2, for the human motion sequence in the human skeleton topology map of each granularity, a human skeleton adjacency matrix of the corresponding granularity is constructed, and graph convolution operations and time series convolution operations are performed to extract the spatial features and time series features of each frame of human motion, and finally the spatiotemporal dynamic features of each granularity of human motion sequence are obtained; step 3, based on the complementarity analysis between features of different granularities, a multi-granularity channel attention mechanism and an improved attention fusion unit are designed to complementarily improve the spatiotemporal dynamic features of different granularities, obtain multi-granularity fusion features and input them into the algorithm decision maker to obtain the motion recognition result.

[0039] The beneficial effects of the embodiments of the present disclosure are as follows: through the scheme of the present disclosure, the concept of multi-granularity fusion processing is introduced into the multi-granularity processing flow, and the human skeleton data is compressed and fused at different granularities; when processing the human motion sequence of each granularity, graph convolution operations and temporal convolution operations are used to extract spatial features and temporal features, and multi-granularity time-space dynamic fusion is performed; multi-granularity complementary improved fusion based on MS-CAM improves recognition accuracy and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 A schematic diagram of a flow chart of a method for human action recognition based on complementary fusion of multi-granularity skeleton spatiotemporal features provided in an embodiment of the present disclosure;

[0042] Figure 2 A schematic diagram of the algorithm structure flow of a human action recognition method based on complementary fusion of multi-granularity skeleton spatiotemporal features provided in an embodiment of the present disclosure;

[0043] Figure 3 A schematic diagram of a multi-granularity compression process provided by an embodiment of the present disclosure;

[0044] Figure 4 A schematic diagram of a multi-granularity compression process of an occlusion dataset provided in an embodiment of the present disclosure;

[0045] Figure 5A flow chart of the complementary improved fusion algorithm structure provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0046] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0047] The following describes the embodiments of the present disclosure through specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0048] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0049] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0050] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0051] The disclosed embodiment provides a method for human motion recognition based on the complementary fusion of multi-granularity skeleton spatiotemporal features, which can be applied to human motion recognition processes in scenarios such as computer vision and deep learning.

[0052] See also Figure 1 , is a flow chart of a method for human action recognition based on complementary fusion of multi-granularity skeleton spatiotemporal features provided by an embodiment of the present disclosure. Figure 1 and Figure 2 As shown, the method mainly comprises the following steps:

[0053] Step 1: for each frame of the human action to be recognized, a multi-granularity human skeleton topology map is generated, and at each granularity, the human skeleton is fused in the direction from the end joint point to the center of the spine, and the main joint point information is retained;

[0054] Furthermore, the step 1 specifically includes:

[0055] Step 1.1, take granularity 0 as the initial granularity, at which the skeleton topology remains in its original state, each joint point is retained, and the initial human skeleton is retained;

[0056] Step 1.2: for each granularity 1 to granularity N, a step-by-step fusion method is adopted to gradually fuse the position of the limb end nodes and the adjacent joint points to obtain N+1 types of human skeleton topology maps of different granularities;

[0057] Step 1.3: According to the human skeleton topology map of each granularity and the matching alignment formula, skeleton compression is performed on the human body posture of each frame in the sequence.

[0058] Furthermore, the matching alignment formula is:

[0059]

[0060] Where D is the input skeleton sequence, k is the number of compression rounds, parts is the fused limb part, p is the current fused limb, cat is the fusion operation, that is, the adjacent nodes at the end of each limb in the third dimension of the sequence are fused, and X k is the obtained fused skeleton sequence, T represents the operation of saving the current skeleton sequence, dim represents the dimension of the fused skeleton sequence, Represents the fusion operation of limb p during the k-th compression process.

[0061] In the specific implementation, for each frame of the human body movement recognition, a multi-granularity human skeleton topology map is generated, and at each granularity, the human skeleton is fused with the end joint point as the starting point and toward the center of the spine, and the main joint point information is retained. The specific compression process can be carried out in the following steps:

[0062] 1. Granularity 0: At the initial granularity, the skeleton topology remains in its original state without any fusion process. Each joint point is retained, and the initial human skeleton is retained.

[0063] 2. Compression fusion of granularity 1 to N: For each subsequent granularity, the present invention adopts a step-by-step fusion method, that is, gradually fusion the position of the terminal nodes of the limbs and the adjacent joints. The specific process is that when the granularity is n, the five terminal joints of the head and limbs in the human skeleton (the corresponding node numbers of the limbs - "head": [3,2], "right arm": [23,24,11,10,9,8], "left arm": [21,22,7,6,5,4], "right leg": [19,18,17,16] and "left leg": [15,14,13,12]) are fused n times with their adjacent joints based on the original skeleton data, and the merging method can be selected from averaging, weighted summation, parameterized calculation and other operations. The joints obtained after merging are used as new terminal joints for subsequent compression, and the trunk nodes ("trunk": [20,1,0]) do not participate in the merging. It should be noted that when a limb is compressed in advance and the terminal joint cannot be fused, the terminal node of the previous round of fusion is used. When all limbs are compressed, that is, only 8 trunk joints are left in the entire skeleton (the corresponding node numbers of the limbs are "head": [2], "right arm": [3], "left arm": [4], "right leg": [5], "left leg": [6] and "trunk": [7,1,0]), the multi-granularity compression process ends, and N+1 types of human skeleton topology maps with different granularities are obtained.

[0064] 3. Multi-granularity skeleton compression: Figure 3 As shown, skeleton compression can be performed on each frame of the human body posture in the sequence according to the human skeleton topology map of each granularity. This step needs to be achieved by matching and aligning the original skeleton map with the skeleton map compressed at the corresponding granularity. The formula used is as follows:

[0065]

[0066] Where D is the input skeleton sequence, k is the number of compression rounds, parts is the fused limb part, p is the current fused limb, cat is the fusion operation, that is, the adjacent nodes at the end of each limb in the third dimension of the sequence are fused. The present invention adopts the average of two nodes, X k is the obtained fusion skeleton sequence, and finally X K ={X K :k=0,1,2,…,n}. Input a frame of human action skeleton sequence (C, T, J). When k=0, no fusion operation is performed, that is, (C, T, J0) is obtained. When k>=1, each limb is fused once every k=k+1 to generate a set of human posture skeleton sequence (C, T, J K ), when k = N, the fusion operation ends. Finally, N+1 sets of human posture skeleton sequences (C, T, J0), (C, T, J1), ..., (C, T, J N). In the matching process, operations such as mapping of nodes of each limb part and adjusting the node sequence number of the edge are involved to ensure that the compressed skeleton graph can accurately represent the original action. When the collected image frame is occluded, the corresponding multi-granularity compression process is as follows Figure 4 shown.

[0067] Step 2: construct a human skeleton adjacency matrix of the corresponding granularity for the human motion sequence in the human skeleton topology graph of each granularity, and perform graph convolution operations and temporal convolution operations to extract the spatial features and temporal features of each frame of human motion, and finally obtain the spatiotemporal dynamic features of the human motion sequence of each granularity;

[0068] Based on the above embodiment, step 2 specifically includes:

[0069] Step 2.1, for each granularity of human action skeleton sequence, construct a human skeleton adjacency matrix of the corresponding granularity;

[0070] Step 2.2, according to the spatiotemporal graph convolution formula, based on the constructed human skeleton adjacency matrix, filtering operation is performed to capture the local spatial information between nodes, and spatial features of different granularities of human posture in each frame are obtained. In addition, for the spatial features of different granularities of human posture in T frames, features are extracted in the time dimension using a sliding window method to obtain the temporal features of the entire action sequence, and the spatiotemporal dynamic features of the human action sequence of each granularity are obtained by combining the spatial features and the temporal features.

[0071] Furthermore, the spatiotemporal graph convolution formula is

[0072]

[0073] Among them, D max is the predefined maximum distance, F in and F out are the input and output feature maps, A d represents the adjacency matrix of the graph distance d, is a normalized diagonal matrix, Indicates A d The k-th row element, α is set to a small value to avoid Λ d For each adjacency matrix, set a learnable matrix M d , to indicate the importance of each edge.

[0074] In the specific implementation, each frame corresponds to a human action posture. After the previous step, each frame has N+1 groups of granular human action skeleton sequences, namely: (C, T, J0), (C, T, J1), ..., (C, T, J N). Therefore, it is necessary to construct a human skeleton adjacency matrix of the corresponding granularity for each human motion sequence, and perform graph convolution operations and temporal convolution operations to extract the spatial and temporal features of each frame of human motion, and finally obtain the spatiotemporal dynamic features of each granularity human motion sequence. The specific process is as follows:

[0075] 1. Construct the human skeleton adjacency matrix: For each granularity of human action sequence, it is necessary to construct the human skeleton adjacency matrix of the corresponding granularity. These matrices describe the connection relationship between each joint point in the human skeleton topology diagram. The sequence number of each adjacency matrix needs to match the compressed and adjusted skeleton sequence, which is the basis for subsequent graph convolution operations.

[0076] 2. Graph convolution operation: Based on the constructed human skeleton adjacency matrix, filtering operation is performed to capture the local spatial information between nodes and obtain spatial features of different granularities of human posture in each frame. Temporal convolution operation: For the spatial features of different granularities of human posture in T frames, features are extracted in the time dimension using a sliding window method to obtain the temporal features of the entire action sequence. The spatiotemporal graph convolution operation formula used in the present invention is as follows:

[0077]

[0078] By inputting N+1 sets of granular human motion skeleton sequences, N+1 spatiotemporal dynamic features will be generated.

[0079] Step 3: Based on the complementarity analysis between features of different granularities, a multi-granularity channel attention mechanism and an improved attention fusion unit are designed to complementarily improve the spatiotemporal dynamic features of different granularities, obtain multi-granularity fusion features and input them into the algorithm decision maker to obtain the action recognition results.

[0080] Based on the above embodiment, step 3 specifically includes:

[0081] Step 3.1, the spatiotemporal dynamic features corresponding to granularity 0 are complementarily improved with the spatiotemporal dynamic features corresponding to granularity 1 to granularity N, and N improved features are obtained, wherein the expression of the complementary improvement is:

[0082] F' 0n =[(Q0*K0*F n )*(Q0*K0*F n ) T ]⊙F0

[0083] Among them, Q0 represents the query matrix generated by linear mapping the feature map F0 corresponding to the granularity 0 of each frame action sequence, K0 represents the key map matrix, and F n Represents the spatiotemporal dynamic characteristics corresponding to any granularity from 1 to N, where n ranges from 1 to N;

[0084] Step 3.2: Based on the multi-granularity channel attention mechanism, the N improved features are fused to obtain multi-granularity fused features.

[0085] In step 3.3, the multi-granularity fusion features are input into the algorithm decision maker to obtain the action recognition result.

[0086] Furthermore, the step 3.2 specifically includes:

[0087] Step 3.2.1, input N improved features X(F' 01 , F' 02 , ..., F' 0N ), the dimension is (C, H, W), the N improved features X are globally averaged pooled to generate a C×1×1 feature map, and after two layers of point-by-point convolution, a normalization layer and an activation function, the data enhanced matrix X1 is obtained, where the expression of the data enhanced matrix X1 is

[0088] X1=Pool(BN(Conv(ReLU(BN(Conv ( X))))))

[0089] Among them, Pool() is the pooling layer, BN() is the data normalization calculation, Conv() is the convolution operation, and ReLU() is the activation function;

[0090] Step 3.2.2, the N improved features X are subjected to two layers of point-by-point convolution, a normalization layer, and an activation function to obtain the data-enhanced matrix X2, where the expression of the data-enhanced matrix X2 is

[0091] X2 = BN(Conv(ReLU(BN(Conv(X))))))

[0092] Among them, BN() is the data normalization calculation, Conv() is the convolution operation, and ReLU() is the activation function;

[0093] Step 3.2.3, perform matrix addition on X1 and X2 to obtain a matrix X with dimensions (C, H, W) 12 , X 12 The attention weight E is generated by the Sigmoid activation function, and the attention weight E is multiplied element by element with the N improved features X to finally obtain the multi-granularity fusion feature F.

[0094] When implementing it, Figure 5 As shown in the figure, based on the complementary analysis between features of different granularities, a multi-granularity channel attention mechanism and an improved attention fusion unit are designed to complement and improve the action features of different granularities. The specific process is as follows:

[0095] (1) F0 and F1 to F n Complementary improvements are made to obtain N improved features. n Take the complementary improvement process of as an example, the details are as follows:

[0096] The feature map (F0, F n ) is input into the complementary improved attention unit for processing. The complementary improved processing formula is:

[0097] F' 0n = [(Q0 *K0*F n )*(Q0*K0*F n ) T ]⊙F0 (3)

[0098] The specific process is as follows:

[0099] Each feature map F0 generates query (Q0) and key (K0) map matrices through linear mapping, Q0*K0 obtains the attention mask (M0) map matrix, and then M0 is combined with F n Multiply them to get F n ', F n 'Generate a query (Q n ), key (K n ) graph matrix, Q n *K n Get the attention mask (M n ) graph matrix, and finally F0 and M n Multiply to generate complementary improved features F 0n ′.

[0100] F0 and F1, F2, ..., F n Input them into the complementary improved attention unit respectively to obtain N complementary improved features F 01 ′,F 02 ′,…,F 0n ′.

[0101] (2) Based on the multi-granularity channel attention mechanism, the N improved features are fused. The specific process is as follows:

[0102] Input N features X (i.e. F 01 ′,F 02 ′, ..., F 0n ′), with dimensions of (C, H, W).

[0103] ① These features X are globally averaged and pooled to generate a C×1×1 feature map. After two layers of point-by-point convolution, a normalization layer, and an activation function, the formula is

[0104] X1=Pool(BN(Conv(ReLU(BN(Conv(X)))))) (4)

[0105] , and get the data enhanced X1.

[0106] ② These features X are passed through two layers of point-by-point convolution, a normalization layer and an activation function. The formula is

[0107] X2=BN(Conv(ReLU)(BN(Conv(X))))) (5)

[0108] , and get the data enhanced X2.

[0109] ③ Add X1 and X2 to get a matrix X with dimensions (C, H, W) 12 , X 12 The attention weight E is generated by the Sigmoid activation function. E and X are multiplied element by element, and the final F is the multi-granularity fusion feature.

[0110] (3) The multi-granularity spatiotemporal dynamic fusion feature F is input into the decision maker of the algorithm to obtain the final action recognition result.

[0111] The human motion recognition method based on complementary fusion of multi-granularity skeleton spatiotemporal features provided in this embodiment introduces the concept of multi-granularity fusion processing in the multi-granularity processing flow, compresses and fuses the human skeleton data at different granularities; when processing the human motion sequence of each granularity, graph convolution operations and temporal convolution operations are used to extract spatial features and temporal features, and multi-granularity spatiotemporal dynamic fusion is performed; multi-granularity complementary improved fusion based on MS-CAM improves recognition accuracy and adaptability.

[0112] The method disclosed in the present invention will be further described below in conjunction with a specific embodiment, using the classic NTU-RGB+D 60 human skeleton dataset, which is a large video dataset widely used in behavior recognition tasks, containing 60 kinds of daily activities and human actions. These datasets were created by Nanyang Technological University and consist of depth information, RGB video, three-dimensional joint point data, and infrared data captured by the Microsoft Kinect v2 sensor. NTU-RGB+D 60 contains 56,880 video samples, covering a wide range of actions. This dataset is very important in behavior recognition research. Its large scale, rich action types, and diverse data types give it significant advantages in improving the generalization ability and accuracy of the model. The samples in each dataset are performed by different participants under multiple perspectives, thereby enhancing the robustness of the model to perspectives and individual differences.

[0113] After obtaining the training set and test set on the relevant data set, the human skeleton action recognition algorithm based on multi-granularity compression recognition is trained using the training set, and then the test set is used for testing to evaluate the algorithm performance. The present invention uses the selection accuracy (ACC) as the measurement indicator and compares it on the complete data set and the occluded data set.

[0114] In order to objectively present the research results of the present invention, we selected several relevant representative research results for comparison. The comparison of the performance measurement methods of the NTU RGB+D60 complete dataset is shown in Table 1. It can be found that the method of the present invention has achieved significant improvements on both the dataset (CS) and the dataset (CV), reaching 95.5% and 98.1% respectively. This improvement in accuracy indicates that the method of the present invention has shown higher reliability in the task of identifying human skeleton movements. These improvements show that the method of the present invention has obvious advantages in improving the accuracy of human skeleton movement recognition.

[0115] The test performance comparison based on the CS limb occlusion dataset is shown in Table 2, and the test performance comparison based on the CS frame occlusion dataset is shown in Table 3. In Tables 2 and 3, the method disclosed in the present invention shows extremely high accuracy under different types of occlusion, whether it is limb occlusion or frame occlusion, with average values ​​of 84.4% and 83.4% respectively.

[0116] Compared with other models, although RA-GCN performs well in certain specific cases, it is still inferior to the method disclosed in this paper. Especially in complex occlusion situations (such as double arm occlusion, right arm occlusion and multi-frame occlusion), the method disclosed in this paper shows more obvious advantages.

[0117] The method disclosed in the present invention not only performs well in the unoccluded condition, but also maintains a high accuracy rate in various occlusion conditions, demonstrating the powerful ability of the model disclosed in the present invention in handling complex occlusion problems.

[0118] Model year CS CV <![CDATA[H-BRNN [1] ]]> 2015 59.1 64.0 <![CDATA[PA-LSTM [2] ]]> 2016 62.9 70.3 <![CDATA[VA-LSTM [3] ]]> 2017 79.4 87.6 <![CDATA[ST-GCN(baseline) [4] ]]> 2018 81.5 88.3 <![CDATA[HCN [5] ]]> 2018 86.5 91.1 <![CDATA[SR-TSL [6] ]]> 2018 84.8 92.4 <![CDATA[PB-GCN [7] ]]> 2018 87.5 93.2 <![CDATA[*2s RA-GCN [8] ]]> 2019 85.8 93.0 <![CDATA[*3s RA-GCN [8] ]]> 2019 85.9 93.5 <![CDATA[GAP [9] ]]> 2023 92.9 97.0 Methods of the present disclosure - 95.5 98.1

[0119] Table 1

[0120]

[0121] Table 2

[0122]

[0123] Table 3

[0124] It should be understood that various parts of the present disclosure may be implemented in hardware, software, firmware or a combination thereof.

[0125] The above is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present disclosure should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for human action recognition based on complementary fusion of multi-granularity skeleton spatiotemporal features, characterized in that: include: Step 1: for each frame of the human action to be recognized, a multi-granularity human skeleton topology map is generated, and at each granularity, the human skeleton is fused in the direction from the end joint point to the center of the spine, and the main joint point information is retained; The step 1 specifically includes: Step 1.1, take granularity 0 as the initial granularity, at which the skeleton topology remains in its original state, each joint point is retained, and the initial human skeleton is retained; Step 1.2: for each granularity 1 to granularity N, a step-by-step fusion method is adopted to gradually fuse the position of the limb end nodes and the adjacent joint points to obtain N+1 types of human skeleton topology maps of different granularities; Step 1.3, according to the human skeleton topology map of each granularity and the matching alignment formula, skeleton compression is performed on each frame of the human body posture in the sequence; Step 2: construct a human skeleton adjacency matrix of the corresponding granularity for the human motion sequence in the human skeleton topology graph of each granularity, and perform graph convolution operations and temporal convolution operations to extract the spatial features and temporal features of each frame of human motion, and finally obtain the spatiotemporal dynamic features of the human motion sequence of each granularity; Step 3: Based on the complementarity analysis between features of different granularities, a multi-granularity channel attention mechanism and an improved attention fusion unit are designed to complementarily improve the spatiotemporal dynamic features of different granularities, obtain multi-granularity fusion features and input them into the algorithm decision maker to obtain the action recognition results; The step 3 specifically includes: Step 3.1, the spatiotemporal dynamic features corresponding to granularity 0 are complementarily improved with the spatiotemporal dynamic features corresponding to granularity 1 to granularity N, and N improved features are obtained, wherein the expression of the complementary improvement is: F' 0n =[(Q0*K0*F n )*(Q0*K0*F n ) T ]⊙F0 Among them, Q0 represents the query matrix generated by linear mapping the feature map F0 corresponding to the granularity 0 of each frame action sequence, K0 represents the key map matrix, and F n Indicates the spatiotemporal dynamic characteristics corresponding to any granularity from 1 to N, with a value range of 1 to N; Step 3.2: Based on the multi-granularity channel attention mechanism, the N improved features are fused to obtain multi-granularity fused features. In step 3.3, the multi-granularity fusion features are input into the algorithm decision maker to obtain the action recognition result.

2. The method according to claim 1, characterized in that , the matching alignment formula is Where D is the input skeleton sequence, k is the number of compression rounds, parts is the fused limb part, p is the current fused limb, cat is the fusion operation, that is, the adjacent nodes at the end of each limb in the third dimension of the sequence are fused, and X k is the obtained fused skeleton sequence, T represents the operation of saving the current skeleton sequence, dim represents the dimension of the fused skeleton sequence, Represents the fusion operation of limb p during the k-th compression process.

3. The method according to claim 2, characterized in that , the step 2 specifically includes: Step 2.1, for each granularity of human action skeleton sequence, construct a human skeleton adjacency matrix of the corresponding granularity; Step 2.2, according to the spatiotemporal graph convolution formula, based on the constructed human skeleton adjacency matrix, filtering operation is performed to capture the local spatial information between nodes, and spatial features of different granularities of human posture in each frame are obtained. In addition, for the spatial features of different granularities of human posture in T frames, features are extracted in the time dimension using a sliding window method to obtain the temporal features of the entire action sequence, and the spatiotemporal dynamic features of the human action sequence of each granularity are obtained by combining the spatial features and the temporal features.

4. The method according to claim 3, characterized in that ,The spatiotemporal graph convolution formula is Among them, D max is the predefined maximum distance, F in and F out are the input and output feature maps, A d represents the adjacency matrix of the graph distance d, is a normalized diagonal matrix, Indicates A d The k-th row element, α is set to a small value to avoid Λ d For each adjacency matrix, set a learnable matrix M d , to indicate the importance of each edge.

5. The method according to claim 4, characterized in that , the step 3.2 specifically includes: Step 3.2.1, input N improved features X(F' 01 , F' 02 , ..., F' 0n ), the dimension is (C, H, W), the N improved features X are globally averaged pooled to generate a C×1×1 feature map, and after two layers of point-by-point convolution, a normalization layer and an activation function, the data enhanced matrix X1 is obtained, where the expression of the data enhanced matrix X1 is X1=Pool(BN(Conv(ReLU(BN(Conv(X)))))) Among them, Pool() is the pooling layer, BN() is the data normalization calculation, Conv() is the convolution operation, and ReLU() is the activation function; Step 3.2.2, the N improved features X are subjected to two layers of point-by-point convolution, a normalization layer, and an activation function to obtain the data-enhanced matrix X2, where the expression of the data-enhanced matrix X2 is X2 = BN(Conv(ReLU(BN(Conv(X)))))) Among them, BN() is the data normalization calculation, Conv() is the convolution operation, and ReLU() is the activation function; Step 3.2.3, perform matrix addition on X1 and X2 to obtain a matrix X with dimensions (C, H, W) 12 , X 12 The attention weight E is generated by the Sigmoid activation function, and the attention weight E is multiplied element-by-element with the N improved features X to finally obtain the multi-granularity fusion feature F.

Images