Group behavior identification method based on 3D human body skeleton

Abstract

The invention relates to a group behavior identification method based on a 3D human body skeleton. The method comprises the following steps: S1, obtaining training set data; s2, extracting a first feature vector of each independent individual based on the first skeleton sequence in combination with a backbone extraction network, and verifying a first individual classification result and a corresponding individual behavior labeling result; s3, splicing the first feature vectors of the independent individuals to obtain a first group feature, and processing the first group feature by adopting a self-attention-cross attention neural network to obtain a second group feature; s4, target features are extracted based on the second group features, a first group behavior classification result is obtained, and model parameters of the backbone extraction network are modified based on the difference between the first group behavior classification result and a corresponding group behavior labeling result; and S5, carrying out group behavior identification on the first skeleton sequence to be identified by utilizing the trained backbone extraction network, the first classification model, the second classification model and the first skeleton sequence to be identified. Compared with the prior art, the method has the advantages of reducing the training sample size level and the like.

Description

Technical Field

[0001] The present invention relates to the field of group behavior recognition, and in particular to a group behavior recognition method based on a 3D human skeleton. Background Art

[0002] Understanding human group behavior has become an important topic in spatiotemporal modeling and is widely used in downstream tasks such as monitoring systems to detect abnormal behavior and autonomous driving systems to analyze pedestrian intentions.

[0003] Although there are some existing technologies that perform behavior recognition by identifying key points of the human skeleton, for example, Chinese patent CN114373225A discloses a method and system for behavior recognition based on the human skeleton. The method includes: obtaining a temporally continuous image sequence; using a posture estimation method to obtain a skeleton sequence; sending the skeleton sequence into a first model and a second model respectively, the first model is a temporal convolutional network for extracting the temporal features of the skeleton sequence, and the second model includes a view adaptation network and a spatial graph convolutional network for extracting the spatial features of the skeleton sequence; fusing the temporal features and the spatial features to output human behavior actions.

[0004] However, the above methods primarily target individual action recognition, not group action recognition. While some researchers have attempted to adapt skeleton keypoint-based action recognition methods to group action recognition, they have also provided some improved technical solutions. For example, in "COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modality," published by Honglu Zhou et al. at the 2022 European Conference on Computer Vision, some existing techniques use a 2D human skeleton sequence as input and propose a multi-scale Transformer-based architecture. This architecture performs attention-based reasoning on tokens at each scale and combines them to recognize group activities. For the input skeleton sequence, the algorithm first extracts single-person-scale features using a self-attention mechanism. These features are then combined pairwise, and further self-attention is used to extract multi-person interaction features and group features. Finally, features at different scales are used for action recognition, and a weighted synthesis is performed to obtain the final result. Furthermore, the algorithm clusters intermediate-scale features to optimize the representation and maintains consistency in cluster assignment across scales, resulting in more accurate final recognition results.

[0005] However, on the one hand, it only supports 2D skeleton sequences as input, and 3D skeleton sequences require additional refinement and retraining. Furthermore, the algorithm is only suitable for classifying cropped group behavior segments, and cannot detect and identify corresponding group behaviors in uncropped segments, resulting in relatively limited application scenarios. Furthermore, the algorithm's extraction of individual-scale features is imperfect. Trained solely on a small amount of group behavior data using a shallow self-attention mechanism, it cannot leverage the vast amount of individual behavior understanding data. This limits the ability of pairwise combinations of individual-scale features to represent group features. Summary of the Invention

[0006] The purpose of the present invention is to provide a group behavior recognition method based on 3D human skeleton in order to solve the above-mentioned defects of the prior art.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] A group behavior recognition method based on 3D human skeleton, comprising:

[0009] Step S1: Obtain training set data, the training set data includes:

[0010] The first skeleton sequence contains the three-dimensional coordinate information of all skeleton key points of N independent individuals in T frames.

[0011] Individual behavior labeling results of each independent individual, as well as group behavior labeling results;

[0012] Step S2: extracting the first eigenvector of each individual based on the first skeleton sequence in combination with the backbone extraction network, inputting the first eigenvector into the trained first classification model to obtain a first individual classification result, and performing verification based on the first individual classification result and the corresponding individual behavior labeling result. If the verification passes, proceed to step S3;

[0013] Step S3: concatenate the first feature vectors of each independent individual to obtain the first group feature, and use the self-attention-cross attention neural network to obtain the second group feature;

[0014] Step S4: extracting target features based on the second group features, classifying the target features and inputting them into the trained second classification model to obtain the first group behavior classification results, and modifying the model parameters of the backbone extraction network based on the difference between the first group behavior classification results and the corresponding group behavior labeling results to complete the training of the backbone extraction network;

[0015] Step S5: Using the trained backbone extraction network, the first classification model and the second classification model, the first skeleton sequence to be identified is used for group behavior recognition.

[0016] The step S2 comprises:

[0017] Step S2-1: sampling the first skeleton sequence using the first sliding window to obtain multiple second skeleton sequences of each independent individual;

[0018] Step S2-2: Using the backbone extraction network, perform feature extraction on each second skeleton sequence to obtain corresponding segment features;

[0019] Step S2-3: averaging all segment features corresponding to the same independent individual to obtain a first feature vector corresponding to each independent individual;

[0020] Step S2-4: Input the first feature vector into the trained first classification model to obtain the first individual classification result, and perform verification based on the first individual classification result and the corresponding individual behavior labeling result. If the verification passes, execute step S3.

[0021] In step S2, if the result of the verification based on the first individual classification result and the corresponding individual behavior labeling result is failure to pass the verification, the type of the backbone extraction network is changed or the parameters of the backbone extraction network are modified.

[0022] The step S3 comprises:

[0023] Step S3-1: concatenate the first feature vectors of each independent individual to obtain the first group feature;

[0024] Step S3-2: Flatten the first group feature in the time dimension and the space dimension respectively to obtain a group time feature focusing on time and a group space feature focusing on space;

[0025] Step S3-3: Use the self-attention-cross attention neural network to process the group time features and group space features to obtain the second group features.

[0026] The mathematical expression of the self-attention-cross attention neural network is:

[0027] F'=atten(atten(F temp ,F temp ),atten(F spatio ,F spatio ))

[0028] +atten(atten(F spatio ,F spatio ),atten(F temp ,F temp ))

[0029] Among them: F' is the second group characteristic, F temp is the group spatial characteristic, F tempis the temporal characteristic of the group.

[0030] The step S4 comprises:

[0031] Step S4-1: Perform a maximum pooling operation on the individual dimensions of the second group feature to obtain the integrated third group feature F final ;

[0032] Step S4-2: Further analyze the third group feature F through the temporal convolutional network final Optimize and obtain a two-dimensional group feature matrix;

[0033] Step S4-3: Obtain the first K peaks in the two-dimensional group feature matrix, and use the obtained peaks to calculate the first K peaks in the third group feature F. final Extract the target features;

[0034] Step S4-4: Input the target feature into the trained second classification model to obtain the first group behavior classification result, and modify the model parameters of the backbone extraction network based on the difference between the first group behavior classification result and the corresponding group behavior labeling result to complete the training of the backbone extraction network.

[0035] The step S4-3 specifically includes:

[0036] Step S4-3-1: Get K peaks of the two-dimensional group feature matrix, corresponding to K groups of subscript sets {(i k ,j k )};

[0037] Step S4-3-2: Select the third group feature F according to the subscript set final Get the K groups of feature sets f with the corresponding subscripts final , corresponding to K candidate group behaviors, the target features are obtained.

[0038] The process of the first group behavior classification result in step S4-4 includes:

[0039] Step S4-4-1: The subscript corresponding to the subscript of the time dimension is set as the midpoint of the group behavior time period;

[0040] Step S4-4-2: Use two sets of multi-layer perceptrons to process K sets of feature sets f final Regress the category, time center offset correction, and time span of each group behavior separately;

[0041] Step S4-3-3: Integrate the group behavior category, the center point of the behavior time period, and the time span as the final recognition result output.

[0042] The backbone extraction network contains 21 layers of graph convolutional neural networks. Each layer of the graph convolutional neural network gradually refines the features and passes them to the next layer. The feature map is constructed with the key points of the human skeleton as the nodes of the graph, the real skeleton connection relationship and the previous and next frame connections as the edges of the graph. All graph convolutional network layers share the feature map.

[0043] A group behavior recognition device based on a 3D human skeleton includes a memory, a processor, and a program stored in the memory. When the processor executes the program, the method described above is implemented.

[0044] A storage medium stores a program, which implements the above method when executed.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. The individual skeleton sequence is the second skeleton sequence extracted from the first skeleton sequence, so that the input data of individual behavior annotation and group behavior annotation are the same. The backbone extraction network is verified in combination with the mature first classification model to ensure that there is a sufficient amount of individual behavior annotation data to complete the selection or training of the backbone extraction network. On this basis, the first group feature is obtained by averaging and splicing, and the selected or trained backbone extraction network is fine-tuned in conjunction with the second classification model. On the one hand, the magnitude requirement of the annotation data for group behavior feature extraction can be reduced, and the graph convolutional neural network can be pre-trained on single-person behavior understanding data and migrated to group behavior understanding to improve the accuracy of the algorithm. On the other hand, relevant information can be mined more deeply, thereby improving the accuracy of group behavior recognition.

[0047] 2. Flatten the first group features along the temporal and spatial dimensions, respectively, to obtain a temporal group feature focused on time and a spatial group feature focused on space. These features are then processed using an attention-cross-attention neural network to obtain a second group feature. Compared to simple nonlinear computation, this method achieves hierarchical, multi-granular feature interactions. Furthermore, the attention mechanism is more suitable for improving model scalability. By superimposing the networks designed in this paper, it can more effectively process complex and rich data.

[0048] 3. The topK peak method is used to obtain the target features, which can flexibly handle the situation where a piece of data contains multiple items to be detected, thereby solving the problem that previous methods cannot handle the detection of uncropped fragments and identify the corresponding group behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 Schematic diagram of the main steps of the method of the present invention. DETAILED DESCRIPTION

[0050] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0051] A group behavior recognition method based on 3D human skeleton, such as Figure 1 As shown, including:

[0052] Step S1: Obtain training set data, which includes:

[0053] The first skeleton sequence contains the three-dimensional coordinate information of all skeleton key points of N independent individuals in T frames.

[0054] Individual behavior labeling results of each independent individual, as well as group behavior labeling results;

[0055] Among them, the independent individual behavior labeling results and the group behavior labeling results come from the same first skeleton sequence. Specifically, the first skeleton sequence has T frames, N independent individuals and C-dimensional skeleton coordinate features. The value of C is 3S, and S is the number of skeleton key points, that is, each skeleton key point includes x, y, and z coordinates.

[0056] Step S2: extracting the first eigenvector of each individual based on the first skeleton sequence in combination with the backbone extraction network, inputting the first eigenvector into the trained first classification model to obtain a first individual classification result, and performing verification based on the first individual classification result and the corresponding individual behavior labeling result. If the verification passes, proceed to step S3;

[0057] Specifically, step S2 includes:

[0058] Step S2-1: Use the first sliding window to sample the first skeleton sequence to obtain multiple second skeleton sequences of each independent individual, where the first sliding window W={W l ,W s}, whose size is W l , the step size is W s .

[0059] Step S2-2: Using the backbone extraction network, perform feature extraction on each second skeleton sequence to obtain corresponding segment features;

[0060] The backbone extraction network can be a mature solution on the market or built independently. The backbone extraction network processes the features of individual fragments separately. In this embodiment, the backbone extraction network includes 9 layers of graph convolutional neural networks. Each layer of graph convolutional neural network gradually refines the features and passes them to the next layer. The graph is constructed with the key points of the human skeleton as the nodes of the graph, the real skeleton connection relationship and the previous and next frame connections as the edges of the graph. This structure is shared by all graph convolutional network layers. The processing of single-person feature fragments can be expressed by the following formula:

[0061]

[0062] in: is the fragment feature, is the second skeleton sequence, and B(·) is the processing operation of the backbone extraction network.

[0063] The individual skeleton sequence is a second skeleton sequence extracted from the first skeleton sequence, so that the input data of the individual behavior annotation and the group behavior annotation are homologous.

[0064] Step S2-3: averaging all segment features corresponding to the same independent individual to obtain a first feature vector corresponding to each independent individual;

[0065] Step S2-4: Input the first feature vector into the trained first classification model to obtain the first individual classification result, and perform verification based on the first individual classification result and the corresponding individual behavior labeling result. If the verification passes, execute step S3.

[0066] In addition, if the result of the verification based on the first individual classification result and the corresponding individual behavior labeling result is that the verification fails, the type of the backbone extraction network is changed or the parameters of the backbone extraction network are modified.

[0067] Step S3: concatenate the first feature vectors of each independent individual to obtain the first group feature, and use the self-attention-cross attention neural network to obtain the second group feature, including:

[0068] Step S3-1: Concatenate the first eigenvectors of the independent individuals to obtain the first group feature, and combine the averaging process. Specifically:

[0069]

[0070] Where: F is the first group feature, Agg represents the average operation in multiple sliding windows, N w Represents the number of segments in each individual sequence. The group feature has T' frames, N independent individuals and C' dimensional semantic features.

[0071] Different from the existing method of directly modeling group interaction relationship characteristics from the single-person skeleton, this application adopts a more refined design of the single-person feature extraction backbone network B to first obtain the single-person skeleton features. In terms of algorithm implementation, this application can use the single-person skeleton features to pre-train the feature extraction network on the single-person behavior understanding dataset, obtain the initialization weights suitable for feature extraction, and then fine-tune it on the target multi-person behavior understanding dataset to obtain ultimately better performance.

[0072] Step S3-2: Flatten the first group feature in the time dimension and the space dimension respectively to obtain a group time feature focusing on time and a group space feature focusing on space;

[0073] Step S3-3: Use the self-attention-cross attention neural network to process the group time features and group space features to obtain the second group features.

[0074] The mathematical expression of the self-attention-cross attention neural network is:

[0075] F'=atten(atten(F temp ,F temp ),atten(F spatio ,F spatio ))

[0076] +atten(atten(F spatio ,F spatio ),atten(F temp ,F temp ))

[0077] Among them: F' is the second group characteristic, F temp is the group spatial characteristic, F temp is the temporal characteristic of the group.

[0078] Step S4: extracting target features based on the second group features, classifying the target features and inputting them into the trained second classification model to obtain the first group behavior classification results, and modifying the model parameters of the backbone extraction network based on the difference between the first group behavior classification results and the corresponding group behavior labeling results to complete the training of the backbone extraction network, including:

[0079] Step S4-1: Perform a maximum pooling operation on the individual dimensions of the second group feature to obtain the integrated third group feature F final ;

[0080] Step S4-2: Further analyze the third group feature F through the temporal convolutional network final Optimize and obtain a two-dimensional group feature matrix;

[0081] Step S4-3: Obtain the first K peaks in the two-dimensional group feature matrix, and use the obtained peaks to calculate the first K peaks in the third group feature F. final Extract the target features;

[0082] Step S4-4: Input the target feature into the trained second classification model to obtain the first group behavior classification result, and modify the model parameters of the backbone extraction network based on the difference between the first group behavior classification result and the corresponding group behavior labeling result to complete the training of the backbone extraction network.

[0083] In some embodiments, step S4-3 specifically includes:

[0084] Step S4-3-1: Get K peaks of the two-dimensional group feature matrix, corresponding to K groups of subscript sets {(i k ,j k )};

[0085] Step S4-3-2: Select the third group feature F according to the subscript set final Get the K groups of feature sets f with the corresponding subscripts final , corresponding to K candidate group behaviors, the target features are obtained.

[0086] The process of the first group behavior classification result in step S4-4 includes:

[0087] Step S4-4-1: The subscript corresponding to the subscript of the time dimension is set as the midpoint of the group behavior time period;

[0088] Step S4-4-2: Use two sets of multi-layer perceptrons to process K sets of feature sets f final Regress the category, time center offset correction, and time span of each group behavior separately;

[0089] Step S4-3-3: Integrate the group behavior category, the center point of the behavior time period, and the time span as the final recognition result output.

[0090] Step S5: Using the trained backbone extraction network, the first classification model and the second classification model, the first skeleton sequence to be identified is used for group behavior recognition.

[0091] In order to verify the effectiveness of this application, an ablation experiment was conducted to test the effectiveness of the self-attention-cross-attention spatiotemporal modeling: while keeping the group feature acquisition method unchanged, the spatiotemporal modeling method proposed by Jianchao Wu et al. in "Learning actorrelation graphs for group activity recognition" and Kirill Gavrilyuk et al. in "Actor-transformers for group activity recognition" was adopted. The algorithm designed in this application achieved a performance improvement of 24.52% and 9% in the mAP indicator, indicating the effectiveness of the module designed in this application.

[0092] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

Claims

1. A group behavior recognition method based on 3D human skeleton, characterized in that: include: Step S1: Obtain training set data, the training set data includes: The first skeleton sequence contains the three-dimensional coordinate information of all skeleton key points of N independent individuals in T frames. Individual behavior labeling results of each independent individual, as well as group behavior labeling results; Step S2: extracting the first eigenvector of each individual based on the first skeleton sequence in combination with the backbone extraction network, inputting the first eigenvector into the trained first classification model to obtain a first individual classification result, and performing verification based on the first individual classification result and the corresponding individual behavior labeling result. If the verification passes, proceed to step S3; Step S3: concatenate the first feature vectors of each independent individual to obtain the first group feature, and use the self-attention-cross attention neural network to obtain the second group feature; Step S4: extracting target features based on the second group features, classifying the target features and inputting them into the trained second classification model to obtain the first group behavior classification results, and modifying the model parameters of the backbone extraction network based on the difference between the first group behavior classification results and the corresponding group behavior labeling results to complete the training of the backbone extraction network; Step S5: Using the trained backbone extraction network, the first classification model and the second classification model, the first skeleton sequence to be identified is used for group behavior recognition.

2. The method for group behavior recognition based on 3D human skeleton according to claim 1, characterized in that: The step S2 comprises: Step S2-1: sampling the first skeleton sequence using the first sliding window to obtain multiple second skeleton sequences of each independent individual; Step S2-2: Using the backbone extraction network, perform feature extraction on each second skeleton sequence to obtain corresponding segment features; Step S2-3: averaging all segment features corresponding to the same independent individual to obtain a first feature vector corresponding to each independent individual; Step S2-4: Input the first feature vector into the trained first classification model to obtain the first individual classification result, and perform verification based on the first individual classification result and the corresponding individual behavior labeling result. If the verification passes, execute step S3.

3. The method for group behavior recognition based on 3D human skeleton according to claim 1, characterized in that: In step S2, if the result of the verification based on the first individual classification result and the corresponding individual behavior labeling result is failure to pass the verification, the type of the backbone extraction network is changed or the parameters of the backbone extraction network are modified.

4. The method for group behavior recognition based on 3D human skeleton according to claim 1, characterized in that: The step S3 comprises: Step S3-1: concatenate the first feature vectors of each independent individual to obtain the first group feature; Step S3-2: Flatten the first group feature in the time dimension and the space dimension respectively to obtain a group time feature focusing on time and a group space feature focusing on space; Step S3-3: Use the self-attention-cross attention neural network to process the group time features and group space features to obtain the second group features.

5. The method for group behavior recognition based on 3D human skeleton according to claim 4, characterized in that: The mathematical expression of the self-attention-cross attention neural network is: F‘=atten(atten(F temp ,F temp ),atten(F spatio ,F spatio ))+atten(atten(F spatio ,F spatio ),atten(F temp ,F temp )) Among them: F' is the second group characteristic, F temp is the group spatial characteristic, F temp is the temporal characteristic of the group.

6. The method for group behavior recognition based on 3D human skeleton according to claim 1, characterized in that: The step S4 comprises: Step S4-1: Perform a maximum pooling operation on the individual dimensions of the second group feature to obtain the integrated third group feature F final ; Step S4-2: Further analyze the third group feature F through the temporal convolutional network final Optimize and obtain a two-dimensional group feature matrix; Step S4-3: Obtain the first K peaks in the two-dimensional group feature matrix, and use the obtained peaks to calculate the first K peaks in the third group feature F. final Extract the target features; Step S4-4: Input the target feature into the trained second classification model to obtain the first group behavior classification result, and modify the model parameters of the backbone extraction network based on the difference between the first group behavior classification result and the corresponding group behavior labeling result to complete the training of the backbone extraction network.

7. The method for group behavior recognition based on 3D human skeleton according to claim 6, characterized in that: The step S4-3 specifically includes: Step S4-3-1: Get K peaks of the two-dimensional group feature matrix, corresponding to K groups of subscript sets {(i k ,j k )}; Step S4-3-2: Select the third group feature F according to the subscript set final Get the K groups of feature sets f with the corresponding subscripts final , corresponding to K candidate group behaviors, the target features are obtained. The process of the first group behavior classification result in step S4-4 includes: Step S4-4-1: The subscript corresponding to the subscript of the time dimension is set as the midpoint of the group behavior time period; Step S4-4-2: Use two sets of multi-layer perceptrons to process K sets of feature sets f final Regress the category, time center offset correction, and time span of each group behavior separately; Step S4-3-3: Integrate the group behavior category, the center point of the behavior time period, and the time span as the final recognition result output.

8. The method for group behavior recognition based on 3D human skeleton according to claim 1, characterized in that: The backbone extraction network contains 21 layers of graph convolutional neural networks. Each layer of the graph convolutional neural network gradually refines the features and passes them to the next layer. The feature map is constructed with the key points of the human skeleton as the nodes of the graph, the real skeleton connection relationship and the previous and next frame connections as the edges of the graph. All graph convolutional network layers share the feature map.

9. A group behavior recognition device based on a 3D human skeleton, comprising a memory, a processor, and a program stored in the memory, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 8 is implemented.

10. A storage medium having a program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images