Group behavior recognition method and device based on pyramid squeeze attention module

Abstract

The application provides a group behavior recognition method and device based on a pyramid extrusion attention module, the method comprising: extracting individual appearance features of each participant in each frame in a video segment; extracting key point features of each participant in each frame in the video segment; inputting the individual appearance features and the key point features of each participant into a trained group pyramid extrusion attention model, the pyramid extrusion attention model assigning attention weights to the individual appearance features of each participant, and calculating the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; constructing a pyramid group appearance feature map based on the weighted fusion features of each participant; and inputting the pyramid group appearance feature map into a trained graph neural network model to recognize the group behavior. The method maintains low computing and storage costs, integrates local and non-local feature information, and realizes accurate image super-resolution reconstruction.

Description

Technical Field

[0001] The present invention relates to the field of video analysis technology, and in particular to a group behavior recognition method and device based on a pyramid squeeze attention module. Background Art

[0002] Group behavior recognition involves identifying the collective behavior of multiple individuals within a scene, all driven by a specific goal. A prior art group behavior recognition method and system based on dynamic dual-stream self-attention (CN118506437A) uses self-attention or graph convolutional neural networks to aggregate interactions between participants into group interaction features. After extracting the participants' apparent features, these features at different time frames are stacked into a spatiotemporal graph. The spatiotemporal graph then analyzes the temporal weights of video frames within a video clip of group behavior to identify group behavior.

[0003] However, the disadvantage of existing technologies is that ordinary graph convolution for group behavior recognition only considers channel attention and ignores spatial attention. Among them, traditional BAM and CBAM consider channel attention and spatial attention, but still have two most important shortcomings: (1) they do not capture spatial information of different scales to enrich the feature space. (2) Spatial attention only considers information in local areas and cannot establish long-distance dependencies. In addition, the subsequent PyConv, Res2Net and HS-ResNet are all used to address these two shortcomings of CBAM, but the computational complexity is too large. Summary of the Invention

[0004] To solve the above technical problems, the present invention proposes a group behavior recognition method and device based on a pyramid squeeze attention module.

[0005] According to a first aspect of the present invention, a method for group behavior recognition based on a pyramid squeezed attention module is provided, the method comprising the following steps:

[0006] Step S1: extracting individual appearance features of each participant in each frame of the video clip based on the trained two-dimensional convolutional neural network; using the trained HRNet network to extract key point features of each participant in each frame of the video clip, wherein the key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip;

[0007] Step S2: Inputting the individual appearance features and key point features of each participant into the trained group pyramid squeezed attention model, the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, and calculates the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; constructing a pyramid group appearance feature map based on the weighted fusion features of each participant;

[0008] Step S3: Input the pyramid group appearance feature map into the trained graph neural network model to identify group behavior.

[0009] Preferably, the group pyramid squeezing attention model includes a bilinear interpolation module and a first branch, a second branch, and a third branch connected in parallel to the bilinear interpolation module, wherein the first branch includes a first pyramid segmentation attention module and a multi-layer perceptron connected in sequence; the second branch includes a first scaling dot product module; the third branch includes a second pyramid segmentation attention module, a first graph convolution layer, a second graph convolution layer, a third graph convolution layer connected in sequence, and a first sub-branch and a second sub-branch connected in parallel to the third graph convolution layer, wherein the first sub-branch includes a similarity calculation module and a multi-layer perceptron module connected in sequence, and the second sub-branch includes a second scaling dot product module.

[0010] Preferably, the individual appearance features of each participant are input into the bilinear interpolation module, and the bilinear interpolation module uses a bilinear interpolation algorithm to interpolate the individual appearance features of each participant to generate a feature Papp', and the feature Papp' is input into the first branch. The first pyramid segmentation attention module performs a fusion feature operation on the feature Papp' to generate a feature Papp", and the multi-layer perceptron reduces the dimension of the feature Papp" to generate a feature Vapp; the feature Papp' is input into the second branch, and the first scaling dot product module establishes a first graph structure Rapp based on the feature Papp'. The first graph structure Rapp uses the participants as nodes to reflect the first connection relationship between the participants; the feature Vapp is fused with the first graph structure Rapp to generate a first pyramid group appearance feature map; the feature Papp' is input into the third branch, and the second pyramid segmentation attention module performs a fusion feature operation on the feature Papp', and the fused feature is input into the first graph convolution layer, and then the first convolution layer is added. The result is input into the second graph convolution layer to obtain the second convolution result, and the second convolution result is input into the third graph convolution layer to generate a third convolution result; the third convolution result is spliced ​​with the key point features of each participant to generate a key point splicing feature, and the key point splicing feature is input into the first sub-branch and the second sub-branch respectively. The similarity calculation module of the first sub-branch calculates the coordinate feature vector of the participant, and the generated coordinate feature vector Pkey" is input into the multi-layer perceptron module. The multi-layer perceptron module includes an input layer, a hidden layer and an output layer. The multi-layer perceptron module converts the coordinate feature vector Pkey" into a feature Vkey; the second sub-branch inputs the key point splicing feature into the second scaling dot product module, and the second scaling dot product module establishes a second graph structure Rkey based on the key point splicing feature. The second graph structure Rkey uses the participants as nodes to reflect the second connection relationship between the participants; the feature Vkey is fused with the second graph structure Rkey to generate a second pyramid group key point feature map.

[0011] Preferably, both the first pyramid segmentation attention module and the second pyramid segmentation attention module calculate the participant's OKS feature:

[0012]

[0013] in, Denotes the OKS feature of the jth key point of the nth participant at time t, d n,j,t represents the Euclidean distance between the coordinate of the jth key point of the nth participant at time t and the coordinate of the jth key point at time t+1, s n,t represents the area of ​​the labeled box of the nth participant at time t, δ j Represents the normalization factor of the jth key point obtained by statistics.

[0014] Preferably, the loss function used is:

[0015]

[0016] in, is the loss function, is the fused group behavior loss function, G represents the true label of the group behavior, Represents the group behavior characteristics after global pooling; are the group behavior loss function based on the individual appearance characteristics of the participants and the group behavior cross loss function based on the key point characteristics of the participants; λ, λ1, and λ2 are weight coefficients.

[0017] According to a second aspect of the present invention, a group behavior recognition device based on a pyramid squeezed attention module is provided, the device comprising:

[0018] Initialization module: configured to extract individual appearance features for each participant in each frame of the video clip based on a trained two-dimensional convolutional neural network; and to extract key point features for each participant in each frame of the video clip using a trained HRNet network. The key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip.

[0019] A fusion module is configured to input the individual appearance features and key point features of each participant into a trained group pyramid squeezed attention model, wherein the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, calculates weighted fusion features of the individual appearance features of each participant and the key point features of the participant, and constructs a pyramid group appearance feature map based on the weighted fusion features of each participant;

[0020] Identification module: configured to input the pyramid group appearance feature map into the trained graph neural network model to identify group behavior.

[0021] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0022] A processor, which is used to execute multiple instructions;

[0023] A memory for storing a plurality of instructions;

[0024] The plurality of instructions are used to be stored by the memory and loaded and executed by the processor to implement the method as described above.

[0025] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the aforementioned method.

[0026] The present invention has the following technical effects:

[0027] This paper introduces a highly efficient Pyramid Squeeze Attention (PSA) module, significantly improving the performance of deep convolutional neural networks in group action recognition visual tasks. By replacing the 3x3 convolution in the bottleneck block of ResNet with a PSA module, GPSANet provides richer multi-scale feature representations and more efficient channel-space interactions without significantly increasing the computational burden.

[0028] This paper uses a pyramid squeezed attention module to integrate the information of the input feature maps using a multi-scale pyramid convolutional structure and effectively extract spatial information of different scales from the feature maps of each channel by squeezing the channel dimension of the input tensor. In addition, by extracting the channel attention weights of the multi-scale feature maps and recalibrating the attention weights of the corresponding channels using a softmax operation, long-range channel dependencies are established.

[0029] This paper constructs a Group Pyramid Squeezed Attention (GPSA) block, which can be easily added to existing backbone networks as a plug-and-play component and significantly improves model performance. It provides stronger multi-scale representation capabilities for group action recognition tasks and can adaptively recalibrate channel attention weights across dimensions.

[0030] The present invention is more efficient in memory usage, effectively aggregates features through dynamic spatial modulation, improves reconstruction performance, while maintaining low computational and storage costs, effectively integrates local and non-local feature information, and achieves more accurate image super-resolution reconstruction.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings, which constitute part of the present invention, are used to provide a further understanding of the present invention. The present invention is described with the following accompanying drawings. In the accompanying drawings:

[0033] Figure 1 This is a flow chart of a group behavior recognition method based on a pyramid squeeze attention module according to an embodiment of the present invention;

[0034] Figure 2 This is a structural block diagram of a group behavior recognition device based on a pyramid squeeze attention module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] First combine Figure 1 The present invention describes a method for group behavior recognition based on a pyramid squeeze attention module according to an embodiment of the present invention. Figure 1 As shown, the method includes the following steps:

[0036] Step S1: extracting individual appearance features of each participant in each frame of the video clip based on the trained two-dimensional convolutional neural network; using the trained HRNet network to extract key point features of each participant in each frame of the video clip, wherein the key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip;

[0037] Step S2: Inputting the individual appearance features and key point features of each participant into the trained group pyramid squeezed attention model, the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, and calculates the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; constructing a pyramid group appearance feature map based on the weighted fusion features of each participant;

[0038] Step S3: Input the pyramid group appearance feature map into the trained graph neural network model to identify group behavior.

[0039] Step S1: extracting individual appearance features of each participant in each frame of the video clip based on a trained two-dimensional convolutional neural network; extracting key point features of each participant in each frame of the video clip using a trained HRNet network, wherein the key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip, wherein:

[0040] The video clip is a sparsely sampled video clip For each video frame in the video clip, a two-dimensional convolutional neural network is used to extract features to obtain a feature map of the group behavior scene corresponding to the video frame. wherein T is the time length of the video clip, WxH is the size of the video frame, and oWx oH is the size of the feature map of the crowd behavior scene. The RoIAlgin algorithm is used to extract the individual behavior appearance features of each participant in the feature map of the crowd behavior scene. In the present application, the RoIAlgin algorithm extracts the individual behavior appearance features of the participants in the crowd behavior based on the participant bounding box coordinates in the data set, and the individual behavior appearance features of each participant are a feature map. The RoIAlgin algorithm also aligns the extracted features, so that the individual behavior appearance feature map of each participant is KxK in size. Then, the individual behavior appearance features of each participant are embedded using the FFLN network (Fourier Features Let Network, FFLN) to obtain the individual appearance features of each participant wherein N is the number of participants in the crowd behavior scene, and d represents the output dimension of the feature embedding operation. The key point coordinates of each participant are extracted by the HRNet network to obtain the participant key point features J represents the number of key points of the participant. The HRNet network is pre-trained using the AIC data set, and the key point coordinates of the participant are estimated based on the bounding box. For each participant, individual appearance features and key point features are obtained. That is, a feature set P represented by the individual features of the participants in the crowd behavior can be constructed:

[0041] P={P app ,P key}

[0042] The step S2: inputting the individual appearance features and key point features of each participant into the trained crowd pyramid squeeze attention model, the pyramid squeeze attention model assigns attention weights to the individual appearance features of each participant, and calculates the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; based on the weighted fusion features of each participant, a pyramid crowd appearance feature map is constructed, wherein:

[0043] The crowd pyramid squeeze attention model includes a bilinear interpolation module, and a first branch, a second branch and a third branch connected in parallel with the bilinear interpolation module, the first branch includes a first pyramid segmentation attention module and a multi-layer perception machine connected in sequence; the second branch includes a first scaling dot product module; the third branch includes a second pyramid segmentation attention module, a first graph convolution layer, a second graph convolution layer, a third graph convolution layer and a first sub-branch and a second sub-branch connected in parallel with the third graph convolution layer in sequence, the first sub-branch includes a similarity calculation module and a multi-layer perception machine module connected in sequence, and the second sub-branch includes a second scaling dot product module;

[0044] The individual appearance features of each participant are input into the bilinear interpolation module, and the bilinear interpolation module uses the bilinear interpolation algorithm to interpolate the individual appearance features of each participant to generate feature Papp', and the feature Papp' is input into the first branch. The first pyramid segmentation attention module performs a fusion feature operation on the feature Papp' to generate feature Papp", and the multi-layer perceptron reduces the dimension of the feature Papp" to generate feature Vapp; the feature Papp' is input into the second branch, and the first scaling dot product module establishes a first graph structure Rapp based on the feature Papp'. The first graph structure Rapp uses the participants as nodes to reflect the first connection relationship between the participants; the feature Vapp is fused with the first graph structure Rapp to generate a first pyramid group appearance feature map, and the first pyramid group appearance feature map reflects the group behavior modeling process based on the graph method; the feature Papp' is input into the third branch, and the second pyramid segmentation attention module performs a fusion feature operation on the feature Papp', and the fused features are input into the first graph convolution layer, and then The first convolution result is input into the second graph convolution layer to obtain a second convolution result, and the second convolution result is input into the third graph convolution layer to generate a third convolution result; the third convolution result is spliced ​​with the key point features of each participant to generate a key point splicing feature, and the key point splicing feature is input into the first sub-branch and the second sub-branch respectively. The similarity calculation module of the first sub-branch calculates the coordinate feature vector of the participant, and the generated coordinate feature vector Pkey" is input into the multi-layer perceptron module. The multi-layer perceptron module includes an input layer, a hidden layer and an output layer. The multi-layer perceptron module converts the coordinate feature vector Pkey" into a feature Vkey; the second sub-branch inputs the key point splicing feature into the second scaled dot product module, and the second scaled dot product module establishes a second graph structure Rkey based on the key point splicing feature. The second graph structure Rkey uses the participants as nodes to reflect the second connection relationship between the participants; the feature Vkey is fused with the second graph structure Rkey to generate a second pyramid group key point feature map, and the second pyramid group key point feature map reflects the group appearance.

[0045] The pyramid segmentation attention (PSA) module in the present invention is a conventional pyramid segmentation attention (PSA) module in the art.

[0046] The Pyramid Squeeze Attention (PSA) module is an efficient deep learning module designed to improve the performance of convolutional neural networks in various computer vision tasks. By introducing multi-scale feature extraction and channel attention mechanisms, PSA effectively integrates information from the input feature map, thereby enhancing the model's representation capabilities.

[0047] PSA processes input features using a multi-scale convolutional architecture. It divides the input feature map into multiple parts along the channel dimension, each of which is convolved with a kernel of a different size. This design allows the module to extract features from different scales, enabling the network to capture more contextual information. Within the PSA module, a Squeeze-and-Excitation (SE) architecture is applied to each feature map at each scale. This architecture first compresses the spatial dimensions of the feature map to 1x1 using an adaptive average pooling layer, thereby capturing global channel information. Next, a fully connected layer (or convolutional layer) is used to adjust the number of channels, a nonlinear transformation is performed using the ReLU activation function, and finally, a sigmoid function is used to output channel attention weights. These weights are used to dynamically adjust the feature responses of each channel, thereby modeling inter-channel dependencies. During the forward propagation of PSA, features at different scales are first convolved to obtain feature representations at each scale. These features are then weighted using the channel attention weights calculated by the SE module, ultimately merging the multi-scale features into a unified feature representation through summation. This process not only enhances the response of important features but also suppresses irrelevant information, thereby improving the final feature expression.

[0048] For example, the dimension of the input feature map is (B, C, H, W), where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map. Channel splitting: The input feature map is split into S parts in the channel dimension, allowing each part to be processed independently. Convolutional layer: Each channel part is extracted through a convolutional layer of different sizes. The size is determined by 2(i+1)+1, where i is the index of the current scale. SE module: The corresponding SE module is applied to the feature map of each scale to obtain the channel attention weight. Feature merging: The weighted multi-scale features are summed to generate the final output feature map.

[0049] Experiments on multiple standard datasets demonstrate that models using the PSA module achieve significant improvements in accuracy and detection performance. Compared to other complex attention mechanisms, PSA significantly reduces computational overhead while maintaining high performance. The PSA module can be easily integrated into existing convolutional networks, providing a plug-and-play component.

[0050] In this paper, the PSA module is used to provide attention weighting for the key point features of the participants, enhancing the focus on the key participant's actions. This can be achieved through a similar attention mechanism, making the key point features more focused on meaningful dynamic behaviors.

[0051] To reduce the noise caused by the pose estimation network in labeling key points, the OKS feature of the participant is calculated according to the OKS (Object Key-point Similarity) of the target key point That is, the first pyramid segmentation attention module and the second pyramid segmentation attention module both calculate the OKS feature of the participant:

[0052]

[0053] Wherein, represents the OKS feature of the jth key point of the nth participant at time t, d n,j,t represents the Euclidean distance between the tth key point of the nth participant at time t and the jth key point of the nth participant at time t+1, s n,t represents the area of the tth key point of the nth participant at time t, δ j represents the normalized factor of the jth key point obtained by statistics, which reflects the uncertainty of manually labeling this type of key point.

[0054] Then, in order to reduce the influence of the network on the estimation deviation of the key point coordinates, the key point coordinates and the difference calculation results of the coordinates are further regularized, and the results are respectively denoted as and

[0055]

[0056] Wherein, k mean represents the average value of the key point in the T time period, k std-mean represents the average value of the key point standard deviation in the T time period. In addition, in order to further extract the space-time features of the key point coordinates, the key point coordinates are embedded along the video space and the video time respectively, and the results are respectively denoted as and

[0057] By splicing the above calculation results, the key point coordinate feature vector is obtained Wherein d k represents the feature dimension of the semantic feature. And P′ key in the above is spliced to obtain the key point feature P″ key , which is input into a multi-layer perception layer to obtain the embedded participant key point feature, which is used as the graph node V key of the participant behavior interaction graph G key .

[0058] Step S3: input the pyramid group appearance feature map into the trained graph neural network model to identify the group behavior, including:

[0059] The first pyramid group appearance feature map G app And the second pyramid group key point feature map G key Input the trained graph neural network model respectively to obtain the first group behavior feature G′ app and the second group behavior characteristic G′ key ; Fusion of the first group behavior feature G' app and the second group behavior characteristic G′ key , obtaining group behavior fusion characteristics; and determining group behavior based on the group behavior fusion characteristics.

[0060] This paper explores an early feature fusion method based on addition and a late feature fusion method based on addition. The results show that the late feature fusion method based on addition achieves better group behavior recognition results.

[0061] Furthermore, the present invention uses a cross-entropy loss function to supervise the model update weights. The entire group behavior recognition process can be implemented in an end-to-end manner, with the cross-entropy loss function supervising the model update.

[0062] The loss function used is:

[0063]

[0064] in, is the loss function, is the fused group behavior loss function, G represents the true label of the group behavior, Represents the group behavior characteristics after global pooling; are the group behavior loss function based on the individual appearance characteristics of the participants and the group behavior cross loss function based on the key point characteristics of the participants; λ, λ1, and λ2 are weight coefficients.

[0065] like Figure 2 As shown, the present invention provides a group behavior recognition device based on a pyramid squeeze attention module, the device comprising:

[0066] Initialization module: configured to extract individual appearance features for each participant in each frame of the video clip based on a trained two-dimensional convolutional neural network; and to extract key point features for each participant in each frame of the video clip using a trained HRNet network. The key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip.

[0067] A fusion module is configured to input the individual appearance features and key point features of each participant into a trained group pyramid squeezed attention model, wherein the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, calculates weighted fusion features of the individual appearance features of each participant and the key point features of the participant, and constructs a pyramid group appearance feature map based on the weighted fusion features of each participant;

[0068] Identification module: configured to input the pyramid group appearance feature map into the trained graph neural network model to identify group behavior.

[0069] An embodiment of the present invention further provides an electronic device, including:

[0070] A processor, which is used to execute multiple instructions;

[0071] A memory for storing a plurality of instructions;

[0072] The plurality of instructions are used to be stored by the memory and loaded and executed by the processor to implement the method as described above.

[0073] An embodiment of the present invention further provides a computer-readable storage medium, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the method described above.

[0074] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0075] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, which may be electrical, mechanical or other forms.

[0076] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0077] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of hardware plus software function unit.

[0078] The integrated unit realized in the form of software function unit can be stored in a computer readable storage medium. The software function unit is stored in a storage medium, including a plurality of instructions for enabling a computer device (which can be a personal computer, a server, or a network cloud server, etc., and needs to be installed with an Ubuntu operating system) to execute part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a variety of program code storage media such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0079] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present application still falls within the scope of the technical solution of the present application.

Claims

1. A group behavior recognition method based on pyramid squeezed attention module, characterized in that: The method comprises the following steps: Step S1: extracting individual appearance features of each participant in each frame of the video clip based on the trained two-dimensional convolutional neural network; using the trained HRNet network to extract key point features of each participant in each frame of the video clip, wherein the key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip; Step S2: Inputting the individual appearance features and key point features of each participant into the trained group pyramid squeezed attention model, the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, and calculates the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; constructing a pyramid group appearance feature map based on the weighted fusion features of each participant; Step S3: inputting the pyramid group appearance feature graph into the trained graph neural network model to identify group behavior; The group pyramid squeezed attention model includes a bilinear interpolation module and a first branch, a second branch, and a third branch connected to the bilinear interpolation module in parallel, wherein the first branch includes a first pyramid segmentation attention module and a multi-layer perceptron connected in sequence; the second branch includes a first scaled dot product module; the third branch includes a second pyramid segmentation attention module, a first graph convolution layer, a second graph convolution layer, a third graph convolution layer connected in sequence, and a first sub-branch and a second sub-branch connected to the third graph convolution layer in parallel, wherein the first sub-branch includes a similarity calculation module and a multi-layer perceptron module connected in sequence, and the second sub-branch includes a second scaled dot product module; The first pyramid segmentation attention module and the second pyramid segmentation attention module both calculate the participant’s OKS features: in, Indicates the Participant No. Key point The OKS feature of the moment, Indicates the Participant No. Key point Time coordinates and their Key point +1 time coordinate Euclidean distance, Indicates the Participants Always mark the box area, Indicates the statistically obtained Normalization factor for the keypoint.

2. The method according to claim 1, wherein The individual appearance features of each participant are input into the bilinear interpolation module, and the bilinear interpolation module interpolates the individual appearance features of each participant using a bilinear interpolation algorithm to generate a feature Papp'. The feature Papp' is input into the first branch, and the first pyramid segmentation attention module performs a feature fusion operation on the feature Papp' to generate a feature Papp''. The multi-layer perceptron reduces the dimension of the feature Papp'' to generate a feature Vapp; The feature Papp' is input into the second branch, and the first scaled dot product module establishes a first graph structure Rapp based on the feature Papp'. The first graph structure Rapp uses participants as nodes to reflect a first connection relationship between participants. The feature Vapp is integrated with the first graph structure Rapp to generate the first pyramid group appearance feature graph; The feature Papp' is input into the third branch, the second pyramid segmentation attention module performs a fusion feature operation on the feature Papp', the fused feature is input into the first graph convolution layer, and the first convolution result is input into the second graph convolution layer to obtain the second convolution result, the second convolution result is input into the third graph convolution layer to generate the third convolution result; the third convolution result is spliced ​​with the key point features of each participant to generate a key point splicing feature, and the key point splicing feature is input into the first sub-branch and the second sub-branch respectively, the similarity calculation module of the first sub-branch calculates the coordinate feature vector of the participant, and the generated coordinate feature is input into the first sub-branch. The eigenvector Pkey'' is input into the multi-layer perceptron module, which includes an input layer, a hidden layer and an output layer. The multi-layer perceptron module converts the coordinate eigenvector Pkey'' into a feature Vkey; the second sub-branch inputs the key point splicing feature into the second scaling dot product module, and the second scaling dot product module establishes a second graph structure Rkey based on the key point splicing feature. The second graph structure Rkey uses participants as nodes to reflect the second connection relationship between participants; the feature Vkey is fused with the second graph structure Rkey to generate a second pyramid group key point feature map.

3. The method according to claim 1, wherein The loss function used is: in, is the loss function, is the fused group behavior loss function, represents the true label of group behavior, Represents the group behavior characteristics after global pooling; They are respectively the group behavior loss function based on the individual appearance characteristics of the participants and the group behavior cross loss function based on the key point characteristics of the participants; 、 、 is the weight coefficient.

4. A group behavior recognition device based on a pyramid squeeze attention module, characterized in that: The device comprises: Initialization module: configured to extract individual appearance features for each participant in each frame of the video clip based on a trained two-dimensional convolutional neural network; and to extract key point features for each participant in each frame of the video clip using a trained HRNet network. The key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip. A fusion module is configured to input the individual appearance features and key point features of each participant into a trained group pyramid squeezed attention model, wherein the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, calculates weighted fusion features of the individual appearance features of each participant and the key point features of the participant, and constructs a pyramid group appearance feature map based on the weighted fusion features of each participant; Recognition module: configured to input the pyramid group appearance feature map into the trained graph neural network model to identify group behavior; The group pyramid squeezed attention model includes a bilinear interpolation module and a first branch, a second branch, and a third branch connected to the bilinear interpolation module in parallel, wherein the first branch includes a first pyramid segmentation attention module and a multi-layer perceptron connected in sequence; the second branch includes a first scaled dot product module; the third branch includes a second pyramid segmentation attention module, a first graph convolution layer, a second graph convolution layer, a third graph convolution layer connected in sequence, and a first sub-branch and a second sub-branch connected to the third graph convolution layer in parallel, wherein the first sub-branch includes a similarity calculation module and a multi-layer perceptron module connected in sequence, and the second sub-branch includes a second scaled dot product module; The first pyramid segmentation attention module and the second pyramid segmentation attention module both calculate the participant’s OKS features: in, Indicates the Participant No. Key point The OKS feature of the moment, Indicates the Participant No. Key point Time coordinates and their Key point +1 time coordinate Euclidean distance, Indicates the Participants Always mark the box area, Indicates the statistically obtained Normalization factor for the keypoint.

5. An electronic device comprising: A processor, which is used to execute multiple instructions; A memory for storing a plurality of instructions; The plurality of instructions are used to be stored in the memory and loaded and executed by the processor according to any one of claims 1 to 3.

6. A computer-readable storage medium, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the method according to any one of claims 1 to 3.

Images