Group behavior recognition method and device based on pyramid extrusion attention module

Abstract

The invention provides a group behavior recognition method and device based on a pyramid extrusion attention module, and the method comprises the steps: extracting the individual apparent characteristics of each participant in each frame in a video clip, and extracting the key point characteristics of each participant in each frame in the video clip; inputting the individual apparent characteristics and the key point characteristics of each participant into a trained group pyramid extrusion attention model, and distributing attention weights for the individual apparent characteristics of each participant by the pyramid extrusion attention model, calculating a weighted fusion feature of the individual apparent feature of each participant and the key point feature of the participant; constructing a pyramid group apparent feature map based on the weighted fusion features of each participant; and inputting the pyramid group apparent feature graph into a trained graph neural network model, and identifying group behaviors. According to the method, low calculation and storage cost is kept, local and non-local feature information is integrated, and accurate image super-resolution reconstruction is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of video analysis, and in particular to a group behavior recognition method and device based on a pyramid squeeze attention module. Background Art

[0002] Group behavior recognition is the overall behavior of multiple individuals in the recognition scene to achieve the same specific goal. Prior art A group behavior recognition method and system based on dynamic dual-stream self-attention (CN118506437A) aggregates the interaction relationship between participants into group interaction features between participants and groups through self-attention or graph convolutional neural network; after extracting the apparent features of the participants, the apparent features on different time frames are stacked into a spatiotemporal graph, and the time weights of the video frames in the video clips of group behavior in the spatiotemporal graph are analyzed to identify group behavior.

[0003] However, the disadvantage of the existing technology is that the ordinary group behavior recognition graph convolution only considers channel attention and ignores spatial attention. Among them, the traditional BAM and CBAM consider channel attention and spatial attention, but there are still two most important disadvantages: (1) spatial information of different scales is not captured to enrich the feature space. (2) Spatial attention only considers the information of local areas and cannot establish long-distance dependencies. In addition, the subsequent PyConv, Res2Net and HS-ResNet are all used to solve these two shortcomings of CBAM, but the computational complexity is too large. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a group behavior recognition method and device based on a pyramid squeeze attention module.

[0005] According to a first aspect of the present invention, a method for group behavior recognition based on a pyramid squeezed attention module is provided, the method comprising the following steps:

[0006] Step S1: extracting individual appearance features of each participant in each frame of the video clip based on the trained two-dimensional convolutional neural network; using the trained HRNet network to extract key point features of each participant in each frame of the video clip, wherein the key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip;

[0007] Step S2: inputting the individual appearance features and key point features of each participant into the trained group pyramid squeezed attention model, the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, and calculates the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; constructing a pyramid group appearance feature map based on the weighted fusion features of each participant;

[0008] Step S3: input the pyramid group appearance feature map into the trained graph neural network model to identify group behavior.

[0009] Preferably, the group pyramid squeezing attention model includes a bilinear interpolation module and a first branch, a second branch, and a third branch connected in parallel to the bilinear interpolation module, wherein the first branch includes a first pyramid segmentation attention module and a multi-layer perceptron connected in sequence; the second branch includes a first scaling dot product module; the third branch includes a second pyramid segmentation attention module, a first graph convolution layer, a second graph convolution layer, a third graph convolution layer connected in sequence, and a first sub-branch and a second sub-branch connected in parallel to the third graph convolution layer, wherein the first sub-branch includes a similarity calculation module and a multi-layer perceptron module connected in sequence, and the second sub-branch includes a second scaling dot product module.

[0010] Preferably, the individual appearance features of each participant are input into the bilinear interpolation module, and the bilinear interpolation module interpolates the individual appearance features of each participant using a bilinear interpolation algorithm to generate a feature Papp', and the feature Papp' is input into the first branch. The first pyramid segmentation attention module performs a fusion feature operation on the feature Papp' to generate a feature Papp", and the feature Papp" is reduced in dimension by a multi-layer perceptron to generate a feature Vapp; the feature Papp' is input into the second branch, and the first scaling dot product module establishes a first graph structure Rapp based on the feature Papp', and the first graph structure Rapp uses the participants as nodes to reflect the first connection relationship between the participants; the feature Vapp is fused with the first graph structure Rapp to generate a first pyramid group appearance feature map; the feature Papp' is input into the third branch, and the second pyramid segmentation attention module performs a fusion feature operation on the feature Papp', and the fused features are input into the first graph convolution layer, and then the first convolution structure Rapp is used as the first graph structure Rapp. The result is input into the second graph convolution layer to obtain the second convolution result, and the second convolution result is input into the third graph convolution layer to generate the third convolution result; the third convolution result is spliced ​​with the key point features of each participant to generate the key point splicing feature, and the key point splicing feature is respectively input into the first sub-branch and the second sub-branch, the similarity calculation module of the first sub-branch calculates the coordinate feature vector of the participant, and the generated coordinate feature vector Pkey" is input into the multi-layer perceptron module, and the multi-layer perceptron module includes an input layer, a hidden layer and an output layer, and the multi-layer perceptron module converts the coordinate feature vector Pkey" into a feature Vkey; the second sub-branch inputs the key point splicing feature into the second scaling dot product module, and the second scaling dot product module establishes a second graph structure Rkey based on the key point splicing feature, and the second graph structure Rkey uses the participants as nodes to reflect the second connection relationship between the participants; the feature Vkey is fused with the second graph structure Rkey to generate a second pyramid group key point feature map.

[0011] Preferably, the first pyramid segmentation attention module and the second pyramid segmentation attention module both calculate the OKS features of the participant:

[0012]

[0013] in, represents the OKS feature of the jth key point of the nth participant at time t, d n,j,t represents the Euclidean distance between the coordinate of the jth key point of the nth participant at time t and the coordinate of the jth key point at time t+1, s n,t represents the area of ​​the labeled box of the nth participant at time t, δ j Represents the normalization factor of the jth key point obtained by statistics.

[0014] Preferably, the loss function used is:

[0015]

[0016] in, is the loss function, is the fused group behavior loss function, G represents the true label of the group behavior, Represents the group behavior characteristics after global pooling; are the group behavior loss function based on the individual appearance characteristics of the participants and the group behavior cross loss function based on the key point characteristics of the participants, respectively; λ, λ 1 , 2 is the weight coefficient.

[0017] According to a second aspect of the present invention, a group behavior recognition device based on a pyramid squeezed attention module is provided, the device comprising:

[0018] Initialization module: configured to extract individual appearance features of each participant in each frame of the video clip based on the trained two-dimensional convolutional neural network; use the trained HRNet network to extract key point features of each participant in each frame of the video clip, wherein the key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip;

[0019] Fusion module: configured to input the individual appearance features and key point features of each participant into the trained group pyramid squeezed attention model, the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, calculates the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; constructs a pyramid group appearance feature map based on the weighted fusion features of each participant;

[0020] Recognition module: configured to input the pyramid group appearance feature map into the trained graph neural network model to recognize group behavior.

[0021] According to a third aspect of the present invention, there is provided an electronic device, comprising:

[0022] A processor, which is used to execute multiple instructions;

[0023] A memory for storing a plurality of instructions;

[0024] The plurality of instructions are used to be stored by the memory and loaded and executed by the processor to implement the method as described above.

[0025] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the aforementioned method.

[0026] The present invention has the following technical effects:

[0027] This paper introduces an efficient pyramid squeeze attention (PSA) module, which significantly improves the performance of deep convolutional neural networks in group behavior recognition visual tasks. By replacing the 3x3 convolution in the bottleneck block of ResNet with the PSA module, GPSANet can provide richer multi-scale feature representation and more effective channel space interaction without significantly increasing the computational burden.

[0028] The present invention uses a pyramid squeezed attention module to integrate the information of the input feature map by using a multi-scale pyramid convolution structure, and effectively extracts spatial information of different scales from the feature map of each channel by squeezing the channel dimension of the input tensor. In addition, by extracting the channel attention weights of the multi-scale feature map and recalibrating the attention weights of the corresponding channels using the Softmax operation, long-range channel dependencies are established.

[0029] The present invention constructs a group pyramid squeezed attention (GPSA) block, which is easy to add to the existing backbone network as a plug-and-play component and can significantly improve the model performance. It provides stronger multi-scale representation capabilities for group behavior recognition tasks and can adaptively recalibrate channel attention weights across dimensions.

[0030] The present invention is more efficient in memory usage, effectively aggregates features through dynamic spatial modulation, improves reconstruction performance, while maintaining low computing and storage costs, effectively integrates local and non-local feature information, and achieves more accurate image super-resolution reconstruction.

[0031] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The accompanying drawings constituting a part of the present invention are used to provide a further understanding of the present invention, and the present invention is described by providing the following accompanying drawings. In the accompanying drawings:

[0033] Figure 1 A schematic flow chart of a group behavior recognition method based on a pyramid squeezed attention module according to an embodiment of the present invention;

[0034] Figure 2 This is a structural block diagram of a group behavior recognition device based on a pyramid squeeze attention module according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] First combine Figure 1 The present invention describes a method for group behavior recognition based on a pyramid squeeze attention module according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0036] Step S1: extracting individual appearance features of each participant in each frame of the video clip based on the trained two-dimensional convolutional neural network; using the trained HRNet network to extract key point features of each participant in each frame of the video clip, wherein the key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip;

[0037] Step S2: inputting the individual appearance features and key point features of each participant into the trained group pyramid squeezed attention model, the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, and calculates the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; constructing a pyramid group appearance feature map based on the weighted fusion features of each participant;

[0038] Step S3: input the pyramid group appearance feature map into the trained graph neural network model to identify group behavior.

[0039] The step S1: extracting individual appearance features of each participant in each frame of the video clip based on the trained two-dimensional convolutional neural network; using the trained HRNet network to extract key point features of each participant in each frame of the video clip, wherein the key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip, wherein:

[0040] The video clip is a sparsely sampled video clip For each video frame in the video clip, a two-dimensional convolutional neural network is used to extract features to obtain a feature map of the group behavior scene corresponding to the video frame. Where T is the time length of the video clip, W×H is the size of the video frame, and oW×oH is the size of the feature map of the group behavior scene. The RoIAlgin algorithm is used to extract the individual behavior appearance characteristics of each participant in the feature map of the group behavior scene. In the present invention, the RoIAlgin algorithm extracts the individual behavior appearance characteristics of group behavior participants based on the coordinates of the participant annotation boxes in the data set. The individual behavior appearance characteristics of each participant are a feature map. The RoIAlgin algorithm also aligns the extracted features so that the individual behavior appearance feature map of each participant is K×K in size. Next, the individual behavior appearance characteristics of each participant are embedded using the FFLN network (Fourier Features LetNetwork, FFLN) to obtain the individual appearance characteristics of each participant. Where N is the number of participants in the group behavior scene, and d represents the output dimension of the feature embedding operation. The HRNet network extracts the key point coordinates of each participant to obtain the key point features of the participant J represents the number of key points of the participants. The HRNet network is pre-trained using the AIC dataset and estimates the key point coordinates of the participants based on the annotation boxes. For each participant, individual appearance features and key point features are obtained. In other words, a feature set P represented by the individual features of the participants in the group behavior can be constructed:

[0041] P={P app ,P key}

[0042] Step S2: inputting the individual appearance features and key point features of each participant into the trained group pyramid squeezed attention model, the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, and calculates the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; constructing a pyramid group appearance feature map based on the weighted fusion features of each participant, wherein:

[0043] The group pyramid squeezed attention model includes a bilinear interpolation module and a first branch, a second branch, and a third branch connected to the bilinear interpolation module in parallel, wherein the first branch includes a first pyramid segmentation attention module and a multi-layer perceptron connected in sequence; the second branch includes a first scaling dot product module; the third branch includes a second pyramid segmentation attention module, a first graph convolution layer, a second graph convolution layer, a third graph convolution layer connected in sequence, and a first sub-branch and a second sub-branch connected to the third graph convolution layer in parallel, wherein the first sub-branch includes a similarity calculation module and a multi-layer perceptron module connected in sequence, and the second sub-branch includes a second scaling dot product module;

[0044] The individual appearance features of each participant are input into the bilinear interpolation module, and the bilinear interpolation module interpolates the individual appearance features of each participant using a bilinear interpolation algorithm to generate a feature Papp', and the feature Papp' is input into the first branch. The first pyramid segmentation attention module performs a fusion feature operation on the feature Papp' to generate a feature Papp", and the feature Papp" is reduced in dimension by a multi-layer perceptron to generate a feature Vapp; the feature Papp' is input into the second branch, and the first scaling dot product module establishes a first graph structure Rapp based on the feature Papp'. The first graph structure Rapp uses the participants as nodes to reflect the first connection relationship between the participants; the feature Vapp is fused with the first graph structure Rapp to generate a first pyramid group appearance feature map, and the first pyramid group appearance feature map reflects the group behavior modeling process based on the graph method; the feature Papp' is input into the third branch, and the second pyramid segmentation attention module performs a fusion feature operation on the feature Papp', and the fused features are input into the first graph convolution layer, and then The first convolution result is input into the second graph convolution layer to obtain the second convolution result, and the second convolution result is input into the third graph convolution layer to generate the third convolution result; the third convolution result is spliced ​​with the key point features of each participant to generate a key point splicing feature, and the key point splicing feature is respectively input into the first sub-branch and the second sub-branch, the similarity calculation module of the first sub-branch calculates the coordinate feature vector of the participant, and the generated coordinate feature vector Pkey" is input into the multi-layer perceptron module, the multi-layer perceptron module includes an input layer, a hidden layer and an output layer, and the multi-layer perceptron module converts the coordinate feature vector Pkey" into a feature Vkey; the second sub-branch inputs the key point splicing feature into the second scaling dot product module, and the second scaling dot product module establishes a second graph structure Rkey based on the key point splicing feature, and the second graph structure Rkey uses the participants as nodes to reflect the second connection relationship between the participants; the feature Vkey is fused with the second graph structure Rkey to generate a second pyramid group key point feature map, and the second pyramid group key point feature map reflects the group appearance.

[0045] The pyramid segmentation attention (PSA) module in the present invention is a conventional pyramid segmentation attention (PSA) module in the art.

[0046] Pyramid Squeeze Attention (PSA) is an efficient deep learning module designed to improve the performance of convolutional neural networks in a variety of computer vision tasks. By introducing multi-scale feature extraction and channel attention mechanism, PSA effectively integrates the information in the input feature map, thereby enhancing the representation ability of the model.

[0047] PSA processes input features through a multi-scale convolutional structure. It divides the input feature map into multiple parts in the channel dimension, and each part is convolved with a convolution kernel of different sizes. This design allows the module to extract features from different scales, allowing the network to capture more contextual information. In the PSA module, a Squeeze-and-Excitation (SE) structure is applied to the feature map of each scale. This structure first compresses the spatial dimension of the feature map to 1x1 through an adaptive average pooling layer to obtain the global information of the channel. Then, a fully connected layer (or convolutional layer) is used to adjust the number of channels, a nonlinear transformation is performed through the ReLU activation function, and finally the channel attention weights are output through the Sigmoid function. These weights are used to dynamically adjust the feature response of each channel, thereby realizing the modeling of the dependency relationship between channels. In the forward propagation process of PSA, the features of different scales are first convolved to obtain the feature representation of each scale. Then, these features are weighted using the channel attention weights calculated by the SE module, and finally the multi-scale features are fused into a unified feature representation by summing. This process not only enhances the response of important features, but also suppresses irrelevant information, thereby improving the final feature expression.

[0048] For example, the dimension of the input feature map is (B, C, H, W), where B is the batch size, C is the number of channels, and H and W are the height and width of the feature map. Channel segmentation: The input feature map is split into S parts in the channel dimension, allowing each part to be processed independently. Convolutional layer: Each channel part is extracted through a convolutional layer of different sizes. The size is determined by 2(i+1)+1, where i is the index of the current scale. SE module: Apply the corresponding SE module to the feature map of each scale to obtain the channel attention weight. Feature merging: The weighted multi-scale features are summed to generate the final output feature map.

[0049] Experiments on multiple standard datasets show that the model using this module has significant improvements in accuracy and detection performance. Compared with other complex attention mechanisms, PSA can effectively reduce computational overhead while maintaining high performance. The PSA module can be easily integrated into existing convolutional networks and is a plug-and-play component.

[0050] In the present invention, the PSA module is used to provide attention weighting for the key point features of the participants, thereby enhancing the focus on the key participant's actions. This can be achieved through a similar attention mechanism, so that the key point features are more focused on meaningful dynamic behaviors.

[0051] In order to reduce the noise caused by the key points of the pose estimation network, the OKS feature of the participant is calculated according to the object key point similarity feature OKS (Object Key-point Similarity, OKS) That is, both the first pyramid segmentation attention module and the second pyramid segmentation attention module calculate the OKS features of the participants:

[0052]

[0053] in, represents the OKS feature of the jth key point of the nth participant at time t, d n,j,t represents the Euclidean distance between the coordinate of the jth key point of the nth participant at time t and the coordinate of the jth key point at time t+1, s n,t represents the area of ​​the labeled box of the nth participant at time t, δ j It represents the normalization factor of the jth key point obtained by statistics, reflecting the uncertainty of manually annotating this type of key point.

[0054] Then, in order to reduce the influence of the estimation deviation of key point coordinates on the network, the key point coordinates and the differential calculation results of the coordinates are further regularized, and the results are recorded as and

[0055]

[0056] Among them, k mean represents the average value of the key points in the T time period, k std-mean In addition, in order to further extract the spatiotemporal features of the key point coordinates, the key point coordinates are embedded in the video spatial domain and video temporal domain respectively, and the results are recorded as and

[0057] By concatenating the above calculation results, we can get the key point coordinate feature vector where d k Represents the feature dimension of semantic features. key Splice and get the key point feature P″ key , after a multi-layer perceptron layer, the embedded key point features of the participants are obtained as the participant behavior interaction graph G key The graph node V key .

[0058] Step S3: inputting the pyramid group appearance feature graph into the trained graph neural network model to identify group behavior, including:

[0059] The first pyramid group appearance feature map G app And the second pyramid group key point feature map G key Input the trained graph neural network model respectively to obtain the first group behavior feature G′ app and the second group behavior characteristic G′ key ; Fusion of the first group behavior feature G′ app and the second group behavior characteristic G′ key , obtaining group behavior fusion characteristics; and determining group behavior based on the group behavior fusion characteristics.

[0060] The present invention explores an early feature fusion method based on addition and a late feature fusion method based on addition respectively. The results show that the late fusion method based on addition achieves better group behavior recognition results.

[0061] Furthermore, the present invention uses a cross entropy loss function to supervise the model to update weights. The entire group behavior recognition process can be performed in an end-to-end manner, and the cross entropy loss function is used to supervise the model update.

[0062] The loss function used is:

[0063]

[0064] in, is the loss function, is the fused group behavior loss function, G represents the true label of the group behavior, Represents the group behavior characteristics after global pooling; are the group behavior loss function based on the individual appearance characteristics of the participants and the group behavior cross loss function based on the key point characteristics of the participants, respectively; λ, λ 1 , 2 is the weight coefficient.

[0065] like Figure 2 As shown, the present invention provides a group behavior recognition device based on a pyramid squeeze attention module, the device comprising:

[0066] Initialization module: configured to extract individual appearance features of each participant in each frame of the video clip based on the trained two-dimensional convolutional neural network; use the trained HRNet network to extract key point features of each participant in each frame of the video clip, wherein the key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip;

[0067] Fusion module: configured to input the individual appearance features and key point features of each participant into the trained group pyramid squeezed attention model, the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, calculates the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; constructs a pyramid group appearance feature map based on the weighted fusion features of each participant;

[0068] Recognition module: configured to input the pyramid group appearance feature map into the trained graph neural network model to recognize group behavior.

[0069] The embodiment of the present invention further provides an electronic device, including:

[0070] A processor, which is used to execute multiple instructions;

[0071] A memory for storing a plurality of instructions;

[0072] The plurality of instructions are used to be stored by the memory and loaded and executed by the processor to implement the method as described above.

[0073] The embodiment of the present invention further provides a computer-readable storage medium, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the method as described above.

[0074] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments may be combined with each other.

[0075] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0076] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0077] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0078] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a physical server, or a network cloud server, etc., and the Ubuntu operating system must be installed) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0079] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention still falls within the scope of the technical solution of the present invention.

Claims

1. A group behavior recognition method based on pyramid squeezed attention module, characterized in that: The method comprises the following steps: Step S1: extracting individual appearance features of each participant in each frame of the video clip based on the trained two-dimensional convolutional neural network; using the trained HRNet network to extract key point features of each participant in each frame of the video clip, wherein the key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip; Step S2: inputting the individual appearance features and key point features of each participant into the trained group pyramid squeezed attention model, the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, and calculates the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; constructing a pyramid group appearance feature map based on the weighted fusion features of each participant; Step S3: input the pyramid group appearance feature map into the trained graph neural network model to identify group behavior.

2. The method according to claim 1, characterized in that The group pyramid squeezing attention model includes a bilinear interpolation module and a first branch, a second branch, and a third branch connected to the bilinear interpolation module in parallel, wherein the first branch includes a first pyramid segmentation attention module and a multi-layer perceptron connected in sequence; the second branch includes a first scaling dot product module; the third branch includes a second pyramid segmentation attention module, a first graph convolution layer, a second graph convolution layer, a third graph convolution layer connected in sequence, and a first sub-branch and a second sub-branch connected to the third graph convolution layer in parallel, wherein the first sub-branch includes a similarity calculation module and a multi-layer perceptron module connected in sequence, and the second sub-branch includes a second scaling dot product module.

3. The method according to claim 2, characterized in that The individual appearance features of each participant are input into the bilinear interpolation module, and the bilinear interpolation module interpolates the individual appearance features of each participant using a bilinear interpolation algorithm to generate a feature Papp', and the feature Papp' is input into the first branch, and the first pyramid segmentation attention module performs a fusion feature operation on the feature Papp' to generate a feature Papp", and the multi-layer perceptron reduces the dimension of the feature Papp" to generate a feature Vapp; The feature Papp' is input into the second branch, and the first scaling dot product module establishes a first graph structure Rapp based on the feature Papp', wherein the first graph structure Rapp uses participants as nodes to reflect a first connection relationship between the participants; The feature Vapp is merged with the first graph structure Rapp to generate a first pyramid group appearance feature graph; The feature Papp' is input into the third branch, the second pyramid segmentation attention module performs a fusion feature operation on the feature Papp', the fused feature is input into the first graph convolution layer, the first convolution result is input into the second graph convolution layer to obtain the second convolution result, the second convolution result is input into the third graph convolution layer to generate the third convolution result; the third convolution result is spliced ​​with the key point features of each participant to generate a key point splicing feature, the key point splicing feature is input into the first sub-branch and the second sub-branch respectively, the similarity calculation module of the first sub-branch calculates the coordinate feature vector of the participant, and the generated coordinate The feature vector Pkey" is input into a multi-layer perceptron module, which includes an input layer, a hidden layer and an output layer. The multi-layer perceptron module converts the coordinate feature vector Pkey" into a feature Vkey; the second sub-branch inputs the key point splicing feature into a second scaling dot product module, and the second scaling dot product module establishes a second graph structure Rkey based on the key point splicing feature. The second graph structure Rkey uses participants as nodes to reflect the second connection relationship between participants; the feature Vkey is fused with the second graph structure Rkey to generate a second pyramid group key point feature map.

4. The method according to claim 2, characterized in that Both the first pyramid segmentation attention module and the second pyramid segmentation attention module calculate the OKS features of the participants: in, represents the OKS feature of the jth key point of the nth participant at time t, d n,j,t represents the Euclidean distance between the coordinate of the jth key point of the nth participant at time t and the coordinate of the jth key point at time t+1, s n,t represents the area of ​​the labeled box of the nth participant at time t, δ j Represents the normalization factor of the jth key point obtained by statistics.

5. The method according to claim 1, characterized in that The loss function used is: in, is the loss function, is the fused group behavior loss function, G represents the true label of the group behavior, Represents the group behavior characteristics after global pooling; They are respectively the group behavior loss function based on the individual appearance characteristics of the participants and the group behavior cross loss function based on the key point characteristics of the participants; λ, λ1, and λ2 are weight coefficients.

6. A group behavior recognition device based on a pyramid squeeze attention module, characterized in that: The device comprises: Initialization module: configured to extract individual appearance features of each participant in each frame of the video clip based on the trained two-dimensional convolutional neural network; use the trained HRNet network to extract key point features of each participant in each frame of the video clip, wherein the key point features are the coordinates of the key points corresponding to each participant in each frame of the video clip; Fusion module: configured to input the individual appearance features and key point features of each participant into the trained group pyramid squeezed attention model, the pyramid squeezed attention model assigns attention weights to the individual appearance features of each participant, calculates the weighted fusion features of the individual appearance features of each participant and the key point features of the participant; constructs a pyramid group appearance feature map based on the weighted fusion features of each participant; Recognition module: configured to input the pyramid group appearance feature map into the trained graph neural network model to recognize group behavior.

7. An electronic device comprising: A processor, which is used to execute multiple instructions; A memory for storing a plurality of instructions; The plurality of instructions are used to be stored in the memory and loaded and executed by the processor according to any one of claims 1 to 5.

8. A computer-readable storage medium, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the method as claimed in any one of claims 1 to 5.

Images