Group behavior recognition method, system and storage medium
By using feature fusion and relationship interaction graph methods, the problem of low accuracy in group behavior recognition in existing technologies is solved, and more accurate group behavior recognition is achieved.
Patent Information
- Application Number
- CN202410897337.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-05
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-07-05
AI Technical Summary
Existing group behavior recognition methods ignore scene context information, resulting in low behavior recognition accuracy. Occlusion of people leads to missing information, making it difficult to identify key people and affecting classification results.
A feature fusion-based group behavior recognition method is adopted, which combines a backbone network, a fusion module, an embedding layer, a relationship reasoning module, a graph convolutional network, and a classifier. By fusing global features and individual features, a relationship interaction graph between people is constructed, and a graph convolutional network is used for training and classification.
It improves the accuracy of group behavior recognition by embedding scene information through the fusion module, focusing on key figures and frames, and thus enhancing recognition accuracy.
Smart Images

Figure CN118429870B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of video analysis, and specifically refers to a group behavior recognition method, system and storage medium. Background Art
[0002] Group behavior recognition is one of the most popular research directions in the field of computer vision. It involves cross-disciplinary research in machine learning, sociology, psychology, etc. It aims to understand and analyze the behavior patterns, interactions and dynamic changes of multiple individuals in a group, and explore the behavioral characteristics and laws of people in different situations.
[0003] At present, many methods are dedicated to building interactive relationships between individuals in multi-person scenarios in order to accurately infer group behavior. Early methods based on recurrent neural networks (RNNs) can effectively explore movement patterns over long periods of time. Existing methods often use convolutional neural networks (CNNs) and graph convolutional networks (GNNs) to simulate individual relationships, and combine attention mechanisms to finely extract relationship information of characters and build relationship interaction graphs. These methods effectively explore the interactive clues between multiple individuals in a group, and have promoted the development of group behavior recognition.
[0004] Although the above methods have achieved good results, the following problems still exist: (1) Ignoring scene context information, resulting in low behavior recognition accuracy. When identifying group behavior, scene context information refers to the environment and background information in the scene where the characters are located that helps to judge individual and group actions. For example, in volleyball, the trajectory of the volleyball usually determines the position and behavior tendency of the players, and plays an important role in judging the group behavior of the players. In addition, the layout and structure of the scene and the dynamic changes of other characters will also have a certain impact on the accuracy of the target person's behavior recognition. However, existing methods focus on modeling the relationship between individuals and rarely explore clues related to characters in the scene. These methods usually directly extract global features containing scene information from the original image, and then use RoI Align to map the features to the region of interest based on the annotated character position, obtain individual features containing character detail information, and finally use individual features to predict the category of group behavior. If only the character detail features are used, their connection with the global scene will be ignored, affecting the final behavior recognition effect.
[0005] (2) There are cases where people are occluded in the image, resulting in the loss of some person information and the inability to accurately obtain appearance details. If only individual characteristics are used to infer group behavior, the accuracy of behavior recognition will be affected due to missing information.
[0006] (3) It is difficult to identify key individuals, causing the classification results to be disturbed by irrelevant individuals. Each individual in the scene has a different degree of influence on group behavior. Only by identifying key individuals can a more accurate group behavior classification be achieved. If a person who is irrelevant to group behavior is selected to predict group behavior, the classification results will be wrong. Summary of the invention
[0007] In order to solve the problems existing in the prior art, the present invention provides a group behavior recognition method and system, which can improve the accuracy of group behavior recognition.
[0008] The technical problem to be solved by the present invention is achieved through the following technical solutions:
[0009] In a first aspect, a method for group behavior recognition based on feature fusion is provided, comprising: obtaining a video frame to be recognized;
[0010] According to the video frames, the group behavior category is obtained based on the pre-trained group behavior recognition model;
[0011] The group behavior recognition model includes: a backbone network, a fusion module, an embedding layer, a relational reasoning module, a graph convolutional network, a global average pooling layer and a classifier.
[0012] In combination with the first aspect, further, the backbone network is VGG-16 or Inception-v3.
[0013] Combined with the first aspect, based on the backbone network, the video frame is used as input to obtain the global features ;
[0014] According to global characteristics Using regional interest alignment to obtain individual features ;
[0015] Based on the fusion module, the global features and individual characteristics As input for fusion, the fusion features finally output by the fusion module are obtained. ;
[0016] Based on the embedding layer, individual features are As input, the output of the embedding layer is combined with the fusion feature output of the fusion module. Perform residual calculation to obtain the fusion features finally output by the fusion module ;
[0017] Based on the relational reasoning module, according to the final fusion features Get the relationship interaction diagram between individuals ;
[0018] Based on the pre-trained graph convolutional network, the relationship interaction graph between individuals And the final fusion feature As input, the output of the graph convolutional network is combined with individual features After the residual calculation, it passes through the global average pooling layer and the classifier in turn to obtain the final group behavior category.
[0019] In combination with the first aspect, the final fusion feature Get the relationship interaction diagram between individuals include:
[0020]
[0021]
[0022] in, The final fusion feature Individual characteristics in the space-time dimension, is the time step, is the number of target individuals in a single image, is the fusion function, For individuals With individuals The appearance similarity between For individuals With individuals The positional relationship between , The final fusion features are In the space-time dimension and individuals Appearance characteristics; The final fusion features are In the space-time dimension and individuals Location characteristics; For individuals With individuals An interaction diagram of the relationships between them.
[0023] Combined with the first aspect, further, the expression for obtaining the group behavior category is as follows:
[0024]
[0025] in, is the final group behavior category, is the classifier, is the global average pooling layer, is the output of the graph convolutional network.
[0026] In combination with the first aspect, further, the fusion module includes: a convolution layer, an attention mechanism module, a linear embedding layer, a Softmax layer, a layer normalization layer and a feedforward network;
[0027] Targeting individual characteristics and global features Position encoding is performed separately to add position information, and then the individual features and global features after position encoding are passed through the convolution layer for channel planning. The expression is as follows:
[0028]
[0029] in, represents the position code, represents a pointwise convolution layer with a 1×1 convolution kernel, Represents individual features or global features after position encoding, After the above calculation, the output , Corresponding to individual characteristics, it is expressed as , corresponding to the global feature, is expressed as ;
[0030] Based on the attention mechanism module, the individual features after the convolution layer are and global features Add a channel attention mechanism, the expression is as follows:
[0031]
[0032] in, represents the channel attention mechanism, To add individual features or global features after the channel attention mechanism, Corresponding to individual characteristics, it is expressed as , corresponding to the global feature, is expressed as ; Indicates the dimension size is The weight matrix of
[0033] The individual features after adding the channel attention mechanism After the linear embedding layer, we get , the expression is as follows:
[0034]
[0035] in, represents a linear embedding layer;
[0036] The individual features that pass through the linear embedding layer And the global features after adding the channel attention mechanism Perform product fusion and obtain intermediate features through the Softmax layer , the expression is as follows:
[0037]
[0038] Among them, Softmax represents the Softmax layer;
[0039] Then the intermediate features Global features after adding channel attention mechanism After product fusion, the individual features after the linear embedding layer are Perform residual calculation to obtain fusion features , the expression is as follows:
[0040]
[0041] Then the fusion features After layer normalization, the layers are normalized again through the feedforward network to obtain fusion features with enhanced representation capabilities. , the expression is as follows:
[0042]
[0043] in, Representation layer normalization layer, represents a feed-forward network;
[0044] Then the fusion features with enhanced representation capabilities are After the convolution layer, the fusion features finally output by the fusion module are obtained , the expression is as follows:
[0045]
[0046] In a second aspect, the present invention further provides a group behavior recognition system, comprising:
[0047] A data acquisition module, used for acquiring a video frame to be identified;
[0048] The behavior recognition module is used to obtain the group behavior category based on the video frame and the pre-trained group behavior recognition model;
[0049] The group behavior recognition model includes: a backbone network, a fusion module, an embedding layer, a relational reasoning module, a graph convolutional network, a global average pooling layer and a classifier.
[0050] In conjunction with the second aspect, further, the operation performed by the behavior recognition module includes:
[0051] Based on the backbone network, the video frame is used as input to obtain global features ;
[0052] According to global characteristics Using regional interest alignment to obtain individual features ;
[0053] Based on the fusion module, the global features and individual characteristics As input for fusion, the fusion features finally output by the fusion module are obtained. ;
[0054] Based on the embedding layer, individual features are As input, the output of the embedding layer is combined with the fusion feature output of the fusion module. Perform residual calculation to obtain the final fusion feature ;
[0055] Based on the relational reasoning module, according to the final fusion features Get the relationship interaction diagram between individuals ;
[0056] Based on the pre-trained graph convolutional network, the relationship interaction graph between individuals And the final fusion feature As input, the output of the graph convolutional network is combined with individual features After the residual calculation, it passes through the global average pooling layer and the classifier in turn to obtain the final group behavior category.
[0057] In combination with the second aspect, further, the fusion module includes: a convolutional layer, an attention mechanism module, a linear embedding layer, a Softmax layer, a layer normalization layer and a feedforward network;
[0058] The operations performed by the fusion module include: and global features Position encoding is performed separately to add position information, and then the individual features after position encoding are and global features Channel planning is performed through the convolution layer, and the expression is as follows:
[0059]
[0060] in, represents the position code, represents a pointwise convolution layer with a 1×1 convolution kernel, Represents individual features or global features, After the above calculation, the output , Corresponding to individual characteristics, it is expressed as , corresponding to the global feature, is expressed as ;
[0061] Based on the attention mechanism module, the individual features after the convolution layer are and global features Add channel attention mechanism, the expression is as follows:
[0062]
[0063] in, represents the channel attention mechanism, To add individual features or global features after the channel attention mechanism, Corresponding to individual characteristics, it is expressed as , corresponding to the global feature, is expressed as ; Indicates the dimension size is The weight matrix of
[0064] The individual features after adding the channel attention mechanism After the linear embedding layer, we get , the expression is as follows:
[0065]
[0066] in, represents a linear embedding layer;
[0067] The individual features that pass through the linear embedding layer And the global features after adding the channel attention mechanism Perform product fusion and obtain intermediate features through the Softmax layer , the expression is as follows:
[0068]
[0069] Among them, Softmax represents the Softmax layer;
[0070] Then the intermediate features Global features after adding channel attention mechanism After product fusion, the individual features after the linear embedding layer are Perform residual calculation to obtain fusion features , the expression is as follows:
[0071]
[0072] Then the fusion features After layer normalization, the layers are normalized again through the feedforward network to obtain fusion features with enhanced representation capabilities. , the expression is as follows:
[0073]
[0074] in, Representation layer normalization layer, represents a feed-forward network;
[0075] Then the fusion features with enhanced representation capabilities are After the convolution layer, the fusion features finally output by the fusion module are obtained , the expression is as follows:
[0076]
[0077] In a third aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.
[0078] Beneficial effects: The present invention uses a group behavior recognition model to more accurately identify group behavior categories. It integrates global scene features and individual character features to obtain more representative fusion features, which are used to construct an interaction relationship graph between characters, which is then trained using a graph convolutional network, and a classifier is used to accurately predict the category of group behavior.
[0079] In particular, the fusion module in the model embeds the scene information contained in the global features into the individual features, providing information-rich fusion features for reasoning about individual actions and group behavior categories. In addition, the attention mechanism module used in the fusion module can effectively focus on key figures in group behavior and key frames in the data set, improving the accuracy of group behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0080] Figure 1 It is a flow chart of a group behavior recognition method based on feature fusion of the present invention;
[0081] Figure 2 It is a framework diagram of the group behavior recognition model in the present invention;
[0082] Figure 3 It is a framework diagram of the feature fusion module in the present invention. DETAILED DESCRIPTION
[0083] In order to make the objectives, technical solutions and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments.
[0084] Example 1
[0085] like Figure 1 and Figure 2As shown, this embodiment introduces a group behavior recognition method based on feature fusion, which includes the following steps:
[0086] Step 1: Video frame acquisition
[0087] Get the original video and crop it into frames.
[0088] Step 2: Prediction of group behavior
[0089] The cropped images are assigned to the pre-trained group behavior recognition model to obtain the group behavior category. The group behavior recognition model (i.e., Global-Individual Features Fusion Network, GIFFNet) includes: backbone network, fusion module, embedding layer, relational reasoning module, graph convolutional network, global average pooling layer and classifier.
[0090] The specific group behavior recognition process is as follows:
[0091] The cropped image is input into the backbone network in RGB mode. The backbone network uses VGG-16 (Visual Geometry Group 16-layer Network) or Inception-v3 (Google Inception-v3Network) to extract the global features of the image. , ,in represents the batch size, represents the time step, represents the channel dimension of global image features, and are the width and height of the image respectively. Then, according to the global features The image is cropped and aligned using RoI Align (region of interest alignment), and the region of interest is mapped to the global features to obtain the individual (in this embodiment, the individual refers to a single person) features containing appearance details. , ,in, The channel dimension representing individual features, Represents the side length of the cropped image. Indicates the number of target persons in a single image.
[0092] Next, the global features obtained and individual characteristics Sent to the fusion module to obtain fusion features, such as Figure 3As shown in the figure, the fusion module (GIFF, global - individual features fusion) includes: a convolutional layer, an attention mechanism module, a linear embedding layer, a Softmax layer, a layer normalization layer and a feedforward network.
[0093] The feature fusion process is as follows:
[0094] First, to better understand the importance of different regions in the image, we use the global features and individual characteristics Position encoding (PE) is performed. In order to reduce the number of parameters, the GIFF module adopts a bottleneck structure, and then the individual features and global features after position encoding are uniformly planned through a pointwise convolution layer (point convolution) with a 1×1 convolution kernel to plan the number of channels of the two features. The expression is as follows:
[0095]
[0096] in, and Represent the position encoding and pointwise convolution layer of 1×1 convolution kernel respectively, Represents individual features or global features after position encoding, After the above calculation, the output , Corresponding to individual characteristics, it is expressed as , Corresponding to the global feature, it is expressed as , , is the number of channels after unification.
[0097] In order to further refine the key information in Lianzhong features, the GIFF module adds a channel attention mechanism. Through the attention mechanism module, and The channel is used to capture the key frames in the video sequence and focus on the key figures in the group. The expression is as follows:
[0098]
[0099] in, Indicates the dimension size is The weight matrix of , They are The batch size and number of channels, To add individual features or global features after the channel attention mechanism, Corresponding to individual characteristics, it is expressed as , corresponding to the global feature, is expressed as , weights and features and Multiplying in the first two dimensions, according to and The importance of the features is weighted to obtain and .
[0100] Next, we will add the individual features after the channel attention mechanism Through a linear embedding layer, the dimension is reduced and the shape is adjusted to obtain , , the expression is as follows:
[0101]
[0102] in, represents a linear embedding layer;
[0103] The individual features that pass through the linear embedding layer And the global features after adding the channel attention mechanism Perform product fusion and obtain intermediate features through the Softmax layer , , the expression is as follows:
[0104]
[0105] Among them, Softmax represents the Softmax layer.
[0106] Then, the intermediate features Global features after adding channel attention mechanism After product fusion is performed to generate features containing global scene information, they are then combined with the individual features after the linear embedding layer. Perform residual calculation to retain the original information in the individual features and obtain the fusion feature that integrates the scene context information , the expression is as follows:
[0107]
[0108] Then the fusion features After layer normalization (LayerNorm), it is passed through the feedforward network and then layer normalization is performed again, and complex nonlinear transformation and regularization operations are performed to further enhance its expressive ability and obtain fusion features with enhanced representation ability. , , the expression is as follows:
[0109]
[0110] in, Representation layer normalization layer, represents a feed-forward network;
[0111] like Figure 2 As shown, the output of the GIFF module needs to be consistent with To facilitate the operation of these two features, a pointwise convolution layer with a 1×1 convolution kernel is used in the GIFF module. Restore the number of channels and adjust Make it The shape of is consistent, and the fusion feature finally output by the fusion module is obtained. , , the expression is as follows:
[0112]
[0113] Then, individual characteristics After the embedding layer and the output of the GIFF module Perform residual calculation to obtain the final fusion feature , then, using Calculate the relationship interaction matrix between characters to infer the category of group behavior, including:
[0114] First, calculate The appearance similarity and position difference between different character features in the image are used to obtain the relationship interaction diagram between the characters. The expression is as follows:
[0115]
[0116] in, The final fusion feature Individual characteristics in the space-time dimension, is the time step, is the number of target individuals in a single image, is the fusion function, For individuals With individuals The appearance similarity between For individuals With individuals The positional relationship between , The final fusion features are In the space-time dimension and individuals Appearance characteristics; The final fusion features are In the space-time dimension and individuals Location characteristics; For individuals With individuals An interaction diagram of the relationships between them.
[0117] Then, we use the pre-trained graph convolutional network (GCN) to map the relationship interaction graph between individuals. And the final fusion feature As input, the output of the graph convolutional network is combined with individual features After the residual calculation, the global average pooling layer and classifier (including individual action classifier and group behavior classifier) are sequentially passed to obtain the final group behavior category, which is expressed as follows:
[0118]
[0119] In the formula, is the classification result of group behavior, is the classifier, is the global average pooling layer.
[0120] Among them, the training formula of the graph convolutional network is as follows:
[0121]
[0122] in, Represents the relationship interaction diagram between individuals (characters), Representative The individual feature representation of the layer, . For the The learnable weights of the layer are centered. is the ReLU activation function. The propagation training of GCN can be designed as multiple layers. To avoid large computational complexity, this paper only uses one layer of GCN for feature training. Therefore, in the above formula That is .
[0123] Example 2
[0124] The present invention also provides a group behavior recognition system, comprising:
[0125] A data acquisition module, used for acquiring a video frame to be identified;
[0126] The behavior recognition module is used to obtain the group behavior category based on the video frame and the pre-trained group behavior recognition model;
[0127] The group behavior recognition model includes: a backbone network, a fusion module, an embedding layer, a relational reasoning module, a graph convolutional network, a global average pooling layer and a classifier.
[0128] The operations performed by the behavior recognition module include:
[0129] Based on the backbone network, the video frame is used as input to obtain global features ;
[0130] According to global characteristics Using regional interest alignment to obtain individual features ;
[0131] Based on the fusion module, the global features and individual characteristics As input for fusion, the fusion features finally output by the fusion module are obtained. ;
[0132] Based on the embedding layer, individual features are As input, the output of the embedding layer is combined with the fusion feature output of the fusion module. Perform residual calculation to obtain the final fusion feature ;
[0133] Based on the relational reasoning module, according to the final fusion features Get the relationship interaction diagram between individuals ;
[0134] Based on the pre-trained graph convolutional network, the relationship interaction graph between individuals And the final fusion feature As input, the output of the graph convolutional network is combined with individual features After the residual calculation, it passes through the global average pooling layer and the classifier in turn to obtain the final group behavior category.
[0135] The fusion module includes: a convolutional layer, an attention mechanism module, a linear embedding layer, a Softmax layer, a layer normalization layer and a feedforward network;
[0136] The operations performed by the fusion module include: and global features Position encoding is performed separately to add position information, and then the individual features after position encoding are and global features Channel planning is performed through the convolution layer, and the expression is as follows:
[0137]
[0138] in, represents the position code, represents a pointwise convolution layer with a 1×1 convolution kernel, Represents individual features or global features, After the above calculation, the output , Corresponding to individual characteristics, it is expressed as , corresponding to the global feature, is expressed as ;
[0139] Based on the attention mechanism module, the individual features after the convolution layer are and global features Add channel attention mechanism, the expression is as follows:
[0140]
[0141] in, represents the channel attention mechanism, To add individual features or global features after the channel attention mechanism, Corresponding to individual characteristics, it is expressed as , corresponding to the global feature, is expressed as ; Indicates the dimension size is The weight matrix of
[0142] The individual features after adding the channel attention mechanism After the linear embedding layer, we get , the expression is as follows:
[0143]
[0144] in, represents a linear embedding layer;
[0145] The individual features that pass through the linear embedding layer And the global features after adding the channel attention mechanism Perform product fusion and obtain intermediate features through the Softmax layer , the expression is as follows:
[0146]
[0147] Among them, Softmax represents the Softmax layer;
[0148] Then the intermediate features Global features after adding channel attention mechanism After product fusion, the individual features after the linear embedding layer are Perform residual calculation to obtain fusion features , the expression is as follows:
[0149]
[0150] Then the fusion features After layer normalization, the layers are normalized again through the feedforward network to obtain fusion features with enhanced representation capabilities. , the expression is as follows:
[0151]
[0152] in, Representation layer normalization layer, represents a feed-forward network;
[0153] Then the fusion features with enhanced representation capabilities are After the convolution layer, the fusion features finally output by the fusion module are obtained , the expression is as follows:
[0154] .
[0155] Example 3
[0156] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the aforementioned group behavior recognition method are implemented.
[0157] The training instructions of the group behavior recognition model in the present invention are as follows:
[0158] Data collection: The experimental data comes from the existing datasets Volleyball and Collective Activity. In practical applications, it is necessary to collect videos containing group behavior scenes and cut them into frames; for each image, the position and behavior category of the people involved in the group behavior are marked.
[0159] Dataset division: The processed video frame data is divided into training set and test set in a fixed ratio, with the ratio of the training set being about 70% and the ratio of the test set being about 30%.
[0160] Model training and testing: The number of training iterations of GIFFNet is set to 60, the initial learning rate lr is set to 1e-4, reduced by 1 / 2 every 10 rounds, and fixed to 1e-5 in the 40th round. The optimizer is Adam optimizer, and the optimization loss function is ,in represents the cross entropy loss function, and are the true label value and the predicted label value, respectively. and is a hyperparameter.
[0161] After calculating the cross entropy loss between the predicted action and the true label, its weight distribution is calculated using the above formula. and The value can be freely adjusted according to actual needs and final effect. Using the formula The loss function is based on the standard cross entropy loss and adopts a dynamic weight allocation strategy, which can solve the problem of imbalanced data sets. and The values of are 0.25 and 0.2 respectively.
[0162] Model evaluation: The two indicators for measuring the accuracy of group behavior recognition are multi-class classification accuracy (MCA) and mean per class accuracy (MPCA).
[0163] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0164] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
Claims
1. A group behavior recognition method, It is characterized in that include: Get the video frame to be recognized; According to the video frames, the group behavior category is obtained based on the pre-trained group behavior recognition model; The group behavior recognition model includes: a backbone network, a fusion module, an embedding layer, a relational reasoning module, a graph convolutional network, a global average pooling layer and a classifier; The backbone network is VGG-16 or Inception-v3, and the backbone network is used to extract the global features X of the image g ; The fusion module includes: a convolutional layer, an attention mechanism module, a linear embedding layer, a Softmax layer, a layer normalization layer and a feedforward network; For the global feature X g and for the global feature X g Individual features X obtained by mapping the region of interest i Position encoding is performed separately to add position information, and then the individual features and global features after position encoding are passed through the convolution layer for channel planning. The expression is as follows: X ′ =Conv 1×1 (PE(X)+X) Among them, PE represents position encoding, Conv 1×1 represents the pointwise convolution layer of 1×1 convolution kernel, X represents the individual feature or global feature after position encoding, and X is calculated by the above formula and outputted as X ′ , X ′ When corresponding to individual characteristics, it is represented as X ′ i , corresponding to the global feature, is represented by X ′ g ; Based on the attention mechanism module, the individual features X after the convolution layer are ′ i and the global feature X ′ g Add a channel attention mechanism, the expression is as follows: Among them, ChannelAtt represents the channel attention mechanism, To add individual features or global features after the channel attention mechanism, Corresponding to individual characteristics, it is expressed as Corresponding to the global feature, it is expressed as w b×c×1×1 Represents a weight matrix with dimension size b×c×1×1; The individual features after adding the channel attention mechanism After the linear embedding layer, we get The expression is as follows: Among them, Linear represents the linear embedding layer; The individual features that pass through the linear embedding layer And the global features after adding the channel attention mechanism Perform product fusion and obtain the intermediate feature X through the Softmax layer tmp , the expression is as follows: Among them, Softmax represents the Softmax layer; Then the intermediate feature X tmp Global features after adding channel attention mechanism After product fusion, the individual features after the linear embedding layer are Perform residual calculation to obtain fusion features The expression is as follows: Then the fusion features After layer normalization, the layers are normalized again through the feedforward network to obtain fusion features with enhanced representation capabilities. The expression is as follows: Among them, LN represents layer normalization layer, FFN represents feed-forward network; Then the fusion features with enhanced representation capabilities are After the convolution layer, the fusion feature X is finally output by the fusion module. fused , the expression is as follows:
2. A group behavior recognition method according to claim 1, It is characterized in that Based on the backbone network, the video frame is taken as input to obtain the global feature X g ; According to the global feature X g Use regional interest alignment to obtain individual features X i ; Based on the fusion module, the global feature X g and individual characteristics X i As input for fusion, the fusion feature X is finally output by the fusion module. fused ; Based on the embedding layer, the individual features X i As input, the output of the embedding layer is combined with the fusion feature X output by the fusion module. fused Perform residual calculation to obtain the final fusion feature F fused ; Based on the relational reasoning module, according to the final fusion feature F fused Get the relationship interaction graph G between individuals; Based on the pre-trained graph convolutional network, the relationship interaction graph G between individuals and the final fusion feature F fused As input, the output of the graph convolutional network is combined with the individual features X i After the residual calculation, it passes through the global average pooling layer and the classifier in turn to obtain the final group behavior category.
3. A group behavior recognition method according to claim 2, It is characterized in that According to the final fusion feature F fused And the preset calculation formula, the relationship interaction diagram X between individuals is obtained as follows: Among them, x i is the final fusion feature F fused Individual features in the spatiotemporal dimension, T is the time step, N is the number of target individuals in a single image, h is the fusion function, is the appearance similarity between individuals i and j, is the positional relationship between individuals i and j, They are the final fusion features F fused The appearance characteristics of individuals i and j in the space-time dimension; They are the final fusion features F fused The position characteristics of individuals i and j in the space-time dimension; G i,j is the interaction diagram between individual i and individual j.
4. A group behavior recognition method according to claim 2, It is characterized in that The expression for obtaining the group behavior category is as follows: Y=C(AvgPool(X GCN +X i )) Among them, Y is the final group behavior category, C is the classifier, AvgPool is the global average pooling layer, X GCN is the output of the graph convolutional network.
5. A group behavior recognition system, It is characterized in that The group behavior recognition system is used to implement the group behavior recognition method according to any one of claims 1 to 4, comprising: A data acquisition module, used for acquiring a video frame to be identified; The behavior recognition module is used to obtain the group behavior category based on the video frame and the pre-trained group behavior recognition model; The group behavior recognition model includes: a backbone network, a fusion module, an embedding layer, a relational reasoning module, a graph convolutional network, a global average pooling layer and a classifier.
6. A group behavior recognition system according to claim 5, It is characterized in that The operations performed by the behavior recognition module include: Based on the backbone network, the video frame is taken as input to obtain the global feature X g ; According to the global feature X g Use regional interest alignment to obtain individual features X i ; Based on the fusion module, the global feature X g and individual characteristics X i As input for fusion, the fusion feature X is finally output by the fusion module. fused ; Based on the embedding layer, the individual features X i As input, the output of the embedding layer is combined with the fusion feature X output by the fusion module. fused Perform residual calculation to obtain the final fusion feature F fused ; Based on the relational reasoning module, according to the final fusion feature F fused Get the relationship interaction graph G between individuals; Based on the pre-trained graph convolutional network, the relationship interaction graph G between individuals and the final fusion feature F fused As input, the output of the graph convolutional network is combined with the individual features X i After the residual calculation, it passes through the global average pooling layer and the classifier in turn to obtain the final group behavior category.
7. A group behavior recognition system according to claim 5, It is characterized in that The fusion module includes: a convolutional layer, an attention mechanism module, a linear embedding layer, a Softmax layer, a layer normalization layer and a feedforward network; The operations performed by the fusion module include: i and the global feature X g Position encoding is performed separately to add position information, and then the individual features X after position encoding are i and the global feature X g Channel planning is performed through the convolution layer, and the expression is as follows: X ′ =Conv 1×1 (PE(X)+X) Among them, PE represents position encoding, Conv 1×1 represents the pointwise convolution layer of 1×1 convolution kernel, X represents individual features or global features, and X is calculated by the above formula to output X ′ , X ′ When corresponding to individual characteristics, it is represented as X ′ i , corresponding to the global feature, is represented by X ′ g ; Based on the attention mechanism module, the individual features X after the convolution layer are ′ i and the global feature X ′ g Add a channel attention mechanism, the expression is as follows: Among them, ChannelAtt represents the channel attention mechanism, To add individual features or global features after the channel attention mechanism, Corresponding to individual characteristics, it is expressed as Corresponding to the global feature, it is expressed as w b×c×1×1 Represents a weight matrix with dimension size b×c×1×1; The individual features after adding the channel attention mechanism After the linear embedding layer, we get The expression is as follows: Among them, Linear represents the linear embedding layer; The individual features that pass through the linear embedding layer And the global features after adding the channel attention mechanism Perform product fusion and obtain the intermediate feature X through the Softmax layer tmp , the expression is as follows: Among them, Softmax represents the Softmax layer; Then the intermediate feature X tmp Global features after adding channel attention mechanism After product fusion, the individual features are combined with the linear embedding layer. Perform residual calculation to obtain fusion features The expression is as follows: Then the fusion features After layer normalization, the layers are normalized again through the feedforward network to obtain fusion features with enhanced representation capabilities. The expression is as follows: Among them, LN represents layer normalization layer, FFN represents feed-forward network; Then the fusion features with enhanced representation capabilities are After the convolution layer, the fusion feature X is finally output by the fusion module. fused , the expression is as follows:
8. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.
