Group behavior recognition method and device based on graph external attention enhancement and medium

Abstract

The application discloses a group behavior recognition method and device based on graph external attention enhancement and a medium, and relates to the technical field of computer vision. The method comprises the following steps: acquiring position information of human joints in each frame of video data; constructing a skeleton space-time graph; based on the skeleton space-time graph, locally modeling the joint nodes in the independent individual by using adaptive graph convolution to obtain local individual behavior features, globally modeling the joint nodes in the independent individual by using a global attention network to obtain global individual behavior features, integrating the local individual behavior features and the global individual behavior features to obtain behavior features of all individuals in the video data; modeling the correlation between the behavior features of different individuals by using graph external attention to capture the interaction relationship between the individuals; and adding and fusing the behavior features of all individuals and the interaction relationship between the individuals to perform group behavior recognition. The application can obtain a more comprehensive graph representation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and more particularly, to a group behavior recognition method and device based on graph external attention enhancement and a medium. BACKGROUND

[0002] With the continuous popularity of high-definition video monitoring, the amount of monitoring video data is rapidly growing. In the field of security, especially in public places, it is crucial to monitor the behavior of the crowd in real time to prevent accidents. In recent years, the development of computer vision technology has significantly improved the accuracy of individual behavior recognition, but there are still deficiencies in recognizing complex interactive behaviors of multiple people. In real life, common behaviors often involve interactions among multiple people, such as handshaking, hugging, and fighting, etc. Compared with single-person actions, group behaviors are more complex, involving not only individual behaviors of each participant, but also interactions, communication, and behavior adaptation among each other. In addition, there are behavioral habits and individual differences among participants, resulting in diverse and varied types of body movements, which further increases the difficulty of recognition. However, existing methods usually process individual behaviors independently, ignoring the direct interaction relationship between participants, and relying solely on a single skeleton graph cannot fully understand interpersonal interaction behaviors or accurately describe their features. Therefore, it is a key problem in the field of skeleton behavior recognition to model and analyze the interaction patterns in group behaviors as a whole.

[0003] The input data of human behavior recognition methods mainly includes two types: RGB data and skeleton data. Compared with RGB data, human skeleton data is not affected by illumination, background, and view angle changes, which makes human behavior recognition methods based on skeleton data have stronger robustness. With the rapid development of deep learning technology, many scholars at home and abroad have devoted to applying deep learning methods to human behavior recognition tasks based on skeleton sequences. These methods can be divided into three categories: methods based on recurrent neural networks, methods based on convolutional neural networks, and methods based on graph convolutional networks. The behavior recognition method based on recurrent neural network first represents the joint node data as a vector sequence, and each vector sequence contains the position information of all joint nodes in a time frame. Then the vector sequence is input into the behavior recognition network with recurrent neural network as the backbone, and finally the behavior recognition result is obtained. The method based on convolutional neural network converts the human skeleton sequence from a vector sequence to a pseudo image, and the width, height and channel of the pseudo image represent the joints, frames and coordinates of the skeleton sequence respectively, and then the pseudo image is input into the convolutional neural network for classification. The method based on graph convolution represents the joint node data as a topological graph, and the vertices are connected by spatial edge lines in the spatial domain, and the corresponding joints between adjacent frames are connected by time edge lines in the time domain, and the attribute feature of each joint node is a spatial coordinate vector. Then the topological graph is input into the behavior recognition network with graph convolution network as the backbone, and finally the behavior recognition result is obtained. The skeleton data essentially exists in the form of a topological graph structure, and the graph convolution is a deep learning method based on graph structure, which can efficiently extract and classify the features of human skeleton data. Therefore, the method based on graph convolution has become the mainstream for processing skeleton data.

[0004] In the process of implementing the present application, the inventors found that the prior art has the following problems:

[0005] (1) In the process of processing group behavior, most methods are based on individuals and process each skeleton graph separately, ignoring the correlation between the interaction parts. Due to the complex interaction patterns between different individuals in the interaction process, it is difficult to capture the correlation between behaviors by relying only on separate skeleton graphs, resulting in an incomplete and inaccurate understanding of the interaction behavior.

[0006] (2) It is unable to simultaneously capture the dynamic interaction and correlation between local and global information. How to effectively integrate and utilize the inherent local clues (such as local individual actions, detail changes) and global clues (such as overall action patterns, multi-person behavior coordination relationships) in multi-person activities remains a major technical challenge. SUMMARY

[0007] To solve the above technical problems, the present application provides a group behavior recognition method based on graph external attention enhancement, device and medium.

[0008] In a first aspect, the present invention provides a method for recognizing group behavior based on graph external attention enhancement, the method comprising:

[0009] Obtain the position information of human joints in each frame of video data;

[0010] Based on the positional information of human joints in each frame, a spatiotemporal graph of the skeleton is constructed.

[0011] Based on the spatiotemporal graph of the skeleton, local modeling of the key points in the independent individuals is performed using adaptive graph convolution to obtain local individual behavior features. Global attention network is used to globally model the key points in the independent individuals to obtain global individual behavior features. The local individual behavior features and global individual behavior features are integrated to obtain the behavior features of all individuals in the video.

[0012] By utilizing graph external attention, we can model the correlations between behavioral characteristics of different individuals and capture the interaction relationships between individuals;

[0013] Group behavior identification is achieved by combining and integrating the behavioral characteristics of all individuals and the interactions between them.

[0014] Furthermore, the positional information of human joints in each frame of the video data is obtained, including:

[0015] The pose skeleton algorithm is used to estimate the pose of a group of people in video data, and the three-dimensional position coordinates of human joints in each frame are obtained as position information.

[0016] Furthermore, based on the positional information of the human joints in each frame, a spatiotemporal graph of the skeleton is constructed, including:

[0017] Construct an individual spatial map of each individual in each frame according to the natural structure of the human body;

[0018] In the individual space graph, nodes representing the same joint of the same individual in adjacent frames are connected to obtain the skeleton spatiotemporal graph.

[0019] Furthermore, based on the aforementioned spatiotemporal graph of the skeleton, adaptive graph convolution is used to locally model the key points in independent individuals, and the calculation process for obtaining the local individual behavioral features is expressed as follows:

[0020]

[0021] In the formula, f out A represents local individual behavioral characteristics. k M represents the adjacency matrix, used to characterize the physical structure of the human body. k S represents the mask matrix. k Let f represent the similarity matrix, used to characterize the similarity between two nodes.in Let W represent the input feature, W represent the weight function, and k represent the k subsets into which the node is divided (the root node itself, the centripetal set, and the centrifugal set).

[0022] Furthermore, a global attention network is used to globally model the key points within an individual, resulting in global individual behavioral features, including:

[0023] Based on the data of frame t, for node Using a three-parameter matrix to pair nodes Perform a linear transformation to obtain the query vector key-value vector Sum value vector

[0024] The connection strength between nodes is calculated based on the inner product of the query vector and the key vector.

[0025] Based on the connection strength and value vector between nodes Global individual behavioral characteristics are calculated.

[0026] Furthermore, the connection strength between nodes is calculated using the dot product of the query vector and the key vector, as follows:

[0027]

[0028] In the formula, Let represent the query vector of the i-th node in the t-th frame. Let represent the key-value vector of the j-th node in the t-th frame. Let T represent the connection strength between the i-th node and the j-th node in the t-th frame, and let T represent the matrix transpose.

[0029] Furthermore, based on the connection strength and value vector between nodes... The global individual behavioral characteristics are calculated using the following formula:

[0030]

[0031] In the formula, Represents a node Global individual behavioral characteristics, d k The dimensions of the query vector and key-value vector are represented by , and softmax represents the normalization exponential function.

[0032] Furthermore, external attention is used to model the associations between behavioral characteristics of different individuals, capturing the interaction relationships between individuals, including:

[0033] Using the behavioral characteristics of different individuals as input graphs, a learnable parameter independent of the input graphs is set. The learnable parameter is an external unit with multiple individual graphs, where each individual graph contains the same number of nodes and serves as shared memory for all input graphs.

[0034] Based on the input graph and learnable parameters, external attention is obtained using the following formula:

[0035] A Inter =norm(XU T )∈R n×n×n

[0036] In the formula, A Inter Let X represent the similarity matrix between nodes and external units of the input graph, X represent the input graph, U represent the learnable parameters, R represent the real space, N represent the number of individual graphs, T represent the matrix transpose, norm represents the normalization function, and n represent the number of joints.

[0037] A is obtained through the following formula. Inter Perform double normalization to obtain the external attention matrix:

[0038]

[0039] In the formula, Let represent the attention value of joint i to joint j, and exp represent the natural exponential function. α represents the attention value of joint k to joint j. i,j α represents the normalized attention value. i,j Represents the elements that make up the external attention matrix;

[0040] The interaction relationships between individuals are obtained by using the inner product of the external attention matrix and the learnable parameter U.

[0041] Secondly, the present invention provides a group behavior recognition device based on graph external attention enhancement, the device comprising:

[0042] The data acquisition unit is configured to acquire the position information of human joints in each frame of video data;

[0043] The spatiotemporal graph construction unit is configured to construct a skeleton spatiotemporal graph based on the positional information of human joints in each frame.

[0044] The behavior feature acquisition unit is configured to, based on the skeleton spatiotemporal graph, use adaptive graph convolution to locally model the key points in an independent individual to obtain local individual behavior features, use a global attention network to globally model the key points in an independent individual to obtain global individual behavior features, and integrate the local individual behavior features and global individual behavior features to obtain the behavior features of all individuals in the video.

[0045] The interaction relationship acquisition unit is configured to model the association between the behavioral features of different individuals using graph external attention, thereby capturing the interaction relationships between individuals;

[0046] The group behavior recognition unit is configured to add and fuse the behavioral characteristics of all individuals and the interaction relationships between individuals to perform group behavior recognition.

[0047] Thirdly, the present invention provides a readable storage medium storing one or more programs that can be executed by one or more processors to implement the method described above.

[0048] The present invention has at least the following beneficial effects:

[0049] This invention also considers that graph convolutional neural networks can directly process topological graphs such as skeleton data, effectively capturing the spatiotemporal relationships and contextual information within the skeleton data. The Transformer structure allows nodes to interact with all other nodes in the graph, facilitating direct modeling of long-distance relationships while addressing typical problems in graph convolution such as oversmoothing. Therefore, this invention introduces graph convolutional networks and transformer structures to construct two feature extraction networks: one for intra-graph and one for extra-graph. These networks learn intra-graph and inter-graph correlations and combine inter-graph correlations with intra-graph local structure and global information. This allows for a more comprehensive graph representation compared to most existing methods. This invention demonstrates significant advantages in scenarios such as dense crowd analysis and complex action understanding, providing a highly reliable solution for group behavior recognition in fields such as intelligent surveillance and human-computer interaction. Attached Figure Description

[0050] Figure 1 A flowchart illustrating an overall process for group behavior recognition based on graph external attention enhancement according to an embodiment of the present invention is shown.

[0051] Figure 2 An example of a skeletal spatiotemporal graph established according to an embodiment of the present invention is shown (taking two people as an example);

[0052] Figure 3 A schematic diagram of an external attention structure according to an embodiment of the present invention is shown;

[0053] Figure 4A structural diagram of a group behavior recognition device based on graph external attention enhancement according to an embodiment of the present invention is shown. Detailed Implementation

[0054] To enable those skilled in the art to better understand the technical solutions of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The embodiments of the present invention will be further described in detail below with reference to the accompanying drawings and specific examples, but this is not intended to limit the present invention. If there is no necessary sequential relationship between the various steps described herein, the order in which they are described as examples should not be considered a limitation. Those skilled in the art should understand that the order can be adjusted, as long as it does not disrupt the logical consistency between them and render the entire process impossible.

[0055] This invention provides a method for recognizing group behavior based on graph-external attention enhancement, such as... Figure 1 A flowchart illustrating an overall process for a group behavior recognition method based on graph external attention enhancement according to an embodiment of the present invention is shown. This group behavior recognition method based on graph external attention enhancement includes the following steps S10-S50.

[0056] S10: Obtain the position information of human joints in each frame of the video data.

[0057] It should be noted that the video data can come from high-definition surveillance video obtained through legal and compliant means. In step S10, existing pose skeleton algorithms can be used to estimate the pose of the group appearing in the video, obtaining the three-dimensional position coordinates of the human joints in each frame. Existing pose skeleton algorithms can include, for example, OpenPose, AlphaPose, VideoPose3D, and MediaPipe BlazePose. This embodiment does not specifically limit the pose skeleton algorithm. In the actual implementation of this invention, the pose skeleton algorithm used includes, but is not limited to, the algorithms listed above.

[0058] S20: Construct a spatiotemporal graph of the skeleton based on the positional information of the human joints in each frame.

[0059] In this embodiment, a spatiotemporal sequence graph of the skeleton is constructed according to the video frame order. Each individual in each frame is constructed with an individual spatial graph based on the natural structure of the human body; after the spatial topology graph of the skeleton sequence is constructed, the nodes representing the same joint of the same individual in adjacent frames are connected to form the spatiotemporal graph of the skeleton.

[0060] For example, a skeletal spatiotemporal graph can be constructed through the following steps:

[0061] S21: Construct individual spatial graphs.

[0062] Specifically, when constructing the individual spatial graph, the positional information of the human joints in each frame is first obtained, including the 3D coordinates of multiple joints. Each human joint corresponds to a node, and the node features include 3D coordinates and other attributes (such as confidence). Secondly, edge connection rules are defined based on the natural structure of the human body. For example, static edges (such as a chain connection of "wrist-elbow-shoulder") are defined based on human anatomy. Finally, the individual spatial graph G is output. t =(V t E t ), where V t Let E be a set of nodes. t Let it be the set of edges.

[0063] S22: Based on the individual spatial graph, connect the nodes representing the same joint of the same individual in adjacent frames to form a skeleton spatiotemporal graph.

[0064] Specifically, based on step S21, a sequence of individual spatial graphs {G1,G2,...,G} for T consecutive frames can be obtained. T}

[0065] Secondly, for each joint (node), its corresponding nodes in adjacent frames are connected via temporal edges, such as connecting the j-th node in frame t to the j-th node in frame t+1. Motion features, such as the displacement vector Δx = x, can be attached to these temporal edges. t+1 -x t Finally, by merging the nodes and edges of all frames, we obtain the skeleton spatiotemporal graph, which can be represented as G. ST =(V ST E ST ), The set of nodes representing the skeletal spatiotemporal graph. E temp Represents the time edge set.

[0066] Taking a two-person example, the established skeletal spacetime diagram is as follows: Figure 2 As shown.

[0067] S30: Based on the spatiotemporal graph of the skeleton, local modeling of the key points in the independent individuals is performed using adaptive graph convolution to obtain local individual behavior features. Global attention network is used to globally model the key points in the independent individuals to obtain global individual behavior features. The local individual behavior features and global individual behavior features are integrated to obtain the behavior features of all individuals in the video data.

[0068] In this embodiment, an adaptive graph convolutional network and a global attention network can be constructed using a graph convolutional network and a transformer architecture. The adaptive graph convolutional module is used to obtain the spatial relationships between adjacent and non-adjacent joints. This module parameterizes two types of graphs: a fixed graph representing the human body's physical structure and a dynamic graph generated based on each sample in the dataset. The spatial attention module captures the connections between each pair of joints in each frame and dynamically establishes these connections based on specific behaviors. The temporal attention module utilizes the changes in the embedding of the same node across different frames to obtain the correlation between different frames.

[0069] Adaptive graph convolution can be defined as:

[0070]

[0071] In the formula, f out A represents local individual behavioral characteristics. k M represents the adjacency matrix, used to characterize the physical structure of the human body. k This represents a mask matrix with a shape and size of N×N, S k Let f represent the similarity matrix, used to characterize the similarity between two nodes. in Let W represent the input feature, W represent the weight function, and k represent the k subsets into which the node is divided (the root node itself, the centripetal set, and the centrifugal set).

[0072] In the formula above, the three matrices (adjacency matrix, mask matrix, and similarity matrix) are combined using addition instead of matrix multiplication. This is because when the adjacency matrix A... k When the corresponding element in the multiplication operation is 0, the 0 cannot be eliminated, which means that no new relation can be generated.

[0073] It should be noted that the adjacency matrix can be the spatial adjacency matrix of the skeleton spatiotemporal graph, such as A. k ∈R N×N N represents the number of joints, A k It can be represented as:

[0074]

[0075] Where A k (i,j)=1 indicates that the i-th node and the j-th node are spatially connected. A k (i,j)=0 indicates that there is no spatial connection between the i-th node and the j-th node.

[0076] The mask matrix can be determined based on the spatiotemporal adjacency matrix of the skeleton spatiotemporal graph. The spatiotemporal adjacency matrix... Represents the Kronecker product, AT The temporal adjacency matrix is ​​represented as:

[0077]

[0078] Where t k The value represents the size of the time window, and Δt represents the time interval.

[0079] The mask matrix can be directly taken as a binary form of the spatiotemporal adjacency matrix, such as M. k It can be represented as:

[0080]

[0081] Where A ST (i,j)>0 indicates that the i-th node and the j-th node are spatiotemporally connected.

[0082] The similarity matrix is ​​calculated based on the node feature matrix of the skeleton spatiotemporal graph. For example, the similarity between two nodes can be represented by calculating the cosine similarity or dot product attention, thereby combining the similarity of all node pairs to form a similarity matrix.

[0083] Global attention networks automatically capture the relationships between any nodes within each frame. Given the data in frame t, for each node... Using a three-parameter matrix The query vector is obtained by performing a linear transformation on the node. key-value vector value vector These three parameter matrices apply to all nodes in a frame. The connection strength between each pair of nodes can be calculated using the inner product of the query vector and the key vector, as follows:

[0084]

[0085] In the formula, Let represent the query vector of the i-th node in the t-th frame. Let represent the key-value vector of the j-th node in the t-th frame. Let T represent the connection strength between the i-th node and the j-th node in the t-th frame, and let T represent the matrix transpose.

[0086] node The final embedding form (i.e., global individual behavioral features) is obtained by weighting and summing its value vectors.

[0087]

[0088] In the formula, Represents a node Global individual behavioral characteristics, d k The dimensions of the query vector and key-value vector are represented by , and softmax represents the normalization exponential function.

[0089] For example, the calculation process of integrating the local individual behavioral features and the global individual behavioral features can be represented as follows:

[0090] h (m) =AttnPool([f out ||z out ])

[0091] In the formula, h (m) Let z represent the behavioral characteristics of the m-th individual. out This represents global individual behavioral features, AttnPool represents attention pooling, and || represents feature concatenation.

[0092] The behavioral characteristics of all individuals in the video data can then be represented as {h} (1) ,...,h (m) ,...,h (M)}, where h (1) and h (M) This represents the behavioral characteristics of the 1st and Mth individuals.

[0093] S40: Use graph external attention to model the associations between behavioral characteristics of different individuals and capture the interaction relationships between individuals.

[0094] like Figure 3 The diagram shows a schematic of the external attention structure. Step S40 calculates the correlation between individuals based on the individual behavioral features obtained in step S30. Individual behavioral features are composed of all node features in the individual graph. External attention is obtained by calculating the relationship between the node features of the input graph and external units (i.e., other individual graphs outside the current graph).

[0095] A Inter =norm(XU T )∈R N×n×n

[0096] In the formula, A Inter Let X represent the similarity matrix between nodes and external units of the input graph, X represent the input graph, U represent the learnable parameters, and U represent the external unit with N individual graphs (each individual graph contains the same number of nodes), which can be regarded as the shared memory of all input graphs. Let R represent the real space, N represent the number of individual graphs, T represent the matrix transpose, norm represent the normalization function, and n represent the number of joints.

[0097] To reduce the amount of computation, after obtaining A Inter Later, for AInter Double normalization is performed, which involves normalizing both rows and columns separately. The calculation process for double normalization is as follows:

[0098]

[0099] In the formula, Let represent the attention value of joint i to joint j, and exp represent the natural exponential function. α represents the attention value of joint k to joint j. i,j α represents the normalized attention value. i,j This represents the elements that make up the external attention matrix.

[0100] Finally, the group behavior features, with X as the input graph, can be obtained using the external attention matrix A' Inter ∈R N×n The inner product with the outer element yields:

[0101] F out =A' Inter ·U

[0102] In the formula, F out It represents the interaction between individuals.

[0103] S50: Combine and integrate the behavioral characteristics of all individuals and the interaction relationships between individuals to identify group behavior.

[0104] In this embodiment, individual behavioral features are used to describe the independent actions of each individual (such as walking, waving). Interaction relationship features are used to quantify the social influence between individuals (such as following, collaborating). For example, if the behavioral features of all individuals form an individual feature matrix H1, and the interaction relationships between all individuals form an interaction feature matrix H2, then the method of adding and fusing the behavioral features of all individuals and the interaction relationships between individuals can be H = H1 + H2. This is a residual addition (direct summation) method, where H represents the fused feature. This addition fusion method is particularly suitable for scenarios where individual behavioral features and interaction relationship features are highly complementary, such as synchronized movements in dance performances or interactive movements in daily social interactions. Of course, gated addition can also be used instead of residual addition. Gated addition is more suitable for scenarios where the importance of individual behavioral features and interaction relationship features differs greatly, such as leader-led behavior in emergency situations. Based on the obtained fused feature H, it can be further processed through average pooling and / or attention pooling to obtain the aggregated group feature h. g The aggregated group features are fed into the classifier, which outputs the probability distribution of group behavior categories.

[0105] For example, the formula for average pooling is:

[0106]

[0107] In the formula, n represents the individual number, N represents the number of individuals, and H[n] represents the fusion characteristics of individual n.

[0108] The formula for calculating the classifier is:

[0109] y = softmax(W c h g +b)

[0110] In the formula, y represents the probability distribution of group behavior categories, and W c Here is the classification weight matrix, b is the bias, and W is the weight matrix. c and b can be determined by training the classifier in advance.

[0111] Taking a football match as an example. For a segment of video footage from a football match, after processing using the steps described above, the individual behavioral characteristics of player A and player B are identified as follows:

[0112] Player A: Running at high speed, leg swing (offensive intention);

[0113] Player B: Lateral movement, arm extension (defensive action).

[0114] Identify the interaction characteristics between player A and player B:

[0115] Interaction characteristics of player A: frequent position exchanges (cooperative relationship) between player A and forward C;

[0116] The interaction characteristics of player B: player B continuously approaches player A (adversarial relationship).

[0117] Based on the individual behavioral characteristics and interaction characteristics of player A and player B, the final classification is "fast counter-attack" tactic.

[0118] Understandably, the application scenarios exemplified above are merely examples and do not imply that the method proposed in this invention must be applied to the above application scenarios. In other embodiments, the method proposed in this invention can also be applied to other application scenarios such as security monitoring, social behavior analysis, analysis of other sports activities, and analysis of cultural and artistic activities.

[0119] This invention also provides a group behavior recognition device based on graph-outside attention enhancement, such as... Figure 4 As shown, the device includes:

[0120] The data acquisition unit 401 is configured to acquire the position information of human joints in each frame of video data;

[0121] Spatiotemporal graph construction unit 402 is configured to construct a skeleton spatiotemporal graph based on the positional information of human joints in each frame;

[0122] The behavior feature acquisition unit 403 is configured to, based on the skeleton spatiotemporal graph, use adaptive graph convolution to locally model the key points in an independent individual to obtain local individual behavior features, use a global attention network to globally model the key points in an independent individual to obtain global individual behavior features, and integrate the local individual behavior features and global individual behavior features to obtain the behavior features of all individuals in the video.

[0123] The interaction relationship acquisition unit 404 is configured to model the association between the behavioral features of different individuals using graph external attention, and capture the interaction relationships between individuals;

[0124] The group behavior recognition unit 405 is configured to add and fuse the behavioral characteristics of all individuals and the interaction relationships between individuals to perform group behavior recognition.

[0125] It should be noted that the structures of the various group behavior recognition devices based on graph extrinsic attention enhancement described in this embodiment belong to the same technical concept as the previously described group behavior recognition methods based on graph extrinsic attention enhancement, and achieve the same beneficial effects through the same principle, which will not be elaborated here.

[0126] This invention also provides a readable storage medium storing one or more programs that can be executed by one or more processors to implement the methods described in any of the above embodiments.

[0127] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for recognizing group behavior based on graph-external attention enhancement, characterized in that, The method includes: Obtain the position information of human joints in each frame of video data; Based on the positional information of human joints in each frame, a spatiotemporal graph of the skeleton is constructed. Based on the spatiotemporal graph of the skeleton, local modeling of the key points in the independent individuals is performed using adaptive graph convolution to obtain local individual behavior features. Global attention network is used to globally model the key points in the independent individuals to obtain global individual behavior features. The local individual behavior features and global individual behavior features are integrated to obtain the behavior features of all individuals in the video data. By utilizing graph external attention, we can model the correlations between behavioral characteristics of different individuals and capture the interaction relationships between individuals; Group behavior identification is achieved by summing and integrating the behavioral characteristics of all individuals and the interaction relationships between them. By utilizing graph external attention, we can model the associations between behavioral characteristics of different individuals and capture the interaction relationships between individuals, including: Using the behavioral characteristics of different individuals as input graphs, a learnable parameter independent of the input graphs is set. The learnable parameter is an external unit with multiple individual graphs, where each individual graph contains the same number of nodes and serves as shared memory for all input graphs. Based on the input graph and learnable parameters, external attention is obtained using the following formula: ； In the formula, This represents the similarity matrix between nodes and external cells in the input graph. Indicates the input image. U Indicates learnable parameters, Let N represent the real number space, and let N represent the number of individuals in the graph. T Indicates matrix transpose. Represents the normalization function. n Indicates the number of joints; The following formula is used to... Perform double normalization to obtain the external attention matrix: ； ； ； In the formula, Indicates key points i For joints j Attention value, This represents the natural exponential function. Indicates key points k For joints j Attention value, This represents the normalized attention value. Represents the elements that make up the external attention matrix; Using external attention matrix and learnable parameters U The inner product yields the interaction relationships between individuals.

2. The method according to claim 1, characterized in that, Obtain the position information of human joints in each frame of the video data, including: The pose skeleton algorithm is used to estimate the pose of a group of people in video data, and the three-dimensional position coordinates of human joints in each frame are obtained as position information.

3. The method according to claim 1, characterized in that, Based on the positional information of human joints in each frame, a spatiotemporal graph of the skeleton is constructed, including: Construct an individual spatial map of each individual in each frame according to the natural structure of the human body; In the individual space graph, nodes representing the same joint of the same individual in adjacent frames are connected to obtain the skeleton spatiotemporal graph.

4. The method according to claim 1, characterized in that, Based on the aforementioned spatiotemporal graph of the skeleton, adaptive graph convolution is used to locally model the key points in an independent individual, and the calculation process for obtaining the local individual behavioral features is expressed as follows: ； In the formula, Indicates local individual behavioral characteristics, This represents the adjacency matrix, used to characterize the physical structure of the human body. Represents the mask matrix, A similarity matrix is ​​used to characterize the similarity between two nodes. Indicates input features, Represents the weighting function. This indicates that the nodes are divided into A subset.

5. The method according to claim 1, characterized in that, By using a global attention network to globally model the key points of an individual, global individual behavioral features are obtained, including: Based on the t Frame data, for nodes Using a three-parameter matrix to analyze nodes Perform a linear transformation to obtain the query vector Key-value vectors Sum value vector ; The connection strength between nodes is calculated based on the inner product of the query vector and the key vector. Based on the connection strength and value vector between nodes Global individual behavioral characteristics are calculated.

6. The method according to claim 5, characterized in that, The connection strength between nodes is calculated using the dot product of the query vector and the key vector, as follows: ； In the formula, Indicates the first t The first frame i The query vector for each node. Let represent the key-value vector of the j-th node in the t-th frame. Indicates the first t The first frame i The node and the first j The connection strength between nodes T This indicates the matrix transpose.

7. The method according to claim 6, characterized in that, Based on the connection strength and value vector between nodes The global individual behavioral characteristics are calculated using the following formula: ； In the formula, Represents a node Global individual behavioral characteristics This represents the dimension of the query vector and the key-value vector. This represents the normalized exponential function.

8. A group behavior recognition device based on graph-external attention enhancement, characterized in that, The device includes: The data acquisition unit is configured to acquire the position information of human joints in each frame of video data; The spatiotemporal graph construction unit is configured to construct a skeleton spatiotemporal graph based on the positional information of human joints in each frame. The behavior feature acquisition unit is configured to, based on the skeleton spatiotemporal graph, use adaptive graph convolution to locally model the key points in an independent individual to obtain local individual behavior features, use a global attention network to globally model the key points in an independent individual to obtain global individual behavior features, and integrate the local individual behavior features and global individual behavior features to obtain the behavior features of all individuals in the video. The interaction relationship acquisition unit is configured to model the association between the behavioral features of different individuals using graph external attention, thereby capturing the interaction relationships between individuals; The group behavior recognition unit is configured to add and fuse the behavioral characteristics of all individuals and the interaction relationships between individuals to perform group behavior recognition; By utilizing graph external attention, we can model the associations between behavioral characteristics of different individuals and capture the interaction relationships between individuals, including: Using the behavioral characteristics of different individuals as input graphs, a learnable parameter independent of the input graphs is set. The learnable parameter is an external unit with multiple individual graphs, where each individual graph contains the same number of nodes and serves as shared memory for all input graphs. Based on the input graph and learnable parameters, external attention is obtained using the following formula: ； In the formula, This represents the similarity matrix between nodes and external cells in the input graph. Indicates the input image. U Indicates learnable parameters, Let N represent the real number space, and let N represent the number of individuals in the graph. T Indicates matrix transpose. Represents the normalization function. n Indicates the number of joints; The following formula is used to... Perform double normalization to obtain the external attention matrix: ； ； ； In the formula, Indicates key points i For joints j Attention value, This represents the natural exponential function. Indicates key points k For joints j Attention value, This represents the normalized attention value. Represents the elements that make up the external attention matrix; Using external attention matrix and learnable parameters U The inner product yields the interaction relationships between individuals.

9. A non-transitory computer-readable storage medium storing instructions, characterized in that, When the instructions are executed by the processor, the method according to any one of claims 1 to 7 is performed.

Images