A method for modeling high-order relationships between topological nodes for topological behavior recognition

Abstract

The application provides a high-order relationship modeling method between topological nodes for topological behavior recognition, designs an iterative modeling method from low-order relationship to high-order relationship based on recursion, and realizes the completeness of modeling through the recursion self-similarity and scalability. The application models the high-order relationship of the completeness through generating the complete dependence between the low-order relationships, extracts more extensive and abstract individual features, effectively utilizes the self-similarity and iteration of the recursion, obtains the modeling complexity of the high-order relationship which is linearly related to the order, thereby avoids the factorial order complexity caused by the introduction of the completeness hyperedge, obtains the advanced features of the individuals in the group containing the high-order information, optimizes the modeling process of the graph neural network, and is beneficial to the group behavior recognition task.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of graph neural network methods for topological behavior recognition, and particularly to a graph convolution method based on modeling high-order relationships between topological nodes. BACKGROUND

[0002] Topology is a data representation method in machine learning, which is used to represent entities and the relationships between entities. It consists of nodes and edges, where nodes represent entities and edges represent relationships between entities. Skeleton-based action recognition is a kind of topological behavior recognition, where nodes are skeletal joints, and the presentation of a skeletal action is jointly expressed and carried by multiple skeletal joints. Modeling low-order relationships between skeletal joints can only distinguish some simple actions with a small scope, such as handshaking. However, some actions with multiple joints, a wide range of action, and complex joint coordination, such as specific actions in street dancing, can only be distinguished by high-order relationships between skeletal joints, because high-order relationships fully exploit the mutual dependence and coordination between skeletal joints. Group behavior recognition is also a kind of topological behavior recognition, where nodes are individuals, and the presentation of group behavior has sociality and groupization. It not only focuses on individual performance, but also values the complex connections between groups. If only low-order relationships between individuals are modeled, only some simple group activities, such as conversations, can be distinguished. The number of participants in complex group behavior is random, the participation form is diverse, and the dependence between individuals is deep, so high-order relationships between individuals are needed to capture this complex behavior form and characteristics. Generating high-order relationships to capture the details and driving of group behavior in a detailed and complete manner is crucial in the fields of intelligent security and automatic control.

[0003] Graph neural networks, particularly graph convolutional neural networks, play a crucial role in recognizing topological behavior, helping models better understand and utilize topological information. In the process of relying on graph convolutional neural networks to process behavior information based on topology, the core step is to model the relationships between topological nodes, as this relates to the transmission of node messages and the aggregation of node features. Most existing works model the relationships between nodes by using predefined prior adjacency or semantic-based dynamic adjacency to achieve the propagation and aggregation of node features, but such relationships belong to low-order relationships between nodes, only considering the action between isolated nodes, thus having the limitations of locality and low order.

[0004] To break through the above limitations, some works introduce hyper-edges to represent the complex many-to-many relationships between multiple nodes, which plays a certain role in modeling the high-order characteristics of nodes. However, the setting of hyper-edges is incomplete, that is, only certain hyper-edges between nodes are considered instead of all hyper-edges, because the factorial-level complexity caused by introducing all hyper-edges is unacceptable. Therefore, it is inevitable to cause the loss of effective high-order relationships between nodes, resulting in the loss of part of the high-order characteristics of nodes. SUMMARY

[0005] In order to overcome the deficiencies of the prior art, the present application provides a high-order relationship modeling method between topological nodes for topological behavior recognition. In order to fully and completely model the high-order relationship between nodes to extract the high-order characteristics of nodes, the present application designs an iterative modeling method from low-order relationship to high-order relationship based on recursion to realize the completeness of modeling with recursive self-similarity and scalability. The key design of the present application is to recursively model high-order relationships by capturing the complete dependence between low-order relationships. The present application models the complete high-order relationship with feasible complexity through recursive means.

[0006] The specific steps of the technical solution adopted by the present application to solve its technical problems are as follows:

[0007] S1. Extracting the first-order relationship between nodes;

[0008] Given the original features of the key nodes in the skeleton or the single nodes in the group F , wherein is the total number of individuals, is the individual feature dimension, and the original features of the nodes are input into the first-order relationship modeling function to obtain the first-order relationship between the key nodes or the single nodes , wherein is the relationship feature dimension, and for the nodes and the nodes , the first-order relationship modeling process is , the first-order relationship between the key nodes or the single nodes;

[0009] S2. Modeling the complete high-order relationship between nodes;

[0010] S2.1. First, the first-order relationship extracted in step S1 is embedded into the first-order modeling element by using the first-order embedding function for second-order modeling, which corresponds to the 2.1 area in Figure 1 and the a area in Figure 2 ;

[0011] S2.2 Then use the first-order dependency generation function Generate complete dependencies between first-order modeling elements For the S-th first-order modeling element and the t First-order modeling element In other words, the dependency generation process between them is as follows: This process corresponds to Figure 1 Region 2.2 and Figure 2 Middle b region;

[0012] S2.3 Finally, the first-order complete dependency obtained in step S2.2 is adopted. Aggregate first-order modeling element This yields complete second-order relations between key points or individual nodes. , For the second-order relationship between joint i and joint j, or between single person i and single person j, this process corresponds to Figure 1 Region 2.3 and Figure 2 In region c, steps S2.2 and S2.3 not only perceived the co-occurrence structure between first-order relations, but also captured the association patterns of binary node groups;

[0013] S2.4 follows the same logic as S2.1 to S2.3, and so on up to the (i+1)th order completeness relation modeling, starting with the i-th order embedding function. The i-th order relation Embedded as i-th order modeling element ,correspond Figure 2 The region a is then used with the i-th order dependency generation function. generate Complete dependencies between level modeling primitives ,correspond Figure 2 In region b, the final i-th order complete dependency is used. Aggregate i-th order modeling element This yields the completeness of the i+1 order relation for each key or individual node. ,correspond Figure 2 region c;

[0014] S2.5 Iteratively find the given maximum order by recursively applying the steps in S2.4. O Completeness of all order relationships between key points or individuals , For the k-th order relationship between joint i and joint j, or between single person i and single person j, corresponding to Figure 1In region 2.5, since modeling higher-order relations is achieved by capturing the complete dependencies between lower-order relations, the higher-order relations obtained by modeling are complete. Furthermore, the modeling of higher-order relations is based on recursive increment, so its modeling complexity increases linearly with the order rather than by factorial.

[0015] S3. Aggregate high-order features using completeness high-order relations;

[0016] S3.1 follows the paradigm of graph convolution, aggregating node features through the adjacency matrix. Where H is the parameter matrix and A is the adjacency matrix. and These are the node input features and node output features, respectively, employing the high-order completeness relationships between key points or individuals modeled in S2. As the original node features of key points in the aggregate skeleton of the adjacency matrix A or of a single person in a group. To obtain the high-order features of joints in a skeleton or nodes of a single person in a group. E is more than A O and Ce Dimensionality, therefore, performing graph convolution requires adaptation. O As a batch processing dimension, it independently generates the features of individuals at various levels. Ce Dimensional adaptation has two approaches: one is to perform channel common aggregation, where all channels of F use a single adjacency matrix to perform feature aggregation; specifically, it involves combining the features of E... Ce Dimension pooling is used to obtain ,Will As F There are two methods: first, aggregation of the common adjacency matrix of all channels; and second, aggregation of independent channels. F Each channel uses its own adjacency matrix to perform feature aggregation; specifically, it transforms the Ce dimension of E to the C dimension of F. ,Will The segments are divided along dimension C and then assigned to the channels of F for channel-independent feature aggregation.

[0017] S3.2 Obtaining the high-order features of joints in the skeleton or nodes of a single person in a group. Then, a compression matrix of order is used. By compressing the order dimension, we can obtain high-level features of key points in a skeleton or nodes of a single person in a group. The whole process corresponds to Figure 1 Middle 3 region, the advanced features at this time It is discriminative and contains a wealth of information, which can effectively improve the performance of skeleton-based action recognition or group behavior recognition.

[0018] In step S1, the first-order relation modeling function There are two implementation methods: one is to use a message function. The discovery of inter-individual mutual information involves identifying the relationship between keypoint i and keypoint j, or between individual i and individual j, as their message. Secondly, it uses a multi-head self-attention mechanism. To explore inter-individual correlations, the number of heads is set to Ce, and the relationship between keypoints i and j, or between individual i and individual j, is defined as their multi-head correlation. .

[0019] In step S2.2, the first-order dependency generating function There are three implementation methods: one is to use a message function. To explore the mutual information between first-order elements, the s-th first-order modeling element... and the t First-order modeling element The dependency between them is the message between them. Secondly, it uses a self-attention mechanism. To explore the correlation between first-order elements, the s-th first-order modeling element... and the t First-order modeling element The dependency between them is the correlation between them. Third, directly address first-order dependencies. Set as The fully connected layer, for the s-th first-order modeling element and the t First-order modeling element In other words, the dependency between them It is acquired through learning.

[0020] In step S2.4, the i-th order dependency generating function There are three implementation methods: one is to use a message function. To explore the mutual information between i-th order elements, the s-th i-th order modeling element and the t i-th order modeling element The dependency between them is the message between them. Secondly, it uses a self-attention mechanism. To explore the correlation between i-th order elements, the s-th i-th order modeling element... and the t i-th order modeling element The dependency between them is the correlation between them. Third, directly assign the i-th order dependency. Set as The fully connected layer, for the s-th i-th modeling element and the t i-th order modeling element In other words, the dependency between them The complete i-th order dependency is obtained through learning and finally used. Aggregate i-th order modeling element This yields the completeness i+1 order relation between individuals.

[0021] An electronic device includes one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to perform the methods described above.

[0022] A computer-readable storage medium storing program code that can be invoked by a processor to perform the method described above.

[0023] The beneficial effects of this invention are as follows:

[0024] 1. By generating complete dependencies between low-order relations to model complete high-order relations, this invention can better understand the complex relationships between key points in the skeleton or individuals in a group compared to simply modeling low-order relations. It can extract broader and more abstract key point features or individual features. Compared to modeling high-order relations by introducing hyperedges, this invention achieves complete modeling of high-order relations without losing potentially useful high-order terms, thus maintaining and enriching high-order features.

[0025] 2. By using recursive incremental modeling, the modeling complexity is simplified. Compared with the theoretical modeling of complete higher-order relations based on hyperedges, this invention effectively utilizes the self-similarity and iterativeness of the recursive approach to obtain the modeling complexity of higher-order relations that is linearly related to the order, thereby avoiding the factorial complexity brought about by introducing complete hyperedges.

[0026] 3. The high-order relations between key points or individuals obtained from modeling are used as the adjacency matrix in graph convolution to perform graph convolution modeling. Completeness high-order relations are used to aggregate the original features of key points in the skeleton or individuals in the group, thereby obtaining high-level features of key points in the skeleton or individuals in the group containing high-order information. This optimizes the modeling process of graph neural networks and is beneficial for skeleton-based action recognition and group behavior recognition tasks. Attached Figure Description

[0027] Figure 1 This is an overall structural diagram of the present invention.

[0028] Figure 2The schematic diagram of the recursive-based iterative modeling from low-order relationship to high-order relationship according to the present application. DETAILED DESCRIPTION

[0029] The present application is further illustrated below in conjunction with the accompanying drawings and examples.

[0030] A high-order relationship modeling method between topological nodes for topological behavior recognition, comprising the steps of:

[0031] S1. Modeling first-order relationship between nodes

[0032] The first-order relationship modeling process is defined as , wherein is the original feature of the key node in the skeleton or the individual node in the group, N is the total number of key nodes or individuals, C is the individual feature dimension, corresponding to Figure 1 F in the 1 area of Ce is the relationship feature dimension, Figure 1 is the first-order relationship between key nodes or individuals, corresponding to E in the 1 area of （1） , the first-order relationship modeling function There are two implementation forms: one is to use a message function to explore the first-order mutual information between key nodes or individuals, and the relationship between key node i and key node j or individual i and individual j is the message between them The second is to use a multi-head self-attention mechanism to explore the first-order correlation between key nodes or individuals, and the number of heads is set to Ce, and the relationship between key node i and key node j or individual i and individual j is the multi-head correlation between them Figure 1 , corresponding to in the 1 area of ;

[0033] S2. Modeling complete high-order relationship between nodes

[0034] S2.1 First-order embedding function is a fully connected layer , corresponding to Figure 1 in the 2.1 area of Figure 2 and in the a area of ;

[0035] is embedded into a first-order modeling element , corresponding to Figure 1 in the 2.1 area of Figure 2 and in the a area of ;

[0036] S2.2 The process of generating first-order complete dependencies is defined as follows: ,in For the complete dependency between first-order modeling elements, corresponding to Figure 2 The b region generated First-order dependency generating function There are three implementation methods: one is to use a message function. To explore the mutual information between first-order elements, the s-th first-order modeling element... and the t First-order modeling element

[0037] The dependency between them is the message between them. Secondly, it uses a self-attention mechanism. To explore the correlation between first-order elements, the... First-order modeling element and the t First-order modeling element The dependency between them is the correlation between them. Third, directly address first-order dependencies. Set as The fully connected layer, for the first First-order modeling element and the t First-order modeling element In other words, the dependency between them

[0038] It is learned, and the overall dependency generation process corresponds to Figure 1 2.2 area and Figure 2 region b;

[0039] S2.3 Completeness of second-order relations between nodes or individuals It is achieved by using first-order complete dependencies. Aggregate first-order modeling element The result is that this process corresponds to Figure 2 region c, Key points With key points Or the second-order relation between single person i and single person j, corresponding to Figure 1 In area 2.3 It integrates the first-order relationship between any two key points or a single person;

[0040] S2.4 Order embedding function It is a fully connected layer , corresponding to Figure 2 In region a ,Will Order relation Embedded as Modeling elements , corresponding to Figure 2 In region a , The generation process of a complete dependency is defined as follows: ,in for The complete dependency between the level model primitives corresponds to Figure 2 The b region generated Same as step S2.2, Order-dependent generating functions There are three implementation methods: one is to use a message function. To come The discovery of mutual information between order elements, the first indivual Modeling elements and the t indivual Modeling elements The dependency between them is the message between them. Secondly, it uses a self-attention mechanism.

[0041] To come Exploring the correlation between orders, the first indivual Modeling elements and the t indivual Modeling elements

[0042] The dependency between them is the correlation between them. Thirdly, directly Rank dependency Set as The fully connected layer, for the first indivual Modeling elements and the t indivual Modeling elements

[0043] In other words, the dependency between them It is obtained through learning and then used. Complete dependency polymerization Modeling elements , get the completeness between individuals order relationship

[0044] , modeling process as shown in the c region of Figure 2 ;

[0045] S2.5 in the case of a given maximum order , the low order relationship and the complete dependence between them are iteratively generated by recursion , so as to model the high order relationship , corresponding to the c region of Figure 2 , and finally get all order complete relationship , corresponding to the 2.5 region of Figure 1 , all order complete dependence is expressed as Since the modeling of high order relationship is achieved by capturing the complete dependence between low order relationships, the high order relationship obtained by modeling is complete. As can be seen from the expression of complete relationship, the representation complexity of high order relationship modeling is , which grows linearly with the order rather than factorial order. As can be seen from the expression of complete dependence, the parameter complexity of high order relationship modeling is , which also grows linearly with the order rather than factorial order, which explains its feasibility in practice. The feasibility is due to the fact that the modeling method of high order relationship of the present application is based on recursive incremental modeling;

[0046] S3. Aggregating high order features using complete high order relationship

[0047] S3.1 Graph convolution uses adjacency matrix to aggregate node features , where

[0048] is a parameter matrix, is an adjacency matrix, and are input features and output features respectively. The present application uses the complete high order relationship between the nodes of the skeleton or the individuals in the group as the adjacency matrix to aggregate the node original features of the nodes of the skeleton or the individuals in the group to obtain the node original features of the nodes of the skeleton or the individuals in the group , more than and and dimension, so the graph convolution needs to be adapted. Here As a batch processing dimension, it independently generates features of key points or individuals at various levels. There are two ways to implement V2EX: one is to perform public channel aggregation, i.e. All channels use an adjacency matrix to perform feature aggregation, specifically... of Dimension pooling is used to obtain ,Will As There are two methods: first, aggregation of the common adjacency matrix of all channels; and second, aggregation of independent channels. Each channel uses its own adjacency matrix to perform feature aggregation; specifically, it uses the adjacency matrix of that channel to perform feature aggregation. of Wei Xiang of Dimensional transformation is performed to obtain ,Will exist Perform segmentation on the dimension and then distribute it to Each channel performs channel-independent feature aggregation;

[0049] S3.2 Obtaining the high-order features of joints in the skeleton or nodes of a single person in a group. Then, a compression matrix of order is used. To compress the order dimension, we can obtain the high-order features of key points in the skeleton or nodes of a single person in a group. The whole process corresponds to Figure 1 In the middle three regions, high-level features effectively summarize the complex structures and dependencies in the skeleton topology or group topology, enabling the model to accurately represent the inherent features and patterns of skeleton information or group information. Therefore, the discriminative clues and complex information content of high-level features are unmatched by low-level features, which will bring performance improvements and enhancements to skeleton-based action recognition tasks and group behavior recognition tasks.

[0050] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for modeling high-order relationships among topological nodes for topological behavior recognition, the method comprising: identifying a plurality of topological nodes; identifying a plurality of relationships among the plurality of topological nodes; and modeling high-order relationships among the plurality of topological nodes based on the plurality of relationships. comprising the steps of: S1. extracting first-order relations among nodes; Given the original feature of a node in a skeleton or a single person in a group F ∈ R N×C , where N is the total number of individuals, C is the feature dimension of individuals, the original feature of a node F is input into a first-order relationship modeling function H e , to obtain the first-order relationship E (1) between the nodes or single persons = H e (F) ∈ R N×N×Ce , where Ce is the relationship feature dimension. For node i and node j, the first-order relationship modeling process is E (1) ij = H e (F i , F j ) ∈ R Ce , that is, the first-order relationship between the nodes or single persons; S2. modeling completeness high-order relations among nodes; S2.1 first embeds the first-order relations E (1) extracted in step S1 by a first-order embedding function M (1) ∈ R N×N×Ce as first-order modeling elements for use in second-order modeling; S2.2 The first order dependency generating function D (1) () generates the complete dependencies between first order modeling elements for the s-th first order modeling element and the t-th first order modeling element is given by S2.3 Finally, the first-order complete dependencies obtained in step S2.2 are adopted Polymerization of first-order modeling elements Obtaining the relationship E between the nodes or the completeness of the second-order relationship between individuals (2) = W (1) X (1) ∈ R N×N×Ce , E (2) ij ∈ Ce is the second-order relationship between node i and node j or individual i and individual j, steps S2.2 and S2.3 not only perceive the co-occurrence structure between first-order relationships, but also capture the association patterns of binary node groups; S2.4 follows the same logic as S2.1 to S2.3, and so on up to the (i+1)th order completeness relation modeling. First, it uses the i-th order embedding function M... (i) () will convert the i-th order relation E (i) ∈R N×N×Ce Embedded as i-th order modeling element Then use the i-th order dependency generation function D (i) Generate complete dependencies between i-th order modeling elements Finally, the obtained complete i-th order dependency W is used. (i) Aggregate i-order modeling element X (i) To obtain the completeness of the i+1 order relation E between key points or individual nodes. (i+1) =W (i) X (i) ∈R N×N×Ce ; S2.5 Iteratively find the given maximum order O of the node or all single between the order of the complete relationship E E R by the recursion of step S2.4 O×N×N×Ce , E (k) ij E R Ce The k-order relationship between node i and node j or single i and single j, since the modeling of high-order relationship is achieved by capturing the complete dependence between low-order relationships, the high-order relationship obtained by modeling is complete, and the modeling of high-order relationship is achieved by recursive increment, so its modeling complexity grows linearly with the order rather than factorial. S3. aggregating high-order features using completeness high-order relations; S3.1 follows the paradigm of graph convolution, aggregating node features F through an adjacency matrix. out =HF in A, where H is the parameter matrix, A is the adjacency matrix, and F in With F out These are the node input features and node output features, respectively, and the high-order completeness relation E∈R between key points or individuals modeled in S2 is used. O×N×N×Ce As the original feature F∈R of key points in the aggregate skeleton of the adjacency matrix A or nodes of a single person in a group. N×C To obtain the high-order features of joints in a skeleton or nodes of a single person in a group. E has two more dimensions, O and Ce, than A. Therefore, performing graph convolution requires adaptation. O is used as a batch processing dimension to independently generate individual features of various orders. There are two ways to adapt the Ce dimension: one is to perform channel common aggregation, that is, all channels of F use an adjacency matrix to perform feature aggregation. Specifically, the Ce dimension of E is pooled to obtain... Will There are two methods for aggregation: first, aggregation is performed using the common adjacency matrix of all channels in F; second, channel-independent aggregation is performed, where each channel of F uses its own unique adjacency matrix to perform feature aggregation. Specifically, this involves transforming the Ce dimension of E to the C dimension of F. Will The segments are divided along dimension C and then assigned to the channels of F for channel-independent feature aggregation. S3.2 Obtain high-level features of joints in the skeleton or single person in the group Then, a compression matrix U ∈ R OC×C Obtain high-level features of joints in the skeleton or single person in the group The high-level features at this time are discriminative and contain rich information, which can improve the performance of skeleton-based action recognition or group behavior recognition.

2. The method of claim 1, wherein the method further comprises: The first-order relationship modeling function H e () has two implementation forms: one is to use a message function H e (F) = MESS(F) to explore the inter-individual mutual information, and the relationship between the node i and the node j or the individual i and the individual j is the message MESS(F i , F j ) ∈ R Ce ; the other is to use a multi-head self-attention mechanism MHSA(F) to explore the inter-individual correlation, and the number of heads is set to Ce, and the relationship between the node i and the node j or the individual i and the individual j is the multi-head correlation MHSA(F i , F j ) ∈ R Ce .

3. The method of claim 1, wherein the method further comprises: In step S2.2, the first-order dependency generating function D (1) There are three implementation methods: one is to use a message function MESS(X) (1) To explore the mutual information between first-order elements, the s-th first-order modeling element... and the t-th first-order modeling element The dependency between them is the message between them. Second, it uses the self-attention mechanism D (1) (X (1) )=SA(X (1) = To explore the correlation between first-order elements, the s-th first-order modeling element and the t-th first-order modeling element The dependency between them is the correlation between them. Third, directly apply the first-order dependency W (1) Set to N 2 ×N 2 The fully connected layer, for the s-th first-order modeling element and the t-th first-order modeling element In other words, the dependency between them It is acquired through learning.

4. The method of claim 1, wherein the method further comprises: In step S2.4, the i-th order dependent generating function D (i) There are three implementation methods: one is to use a message function D. (i) (X (i) ) = MESS(X (i) To explore the mutual information between i-th order elements, the s-th i-th order modeling element... and the t-th i-th order modeling element The dependency between them is the message between them. Second, use self-attention mechanisms. To explore the correlation between i-th order elements, the s-th i-th order modeling element... and the t-th i-th order modeling element The dependency between them is the correlation between them. Third, directly assign the i-th order dependency to W. (i) Set to N 2 ×N 2 The fully connected layer, for the s-th i-th modeling element and the t-th i-th order modeling element In other words, the dependency between them The complete i-th order dependency W is obtained through learning and finally used. (i) Aggregate i-order modeling element X (i) The completeness i+1 order relation E between individuals is obtained. (i+1) =W (i) X (i) ∈R N×N×Ce .

5. An electronic device, comprising: one or more processors; memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs configured to perform the method of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program codes, the program codes can be called and executed by the processor to perform the method of claims 1-4.

Images