Modeling method for high-order relation between topological nodes for topological behavior recognition

Abstract

The invention provides an inter-topological-node high-order relation modeling method for topological behavior recognition, and designs a recursion-based iterative modeling method from a low-order relation to a high-order relation, so that the completeness of modeling is realized through the self-homogeneity and expandability of recursion. According to the method, the complete high-order relation is modeled by generating complete dependency between low-order relations, wider and abstract individual features are extracted, the autohomogeneity and iteration of a recursion mode are effectively utilized, the modeling complexity of the high-order relation in linear correlation with the order is obtained, and the modeling complexity of the high-order relation in linear correlation with the order is improved. Therefore, the factorial complexity caused by introducing a complete hyperedge is avoided; the high-order features of the individuals in the group containing the high-order information are obtained, the modeling process of the graph neural network is optimized, and a group behavior recognition task is facilitated.

Description

Technical Field

[0001] The present invention relates to the field of graph neural network methods for topological behavior recognition, and in particular to graph convolution methods based on modeling high-order relationships between topological nodes. Background Art

[0002] Topological structure is a data representation method in machine learning, which is used to represent entities and the relationship between entities. It consists of nodes and edges, where nodes represent entities and edges represent the relationship between entities. Skeleton-based action recognition is a topological behavior recognition, in which nodes are skeletal joints, and the presentation of skeletal actions is jointly expressed and carried by multiple skeletal joints. Modeling low-order relationships between skeletal joints can only distinguish some simple actions with small scopes, such as handshakes; while some actions with many linkage joints, wide scopes, and complex joint coordination, such as specific actions in street dance, can only be distinguished by high-order relationships between skeletal joints, because high-order relationships fully exploit the mutual dependence and coordination between skeletal joints. Group behavior recognition is also a topological behavior recognition, in which nodes are single people, and the presentation of group behavior is social and group-oriented. It not only focuses on individual performance, but also pays more attention to the complex connections between groups. If we only model low-level relationships between individuals, we can only identify some simple group activities, such as conversations; complex group behaviors involve random numbers of participants, diverse forms of participation, and deep-seated dependencies between individuals, so high-level relationships between individuals are needed to capture such complex behavioral manifestations and characteristics. Generating high-level relationships to capture the details and drivers of group behaviors in detail is critical in the fields of intelligent security and automatic control.

[0003] Graph neural networks, especially graph convolutional neural networks, play a key role in identifying topological behaviors and can help models better understand and utilize topological information. In the process of relying on graph convolutional neural networks to process behavioral information based on topological structures, the core step is to model the relationship between topological nodes, because this is related to the transmission of node messages and the aggregation of node features. Most existing works use predefined prior adjacencies or semantic-based dynamic adjacencies to model the relationship between nodes to achieve the propagation and aggregation of node features, but this relationship belongs to a low-order relationship between nodes and only considers the role of isolated nodes, so it has the limitations of locality and low order.

[0004] In order to overcome the above limitations, some works have introduced hyperedges to represent the many-to-many complex relationships between multiple node bodies, which plays a certain role in modeling the high-order characteristics of nodes. However, the setting of hyperedges is incomplete, that is, only specific hyperedges between nodes are considered instead of all hyperedges, because the factorial complexity caused by introducing all hyperedges is unacceptable. Therefore, it is inevitable that the effective high-order relationships between nodes will be lost, resulting in the loss of some high-order characteristics of the nodes. Summary of the invention

[0005] In order to overcome the deficiencies of the prior art, the present invention provides a method for modeling high-order relationships between topological nodes for topological behavior recognition. In order to fully and completely model the high-order relationships between nodes to extract the high-order features of the nodes, the present invention designs an iterative modeling method from low-order relationships to high-order relationships based on recursion, and achieves the completeness of modeling with the self-identity and scalability of recursion. The key design of the present invention is to recursively model high-order relationships by capturing the complete dependencies between low-order relationships. The present invention recursively models complete high-order relationships with feasible complexity.

[0006] The specific steps of the technical solution adopted by the present invention to solve its technical problem are as follows: S1. Extract the first-order relationship between nodes; The original features of the joints in a given skeleton or the nodes of a single person in a group F ,in is the total number of individuals, is the individual feature dimension, and the original feature of the node Input to the first-order relational modeling function , get the first-order relationship between joint points or individuals ,in is the relationship feature dimension, for nodes and nodes For example, the first-order relationship modeling process is , first-order relations between joints or single persons; S2. Completeness of high-order relations between modeling nodes; S2.1 First, through the first-order embedding function The first-order relation extracted in step S1 Embedded as a first-order modeling element , for second-order modeling, this process corresponds to Figure 1 2.1 area and Figure 2 Middle a area; S2.2 Then use the first-order dependency generating function Generates complete dependencies between first-order modeling elements , for the Sth first-order modeling element and t First-order modeling element For example, the dependency generation process between them is This process corresponds to Figure 1 2.2 area and Figure 2 Middle b area; S2.3 Finally, the first-order complete dependency obtained in step S2.2 is adopted Aggregate first-order modeling elements , get the complete second-order relationship between joint points or individuals , is the second-order relationship between joint point i and joint point j or between single person i and single person j. This process corresponds to Figure 1 2.3 area and Figure 2 In the middle c region, steps S2.2 and S2.3 not only perceive the co-occurrence structure between first-order relations, but also capture the association pattern of binary node groups; S2.4 is similar to S2.1 to S2.3, and can be extended to the i+1 order completeness relation modeling. First, we use the i-order embedding function The i-th order relation Embedded as i-order modeling element ,correspond Figure 2 The a region, and then use the i-order dependent generating function generate Complete dependencies between hierarchical model elements ,correspond Figure 2 The b region is finally used to obtain the complete dependence of order i Aggregate i-order modeling elements , get the completeness i+1 order relation of the joint point or single room ,correspond Figure 2 c area; S2.5 Iterate and find the given maximum order by recursively performing step S2.4 O All order completeness relations between joint points or individuals under , is the k-order relationship between joint point i and joint point j or between single person i and single person j, corresponding to Figure 1 In the middle 2.5 region, since modeling high-order relations is achieved by capturing the complete dependencies between low-order relations, the modeled high-order relations are complete, and the modeling of high-order relations is based on recursive increments, so its modeling complexity grows linearly with the order rather than factorially. S3. Aggregate high-order features using completeness high-order relations; S3.1 Following the paradigm of graph convolution, node features are aggregated through the adjacency matrix , where H is the parameter matrix, A is the adjacency matrix, and They are node input features and node output features, respectively, using the complete high-order relationship between joint points or individuals modeled in S2 As the adjacency matrix A aggregates the original features of the joints in the skeleton or the nodes of a single person in the group , to obtain the high-order features of the joints in the skeleton or the nodes of a single person in the group ; E is more than A O and Ce dimension, so performing graph convolution requires adaptation. O As a batch processing dimension, it independently generates individual features of each order. Ce There are two ways to adapt the dimension: one is to perform channel common aggregation, that is, all channels of F use an adjacency matrix to perform feature aggregation, specifically, E Ce Dimensional pooling is performed to obtain ,Will As F The first is to aggregate the common adjacency matrix of all channels, and the second is to perform channel-independent aggregation, that is, F Each channel of uses the channel-specific adjacency matrix to perform feature aggregation, specifically, converting the Ce dimension of E to the C dimension of F to obtain ,Will Split it in the C dimension and assign it to each channel of F for channel-independent feature aggregation; S3.2 Obtaining high-order features of joints in a skeleton or nodes of a single person in a group Then, use a compression matrix of order Compress the order dimension to obtain the high-level features of the joints in the skeleton or the nodes of a single person in the group , the whole process corresponds to Figure 1 Middle 3 area, advanced features at this time It is discriminative and contains rich information, which can effectively improve the performance of skeleton-based action recognition or group behavior recognition.

[0007] In step S1, the first-order relationship modeling function There are two implementation forms: one is to use a message function To explore mutual information between individuals, the relationship between joint point i and joint point j or between single person i and single person j is the message between them , and secondly, use the multi-head self-attention mechanism To explore the correlation between individuals, set the number of heads to Ce, and the relationship between joint point i and joint point j or single person i and single person j is the multi-head correlation between them .

[0008] In step S2.2, the first-order dependency generating function There are three implementation forms: one is to use a message function To explore the mutual information between first-order elements, the sth first-order modeling element and t First-order modeling element The dependencies between them are the messages between them , and secondly, use the self-attention mechanism To explore the correlation between first-order elements, the sth first-order modeling element and t First-order modeling element The dependency between them is the correlation between them , and the third is to directly transform the first-order dependency Set to The fully connected layer, for the s-th first-order modeling element and t First-order modeling element For example, the dependence between them It is obtained through learning.

[0009] In step S2.4, the i-th order dependent generating function There are three implementation forms: one is to use a message function To explore the mutual information between i-order elements, the s-th i-order modeling element and t i-th order modeling element The dependencies between them are the messages between them , and secondly, use the self-attention mechanism To explore the correlation between i-order elements, the s-th i-order modeling element and t i-th order modeling element The dependency between them is the correlation between them , the third is to directly transform the i-th order dependency Set to The fully connected layer, for the s-th i-th order modeling element and t i-th order modeling element For example, the dependence between them It is obtained through learning, and finally the i-order complete dependency is obtained. Aggregate i-order modeling elements , we get the complete i+1 order relation between individuals An electronic device comprises one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method as described above.

[0010] A computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method as described above.

[0011] The beneficial effects of the present invention are: 1. By generating complete dependencies between low-order relations to model complete high-order relations, compared with modeling only low-order relations, the present invention can better understand the complex relations between joints in a skeleton or between individuals in a group, and extract broader and more abstract joint features or individual features. Compared with modeling high-order relations by introducing hyperedges, the present invention achieves complete modeling of high-order relations without losing potentially useful high-order terms, thus maintaining and enriching high-order features; 2. The modeling complexity is simplified by incremental modeling based on recursion. Compared with the theoretical complete high-order relationship modeling based on hyperedges, the present invention effectively utilizes the self-identity and iteration of the recursive method to obtain the modeling complexity of the high-order relationship that is linearly related to the order, thereby avoiding the factorial complexity brought by the introduction of complete hyperedges; 3. The modeled joints or high-order relationships between individuals are used as the adjacency matrix in the graph convolution to perform graph convolution modeling, and the complete high-order relationships are used to aggregate the original features of the joints in the skeleton or the individuals in the group, thereby obtaining the high-level features of the joints in the skeleton or the individuals in the group containing high-order information. This optimizes the modeling process of the graph neural network and is beneficial to skeleton-based action recognition and group behavior recognition tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is the overall structure diagram of the present invention.

[0013] Figure 2 This is a schematic diagram of the iterative modeling from low-order relations to high-order relations based on recursion proposed by the present invention. DETAILED DESCRIPTION

[0014] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0015] A method for modeling high-order relationships between topological nodes for topological behavior recognition comprises the following steps: S1. Modeling first-order relationships between nodes The first-order relationship modeling process is defined as ,in is the original feature of the joints in the skeleton or the nodes of a single person in the group, N is the total number of joints or single people, and C is the individual feature dimension, corresponding to Figure 1 F and Ce in the middle 1 area are the characteristic dimensions of the relationship. is a first-order relationship between joints or individuals, corresponding to Figure 1 E in the middle 1 area （1） , a first-order relational modeling function There are two implementation forms: one is to use a message function The first-order mutual information between nodes or individuals is mined. The relationship between node i and node j or between person i and person j is the message between them. , and secondly, use the multi-head self-attention mechanism To explore the first-order correlation between joints or individuals, set the number of heads to Ce, and the relationship between joint i and joint j or between individual i and individual j is the multi-head correlation between them , corresponding to Figure 1 In the 1st area ; S2. Completeness of high-order relationships between modeling nodes S2.1 First-order embedding function It is a fully connected layer , corresponding to Figure 1 2.1 Area and Figure 2 In the a area , the first-order relation extracted in step S1 Embedded as a first-order modeling element , corresponding to Figure 1 2.1 Area and Figure 2 In the a area ; S2.2 The generation process of first-order complete dependencies is defined as ,in The complete dependencies between the first-order modeling elements correspond to Figure 2 The generated , the first-order dependency generating function There are three implementation forms: one is to use a message function To explore the mutual information between first-order elements, the sth first-order modeling element and t First-order modeling element The dependencies between them are the messages between them , and secondly, use the self-attention mechanism To explore the correlation between the first-order elements, First-order modeling element and t First-order modeling element The dependency between them is the correlation between them , and the third is to directly transform the first-order dependency Set to The fully connected layer, for the First-order modeling element and t First-order modeling element For example, the dependence between them is obtained through learning, and the overall dependency generation process corresponds to Figure 1 2.2 area and Figure 2 b area; S2.3 Completeness of Second-Order Relationships between Nodes or Individuals is done by using first-order complete dependencies Aggregate first-order modeling elements This process corresponds to Figure 2 The c region, For joint points With joint points or the second-order relationship between person i and person j, corresponding to Figure 1 In the 2.3 area , which incorporates the first-order relationship between any two joints or individuals; S2.4 Order embedding function It is a fully connected layer , corresponding to Figure 2 In the a area ,Will Order Relationship Embed as Step Modeling Element , corresponding to Figure 2 In the a area , The generation process of the order-complete dependency is defined as ,in for The complete dependencies between the order modeling elements, corresponding to Figure 2 The generated , same as step S2.2, Order-dependent generating function There are three implementation forms: one is to use a message function To carry out The discovery of mutual information between classes, indivual Step Modeling Element and t indivual Step Modeling Element The dependencies between them are the messages between them , and secondly, use the self-attention mechanism To carry out Exploration of correlations between classes, indivual Step Modeling Element and t indivual Step Modeling Element The dependency between them is the correlation between them , the third is to directly Order Dependency Set to The fully connected layer, for the indivual Step Modeling Element and t indivual Step Modeling Element For example, the dependence between them It is obtained through learning and finally used. Complete dependencies polymerization Step Modeling Element , and obtain the completeness between individuals Order Relationship , The modeling process is as follows Figure 2 As shown in area c; S2.5 At a given maximum order In the case of, iteratively generate lower first-order relations by recursion And the complete dependencies , thereby modeling higher-order relationships , corresponding to Figure 2 c region, and finally obtain all order completeness relations , corresponding to Figure 1 In the middle 2.5 region, the complete dependence of all orders is expressed as Since modeling high-order relations is achieved by capturing the complete dependencies between low-order relations, the modeled high-order relations are complete. From the representation of completeness relations, it can be seen that the representation complexity of high-order relation modeling is , which varies with the order The growth is linear rather than factorial. As can be seen from the complete dependency representation, the parameter complexity of high-order relationship modeling is , which also varies with the order The linear growth rather than factorial growth indicates its feasibility in practice. The feasibility is attributed to the fact that the high-order relational modeling method of the present invention is based on recursive incremental modeling. S3. Aggregate high-order features using complete high-order relations S3.1 Graph convolution uses adjacency matrix to aggregate node features ,in is the parameter matrix, is the adjacency matrix, and are input features and output features respectively. The present invention uses the complete high-order relationship between joint points or single persons. As an adjacency matrix To aggregate the original features of the joints in the skeleton or the nodes of a single person in the group In order to obtain the original features of the joints in the skeleton or the nodes of a single person in the group , Compare More and dimension, so the graph convolution needs to be adapted. As a batch processing dimension, it can independently generate the features of joint points or single people at different orders. There are two ways to maintain adaptation: one is to perform channel public aggregation, that is, All channels of use an adjacency matrix to perform feature aggregation, specifically of Dimensional pooling is performed to obtain ,Will As The first is to aggregate the common adjacency matrix of all channels, and the second is to perform channel-independent aggregation, that is, Each channel of performs feature aggregation using the channel-specific adjacency matrix. Specifically, of Wei Xiang of Dimension transformation is performed to obtain ,Will exist Divide it on the dimension and then distribute it to Each channel of the channel performs channel-independent feature aggregation; S3.2 Obtaining high-order features of joints in a skeleton or nodes of a single person in a group Then, use a compression matrix of order To compress the order dimension, obtain the high-order features of the joints in the skeleton or the nodes of a single person in the group , the whole process corresponds to Figure 1 In the middle 3 regions, high-level features effectively summarize the complex structures and dependencies in the skeleton topology or group topology, enabling the model to accurately represent the intrinsic characteristics and patterns of skeleton information or group information. Therefore, the discriminative clues and complex information content of high-level features are incomparable to low-level features, which will bring performance improvements and enhancements to skeleton-based action recognition tasks and group behavior recognition tasks.

[0016] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for modeling high-order relationships between topological nodes for topological behavior recognition, characterized in that The steps include: S1. Extract the first-order relationship between nodes; Given a joint point in a skeleton or a single person in a group, the original feature F∈R N×C , where N is the total number of individuals, C is the dimension of individual features, and the original feature F of the node is input into the first-order relationship modeling function H e (), get the first-order relationship E between joint points or single persons (1) =H e (F)∈R N×N×Ce , where Ce is the relationship feature dimension. For nodes i and j, the first-order relationship modeling process is E (1) ij =H e (F i , F j )∈R Ce , first-order relations between joints or single persons; S2. Completeness of high-order relations between modeling nodes; S2.1 First, through the first-order embedding function M (1) () The first-order relation E extracted in step S1 is (1) ∈R N×N×Ce Embedded as a first-order modeling element For use in second-order modeling; S2.2 Then use the first-order dependency generating function D (1) () Generates complete dependencies between first-order modeling elements For the Sth first-order modeling element and the tth first-order modeling element For example, the dependency generation process between them is S2.3 Finally, the first-order complete dependency obtained in step S2.2 is adopted Aggregate first-order modeling elements Get the complete second-order relationship E between joint points or individuals (2) =W (1) X (1) ∈R N×N×Ce , E (2) ij ∈ Ce For the second-order relationship between joint point i and joint point j or between person i and person j, steps S2.2 and S2.3 not only perceive the co-occurrence structure between the first-order relationships, but also capture the association pattern of binary node groups; S2.4 is similar to S2.1 to S2.3, and can be extended to the i+1 order completeness relation modeling. First, the i-order embedding function M (i) () The i-th order relation E (i) ∈R N×N×Ce Embedded as i-order modeling element Then use the i-order dependency generating function D (i) Generate complete dependencies between i-order modeling elements Finally, we use the obtained i-order complete dependency W (i) Aggregate i-order modeling element X (i) , get the completeness i+1 order relation E of the joint point or single room (i+1) =W (i) X (i) ∈R N×N×Ce ; S2.5 Iterate and find all order completeness relations E∈R between joint points or single persons under a given maximum order O by recursively performing step S2.4 O×N×N×Ce , E (k) ij ∈R Ce is the k-order relationship between joint point i and joint point j or between person i and person j. Since modeling high-order relationships is achieved by capturing the complete dependencies between low-order relationships, the modeled high-order relationships are complete, and the modeling of high-order relationships is based on recursive increments, so its modeling complexity grows linearly with the order rather than factorially. S3. Aggregate high-order features using completeness high-order relations; S3.1 follows the paradigm of graph convolution and aggregates node features F through the adjacency matrix out =HF in A, where H is the parameter matrix, A is the adjacency matrix, and F in With F out They are the node input features and node output features respectively, and the complete high-order relationship E∈R between the joint points or individuals modeled in S2 is adopted O×N×N×Ce As the adjacency matrix A aggregates the original features of the joints in the skeleton or the nodes of a single person in the group F∈R N×C , to obtain the high-order features of the joints in the skeleton or the nodes of a single person in the group E has more O and Ce dimensions than A, so the graph convolution needs to be adapted. O is used as the batch processing dimension to independently generate individual features of each order. There are two ways to adapt the Ce dimension: one is to perform channel common aggregation, that is, all channels of F use an adjacency matrix to perform feature aggregation. Specifically, the Ce dimension of E is pooled to obtain Will The first is to aggregate as the common adjacency matrix of all channels of F, and the second is to perform channel-independent aggregation, that is, each channel of F uses the channel-specific adjacency matrix to perform feature aggregation. Specifically, the Ce dimension of E is converted to the C dimension of F to obtain Will Split it in the C dimension and assign it to each channel of F for channel-independent feature aggregation; S3.2 Obtaining high-order features of joints in a skeleton or nodes of a single person in a group Then, a compression matrix U∈R is used OC×C Compress the order dimension to obtain the high-level features of the joints in the skeleton or the nodes of a single person in the group The whole process corresponds to area 3 in Figure 1. The high-level features at this time are It is discriminative and contains rich information, which can effectively improve the performance of skeleton-based action recognition or group behavior recognition.

2. The method for modeling high-order relationships between topological nodes for topological behavior identification according to claim 1, characterized in that: In step S1, the first-order relationship modeling function H e () has two implementation forms: one is to use a message function H e (F) = MESS(F) to explore the mutual information between individuals. The relationship between joint point i and joint point j or between single person i and single person j is the message MESS(F) between them. i , F j )∈R Ce , and secondly, use the multi-head self-attention mechanism MHSA(F) to explore the correlation between individuals. The number of heads is set to Ce, and the relationship between joint point i and joint point j or single person i and single person j is the multi-head correlation MHSA(F i , F j )∈R Ce .

3. The method for modeling high-order relationships between topological nodes for topological behavior identification according to claim 1, characterized in that: In step S2.2, the first-order dependency generating function D (1) There are three implementation forms: one is to use a message function MESS (X (1) ) to explore the mutual information between first-order elements, and the sth first-order modeling element and the tth first-order modeling element The dependencies between them are the messages between them The second is to use the self-attention mechanism D (1) (X (1) )=SA(X (1) ) = explore the correlation between first-order elements, the sth first-order modeling element and the tth first-order modeling element The dependency between them is the correlation between them The third is to directly use the first-order dependency W (1) Set to N 2 ×N 2 The fully connected layer, for the s-th first-order modeling element and the tth first-order modeling element For example, the dependence between them It is obtained through learning.

4. The method for modeling high-order relationships between topological nodes for topological behavior identification according to claim 1, characterized in that: In step S2.4, the i-th order dependent generating function D (i) There are three implementation forms: one is to use a message function D (i) (X (i) )=MESS(X (i) ) to explore the mutual information between i-order elements, and the s-th i-order modeling element and the t-th i-th order modeling element The dependencies between them are the messages between them The second is to use the self-attention mechanism To explore the correlation between i-order elements, the s-th i-order modeling element and the t-th i-th order modeling element The dependency between them is the correlation between them The third is to directly rely on the i-th order W (i) Set to N 2 ×N 2 The fully connected layer, for the s-th i-th order modeling element and the t-th i-th order modeling element For example, the dependence between them is obtained through learning, and finally the i-order complete dependency W is obtained. (i) Aggregate i-order modeling element X (i) , we get the complete i+1 order relation E between individuals (i+1) =W (i) X (i) ∈R N×N×Ce .

5. An electronic device, characterized in that: The method comprises one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to claims 1-4.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, and the program codes can be called by a processor to execute the method according to claims 1-4.

Images