Posture-driven multi-layer perception method and system for character interaction behaviors

Abstract

The invention provides a posture-driven character interaction behavior multilayer perception method and system, and belongs to the field of computer vision and behavior recognition, and the method comprises the steps: S1, carrying out the feature extraction of an input picture through an AFE module, and carrying out the flattening operation to obtain an image feature embedding R; s2, skeleton key points of the input picture are obtained through a human body posture detection method, and posture embedding P is generated from the skeleton key points through an AGCN module; s3, inputting R and P into a cross-domain feature fusion module for fusion embedding to obtain a fusion feature F; s4, inputting the fusion feature F, the image position code and the HOI query into a Transform model to obtain an output embedding D; and decoding the D into a quintuple HOI instance by adopting a plurality of multi-layer perceptron (MLP) branches, and predicting the interaction behavior between the figures. The method aims at solving the problems that in the prior art, human body posture information is lacked, and the character interaction action recognition accuracy is low.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and behavior recognition, and particularly relates to a gesture-driven multi-layer perception method and system for human interaction behavior. Background Art

[0002] Human-Object Interaction (HOI) detection is a cutting-edge and challenging work in the field of computer vision. It is an interdisciplinary subject of target detection and behavior recognition. It is a high-level scene semantic understanding with people as the main body in images and videos. The challenge of HOI is that it needs to reason about fuzzy, complex and difficult-to-recognize interactive behaviors. While locating people and objects in the image, it is also necessary to identify and classify object categories and interactive actions. With the development of society, the urbanization rate has increased and the number of communities has increased. How to prevent the occurrence of dangerous events in the community and achieve pre-prevention and post-event tracing. This is a very challenging problem. It not only needs to pay attention to fine-grained human-centered action interactions (armed and about to commit crimes), but also includes multiple actions that occur at the same time (riding a bicycle to hit a pedestrian). For this reason, a human interaction behavior recognition method is needed to solve the problem of the lack of identification, analysis and prevention capabilities of dangerous behaviors in existing community monitoring, thereby reducing the occurrence of dangerous events in the community and protecting the personal safety of residents.

[0003] Existing methods for detecting interactive relationships between people can be roughly divided into two categories, one is sequential HOI detection, i.e., a two-stage method, and the other is parallel detection, i.e., a one-stage method. The two-stage method divides the HOI detection task into two subtasks: target detection and interaction reasoning. The target detection stage uses a pre-trained target detection model to detect people and objects in the image, and then matches them one by one into paired suggestions, while the interaction reasoning stage infers the interaction based on the features of the person-object pair. Since the two-stage detection method requires pairing the detected people and objects before predicting the interaction, it will incur expensive computational costs. The one-stage detector can detect HOI triplets directly from the image, but they still require a manual post-processing stage to match the matching rules.

[0004] The current Transformer-based end-to-end HOI detection algorithm has the following problems: First, the algorithm regards HOI detection as a set prediction and often only focuses on the target area with high confidence, which leads to the neglect of the key area where the interaction occurs. Second, the initial interaction query is usually in an unknown state and lacks intuitive interaction relationships, which may lead to low prediction accuracy. In addition, the algorithm fails to fully consider the human posture with strong topological features, which further limits the improvement of the detection effect.

[0005] Therefore, a multi-layer perception and recognition method for human interaction behavior is urgently needed to solve the problems of low accuracy and lack of human posture information in the current human interaction detection system. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a gesture-driven multi-layer perception method for human interaction behavior, comprising the following steps:

[0007] Step S1: The input image is subjected to feature extraction through the AFE module, and then a one-dimensional image feature embedding R is obtained through a flattening operation;

[0008] Step S2: obtaining the skeleton key points of the input image by a human posture detection method, and embedding the skeleton key points into posture P through the AGCN module;

[0009] Step S3: Input R and P into the cross-domain feature fusion module for fusion embedding to obtain the fusion feature F;

[0010] Step S4: Input the fused feature F, position encoding and HOI query into the Transformer model to obtain an output embedding D; use multiple MLPs to decode D into HOI instances through quintuple matching, and predict the interaction behavior between characters.

[0011] Beneficial effects:

[0012] 1. The present invention designs an AFE module to adaptively assign weights to the characteristics of the person interaction channel, reduce the non-significant features in the person interaction, and enhance the perception ability of the objects of interest in the person interaction recognition.

[0013] 2. The present invention designs a cross-domain feature fusion module of human body posture and image to identify the relative position and motion information of various parts of the human body and improve the accuracy of human interactive action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flowchart of a gesture-driven multi-layer perception method for interactive behaviors of characters according to the present invention;

[0015] Figure 2 It is a structural diagram of the AFE module;

[0016] Figure 3 It is a schematic diagram of the structure of the SE attention mechanism module;

[0017] Figure 4 It is a schematic diagram of the structure of the adaptive graph convolutional network AGCN;

[0018] Figure 5 Schematic diagram of the structure of the adaptive graph convolution block;

[0019] Figure 6 It is a structural diagram of the feature fusion module;

[0020] Figure 7 This is a schematic diagram of the Transformer structure;

[0021] Figure 8 The overall architecture diagram of the gesture-driven multi-layer perception method of human interaction behavior of the present invention is shown;

[0022] Fig. 9 This is a structural block diagram of a gesture-driven multi-layer perception system for interactive human behavior according to the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0024] Embodiment 1

[0025] like Figure 1 As shown, an embodiment of the present invention provides a gesture-driven multi-layer perception method for interactive behaviors of characters, comprising the following steps:

[0026] Step S1: The input image is subjected to feature extraction through the AFE module, and then a one-dimensional image feature embedding R is obtained through a flattening operation;

[0027] Step S2: Obtain the skeleton key points of the input image through the human posture detection method, and embed the skeleton key points into posture P through the AGCN module;

[0028] Step S3: Input R and P into the cross-domain feature fusion module for fusion embedding to obtain the fusion feature F;

[0029] Step S4: Input the fused feature F, position encoding and HOI query into the Transformer model to obtain the output embedding D; use multiple MLPs to decode D into HOI instances through quintuple matching and predict the interaction behavior between characters.

[0030] In one embodiment, the above step S1: extracting features of the input image through the AFE module, and then obtaining the image feature embedding R through a flattening operation, specifically includes:

[0031] Step S11: Introduce the SE attention mechanism based on the residual module to construct the AFE module;

[0032] The present invention designs an AFE (Adaptive Feature Enhancement) module as the backbone network to extract visual features, enhances the learning of convolutional features by introducing an SE (Squeeze-and-Excitation) attention mechanism module, and embeds the SE module into the residual module of ResNet to increase the network's sensitivity to object information features, which can be utilized by subsequent conversions.

[0033] like Figure 2 Shown is a schematic diagram of the structure of the AFE module;

[0034] Figure 3 Schematic diagram of the structure of the SE attention mechanism module.

[0035] Step S12: First, through the global pooling operation, The input image is compressed into a The eigenvector z of

[0036] (1)

[0037] Among them, H, W and C represent the height, width and number of channels of the input image respectively. Represents the pixel value of the input image at (i, j); represents the compression function, Perform compression operations;

[0038] Step S13: A gate mechanism consisting of two layers of full connections is used to fully capture channel dependencies, and Sigmoid activation is used to obtain the attention weight s:

[0039]

[0040] Among them, the dimension of s is , used to characterize the weights of C feature maps in the feature map; is the activation function; Represents the Sigmoid activation function; ReLU activation function. , represents two different weight matrices, Compress the number of channels C to , Restore the number of channels to C;

[0041] In this step, in order to utilize the information gathered in the S12 compression operation, an excitation operation is designed to fully capture channel dependencies. The excitation operation is implemented through two layers of activation functions: ReLU and Sigmoid.

[0042] Step S14: Use the Scale operation to weight the attention weight s to the features of each channel to obtain , use a 1×1 projection convolution layer to reduce the number of channels from C to d, and get the final output ; Finally, use the flatten operator to The spatial dimension of folding is used to get the image feature embedding ;

[0043] In the embodiment of the present invention, C=2048, d=256.

[0044] In one embodiment, the above step S2: obtaining the skeleton key points of the input image by a human posture detection method, and embedding the skeleton key points into posture P through the AGCN module, specifically includes:

[0045] Step S21: Use Openpose to generate the two-dimensional coordinates of the key points of the human body from the input image and construct a spatial map , where the key points are graph nodes and the lines between nodes are edges; based on two-dimensional coordinates and its quantity Constructing bone point data ,right Use the ReLU activation function and the fully connected layer FC to get the graph convolution input :

[0046] (3)

[0047] in, ; Indicates the number of input channels of the graph convolutional network; represents the FC fully connected layer;

[0048] Step S22: Input the adaptive graph convolution network AGCN, and extract the posture features through L layers of adaptive graph convolution blocks. The formula of each layer of adaptive graph convolution blocks is as follows:

[0049] (4)

[0050] in, Indicates the number of layers; is the parameter matrix of graph convolution, Indicates the number of output channels; is the adjacency matrix, where the B matrix represents the attention mechanism of the connection strength between two nodes. The B matrix is ​​completely learnable and its initial value is 1; the A matrix represents the similarity adjacency matrix between nodes. The normalized Gaussian function is used to calculate the similarity between nodes, where the normalized Gaussian function calculation is equivalent to the matrix operation of SoftMax:

[0051] (5)

[0052] Among them, SoftMax is used for normalized Gaussian function calculation; are the parameters of the Gaussian embedding function;

[0053] Finally, the output of the L-th layer adaptive graph convolution block is subjected to global average pooling GAP to obtain the posture feature embedding .

[0054] like Figure 4 , which is a schematic diagram of the structure of an adaptive graph convolutional network (AGCN). The embodiment of the present invention uses three layers of adaptive graph convolution blocks to construct the AGCN, that is, L=3. Figure 5 Schematic diagram of the structure of the adaptive graph convolution block.

[0055] In one embodiment, the above step S3: inputting R and P into the cross-domain feature fusion module for fusion embedding to obtain the fusion feature F specifically includes:

[0056] Embedding image features and posture feature embedding Input the cross-domain feature fusion module; the feature fusion module includes an encoder, which includes n encoding layers. Each encoding layer consists of a multi-head self-attention unit and a feedforward neural network unit, and is represented by EncoderLayer, as shown in formula (6):

[0057] (6)

[0058] Among them, the initial query for , No. The encoding layer embeds the image features into R as key-value pairs key and value;

[0059] Finally, the feature fusion module outputs the fused feature F.

[0060] like Figure 6 Shown is a structural diagram of the feature fusion module.

[0061] In one embodiment, the above step S4: inputting the fused feature F, the position encoding and the HOI query into the Transformer model to obtain the output embedding D; using multiple MLPs to decode D into HOI instances through quintuple matching, and predicting the interaction behavior between characters, specifically includes:

[0062] Step S41: Encode F and position The input is a Transformer encoder composed of a multi-head self-attention module and a feed-forward network, which is used to summarize global information, and the output is a global memory feature;

[0063] The encoder of the embodiment of the present invention is built on the standard Transformer architecture, with a multi-head self-attention module and a feed-forward network (FFN). In order to make it able to distinguish the relative positions in the sequence, the position encoding is added to the input of each attention layer. The sum of the fused feature features and position encoding is fed into the Transformer encoder to summarize the global information, and the output of the encoder is represented as the global memory feature.

[0064] Step S42: Global memory features, HOI query and position encoding Input Transformer decoder, output output embedding containing full image context information , is the number of query vectors;

[0065] The decoder of the embodiment of the present invention is built on the Transformer architecture. Unlike the encoder, it includes an additional multi-head cross attention layer. The decoder embeds the position (in Figure 7 HOI query) Transformed into an output embedding that contains full-image context information , is the number of query vectors. They are then decoded into HOI instances by the MLP in the subsequent steps. The decoder has three inputs, one is the global memory feature from the encoder, one is the HOI query, and one is the position encoding. There are two types of attention in the decoder, one is the self-attention of the HOI query, and the other is the attention between the HOI query and the encoder features. In the latter, Q is the HOI query, , while K and V are encoder features, . For the multi-head cross-attention layer, Value comes directly from the global memory feature, Key is the sum of the global memory feature and the input position encoding, and the query is the sum of the input position encoding and the input HOI query. For the self-attention layer, all queries, keys, and values ​​come from the HOI query or the output of the previous decoder layer. The formula for self-attention is as follows:

[0066] (7)

[0067] like Figure 7 As shown in the figure, it is a schematic diagram of the Transformer network structure.

[0068] Step S43: Each HOI instance is defined as a five-tuple { }, representing person, interaction, object, person box, object box respectively; each output embedding D is decoded into a HOI instance through 5 MLP branches, where 3 layers of MLP branches predict person , Object and interaction Confidence of ; 2 three-layer MLP branches predict human frames and object frame ;

[0069] The single-layer MLP branches used in the embodiment of the present invention for predicting confidence all use the SoftMax function. For the human confidence branch, the output size is 2, which represents the confidence of the foreground and background. For the object confidence branch and the interaction confidence branch, the output size is , represents the confidence of all C-class objects or actions defined in the dataset plus a background. For the person box and object box branches, the output size is 4, representing the normalized center coordinates of the box , height, and width.

[0070] Step S44: Obtain the corresponding ( , ( , then enumerate the detected matches ( , ) Yes, predict the interactions between characters through interaction classification .

[0071] Calculate the person frame separately , object box , object class probability , human probability , interaction class probability , are the sigmoid function and the softmax function respectively. In the proposed method, the object detector and the interaction classifier are optimized separately. In contrast, HOI detection is regarded as a set prediction problem of binary matching between the prediction and the ground truth, directly predicting the elements in the HOI set and optimizing the proposed HOI matching loss in a unified way. Assuming that the model outputs a fixed size A set of predictions, where is chosen to be larger than the number of HOI relations in an image. Let the set of predicted HOIs be represented as , the set of true HOIs is expressed as , where M ≤ N. M represents the number of real interactions in the image, Indicates no interaction. By padding the background truth set, the lengths of the two sets can be made equal.

[0072] This expresses the match as an injective function: ,in No. The index of the predicted HOI with the true value. The matching cost function is defined as:

[0073] (8)

[0074] in, Matching cost between true labels and predictions. At each step of training, the best one-to-one match between the true set and the current prediction set should be found first. The following matching cost is designed for HOI:

[0075] (9)

[0076] in, Represents the loss between people, objects and interactions using standard Cross entropy loss calculation, is the regression loss of the person box and the object box, using GIOU loss and The weighted sum of losses. and is a hyperparameter of the loss weight. Finally, the Hungarian algorithm is used to find the bipartite matching.

[0077] Figure 8 The overall architecture diagram of the gesture-driven multi-layer perception method for human interaction behavior of the present invention is shown.

[0078] Embodiment 2

[0079] like Fig. 9 As shown, the embodiment of the present invention provides a gesture-driven multi-layer perception system for interactive behaviors of characters, comprising the following modules:

[0080] AFE module is used to extract features from the input image through the AFE module, and then obtain the one-dimensional image feature embedding R through flattening operation;

[0081] The AGCN module is used to obtain the skeleton key points of the input image through the human posture detection method, and the skeleton key points are generated into the posture embedding P through the AGCN module;

[0082] The cross-domain feature fusion module is used to input R and P into the cross-domain feature fusion module for fusion embedding to obtain the fusion feature F;

[0083] The character interaction prediction module is used to input the fusion feature F, position encoding and HOI query into the Transformer model to obtain the output embedding D; multiple MLPs are used to decode D into HOI instances through quintuple matching and predict the interaction behavior between characters.

Claims

1. A gesture-driven multi-layer perception method for human interaction behavior, characterized in that: include: Step S1: The input image is subjected to feature extraction through the AFE module, and then a one-dimensional image feature embedding R is obtained through a flattening operation; Step S2: obtaining the skeleton key points of the input image by a human posture detection method, and embedding the skeleton key points into posture P through the AGCN module; Step S3: Input R and P into the cross-domain feature fusion module for fusion embedding to obtain the fusion feature F; Step S4: Input the fused feature F, position encoding and HOI query into the Transformer model to obtain an output embedding D; use multiple MLPs to decode D into HOI instances through quintuple matching, and predict the interaction behavior between characters.

2. The gesture-driven multi-layer perception method for interactive behavior of characters according to claim 1 is characterized in that: The step S1: extracting features from the input image through the AFE module, and then obtaining a one-dimensional image feature embedding R through a flattening operation, specifically includes: Step S11: Introduce the SE attention mechanism based on the residual module to construct the AFE module; Step S12: Through the global pooling operation, The input image is compressed into a feature vector z: (1) Among them, H, W and C represent the height, width and number of channels of the input image respectively. Represents the pixel value of the input image at (i, j); represents the compression function, Perform compression operations; Step S13: A gate mechanism consisting of two layers of full connection is used to fully capture channel dependencies, and Sigmoid activation is used to obtain the attention weight s: Among them, the dimension of s is , used to characterize the weights of C feature maps in the feature map; is the activation function; Represents the Sigmoid activation function; ReLU activation function. , represents two different weight matrices, Compress the number of channels C to , Restore the number of channels to C; Step S14: Using the Scale operation, the attention weight s is weighted to the features of each channel to obtain , reducing the number of channels from C to d, and obtaining the final output ; Use the flatten operator to The spatial dimension is reduced to obtain image feature embedding .

3. The gesture-driven multi-layer perception method for interactive behavior of characters according to claim 2 is characterized in that: The step S2: obtaining the skeleton key points of the input image by a human posture detection method, and embedding the skeleton key points into posture P through the AGCN module, specifically includes: Step S21: Use Openpose to generate the two-dimensional coordinates of the key points of the human body from the input image and construct a spatial map , where the key points are graph nodes and the lines between the nodes are edges; based on the two-dimensional coordinates and its quantity Constructing bone point data ,right Use the ReLU activation function and the fully connected layer FC to get the graph convolution input : (3) in, ; Indicates the number of input channels of the graph convolutional network; represents the FC fully connected layer; Step S22: Input the adaptive graph convolutional network AGCN, and extract the posture features through L layers of adaptive graph convolution blocks: (4) in, Indicates the number of layers; is the parameter matrix of graph convolution, Indicates the number of output channels; is the adjacency matrix, where the B matrix represents the attention mechanism of the connection strength between two nodes; the A matrix represents the similarity adjacency matrix between nodes: (5) in, The SoftMax function is used to calculate the normalized Gaussian function; are the parameters of the Gaussian embedding function; Finally, the output of the L-th layer adaptive graph convolution block is subjected to global average pooling GAP to obtain the posture feature embedding .

4. The gesture-driven multi-layer perception method for interactive behavior of characters according to claim 3 is characterized in that: The step S3: inputting R and P into the cross-domain feature fusion module for fusion embedding to obtain the fusion feature F, specifically includes: Embedding the image features and the posture feature embedding Input cross-domain feature fusion module; the feature fusion module is composed of an encoder, the encoder includes n encoding layers, each encoding layer is composed of a multi-head self-attention unit and a feedforward neural network unit, and is represented by EncoderLayer, as shown in formula (6): (6) Among them, the initial query for , No. The encoding layer embeds the image features into R as key-value pairs key and value; Finally, the feature fusion module outputs the fused feature F.

5. The gesture-driven multi-layer perception method for interactive behavior of characters according to claim 4 is characterized in that: The step S4: inputting the fusion feature F, the position code and the HOI query into the Transformer model to obtain an output embedding D; Multiple MLPs are used to decode D into HOI instances through quintuple matching and predict the interaction behaviors between characters, including: Step S41: Encode F and position The input is a Transformer encoder composed of a multi-head self-attention module and a feed-forward network, which is used to summarize global information, and the output is a global memory feature; Step S42: The global memory feature, HOI query and position encoding Input Transformer decoder, output output embedding containing full image context information , is the number of query vectors; Step S43: Each HOI instance is defined as a five-tuple { }, representing person, interaction, object, person box, object box respectively; each output embedding D is decoded into HOI instance through 5 MLPs, where 3 MLP branches predict the confidence of person, object and interaction respectively; 2 three-layer MLP branches predict person box and object box; Step S44: Obtain the corresponding ( , ( , then enumerate the detected matches ( , ) Yes, predict the interactions between characters through interaction classification .

6. A gesture-driven multi-layer perception system for interactive human behavior, characterized in that: Includes the following modules: AFE module is used to extract features from the input image through the AFE module, and then obtain the one-dimensional image feature embedding R through flattening operation; An AGCN module is used to obtain the skeleton key points of the input image through a human posture detection method, and embed the skeleton key points into posture P through the AGCN module; The cross-domain feature fusion module is used to input R and P into the cross-domain feature fusion module for fusion embedding to obtain the fusion feature F; The character interaction prediction module is used to input the fusion feature F, position encoding and HOI query into the Transformer model to obtain the output embedding D; use multiple MLPs to decode D into HOI instances through quintuple matching, and predict the interaction behavior between characters.

Images