A human interaction behavior recognition method and system based on multi-view contrast

Abstract

The application belongs to the field of computer vision, and provides a human interaction behavior recognition method and system based on multi-view comparison, which acquires position information of human joints in each frame of video data; constructs a skeleton space-time graph based on the position information of human joints in each frame; and constructs an enhanced view with deleted nodes or edges based on the skeleton space-time graph through a graph convolutional neural network, and adaptively deletes edges or nodes, which can well alleviate the uneven distribution problem; adopts an information bottleneck principle to reserve sufficient information meeting a minimum for a behavior recognition task in each view, increase the difference between the enhanced view and the original skeleton space-time graph, and maximize information related to the behavior recognition task to obtain a multi-view representation; and obtains a human interaction behavior recognition result according to the multi-view representation, so as to better learn a multi-view representation of the interaction behavior from different aspects.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to a method and system for recognizing human interaction behavior based on multi-view comparison. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] With the gradual promotion and popularization of high-definition video surveillance, the amount of surveillance video data is also increasing. In the security field, especially in public places, real-time monitoring of people's behavior is needed to prevent accidents. While the accuracy of individual behavior recognition has greatly improved with the rapid development of computer vision technology, the problem of complex human activities involving multiple people has not yet been fully solved. In real life, many common behaviors are interactive, such as handshakes, hugs, and fights. Compared to single-person actions, interactive actions are more complex, involving more types of body movements and more diverse changes in limbs. Therefore, efficiently extracting the features of interactive behaviors and modeling and analyzing them is a challenging problem.

[0004] Based on the source of human motion data, human behavior recognition methods can be divided into three categories: RGB video-based human behavior recognition methods, depth map-based human behavior recognition methods, and skeleton sequence-based human behavior recognition methods. RGB video data provides the necessary spatial and temporal information for human behavior recognition, but it lacks information about the human body's structural movements in three-dimensional space (such as the position and angle of various joints or body parts, and their relative relationships). It only provides a two-dimensional spatial state of the person and is easily affected by complex backgrounds, lighting, and changes in viewing angle, leading to decreased accuracy in motion recognition. Compared to RGB video data, depth maps can provide information such as the distance between the viewpoint and the object, the coordinates of various joints or body parts in three-dimensional space, and the outline and texture of the human body in three-dimensional space. This allows for the separation of the person from the background; however, depth map methods require computing devices with large memory and strong computing power. Skeleton sequences, on the other hand, can define human posture through the relative position information of joints, more realistically representing the geometric structure of human motion patterns. Compared to image features, skeleton features are more compact, provide a more detailed description of human motion, and are less affected by changes in lighting and background.

[0005] In summary, the inventors have discovered the following technical problems with the existing technology:

[0006] (1) Interaction noise. In interactive behavior, noise interference often exists due to sensor errors or occlusion, making it impossible to clearly simulate the relationship between the body parts of the subjects interacting, which is key information for interaction recognition. Graph convolution-based models are easily affected by the quality of the input graph, which means that aggregating misleading neighborhood information may lead to suboptimal performance.

[0007] (2) Skeleton data is diverse and complex. Different people may have different heights, body types, postures, and movement patterns, and the same behavior may have different execution speeds, amplitudes, and angles. This inconsistent data distribution will make graph convolution-based models prone to bias towards data with certain distributions, hindering the learning of behavioral representations. Summary of the Invention

[0008] To address at least one of the technical problems in the background art, this invention provides a human interaction behavior recognition method and system based on multi-view comparison. It can learn whether to delete edges or nodes, transform the original skeleton graph into related views, and then integrate different views into a compact representation of the downstream behavior recognition task. At the same time, it is jointly optimized with the downstream behavior recognition task in an end-to-end manner to further improve the robustness of the model.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] The first aspect of this invention provides a method for recognizing human interaction behavior based on multi-view comparison, comprising the following steps:

[0011] Obtain the position information of human joints in each frame of video data;

[0012] Based on the positional information of human joints in each frame, a spatiotemporal graph of the skeleton is constructed.

[0013] Based on the skeleton spatiotemporal graph, an enhanced view is constructed by adaptively deleting edges or nodes of the skeleton spatiotemporal graph through a graph convolutional neural network.

[0014] By adopting the information bottleneck principle, the difference between the enhanced view and the original skeleton spatiotemporal graph is increased, while maximizing the information related to the behavior recognition task. In each view, sufficient information that meets the minimum requirement for the behavior recognition task is retained, resulting in a multi-view representation.

[0015] The human interaction behavior recognition results are obtained by classifying the obtained multi-view representations.

[0016] A second aspect of the present invention provides a human interaction behavior recognition system based on multi-view comparison, comprising:

[0017] The joint information acquisition module is used to acquire the position information of human joints in each frame of video data;

[0018] The skeleton spatiotemporal graph construction module is used to construct a skeleton spatiotemporal graph based on the positional information of human joints in each frame.

[0019] An enhanced view building module is used to adaptively delete edges or nodes of a skeleton spatiotemporal graph using a graph convolutional neural network, based on the skeleton spatiotemporal graph, to build an enhanced view with deleted nodes or edges.

[0020] The multi-view representation module is used to adopt the information bottleneck principle to increase the difference between the enhanced view and the original skeleton spatiotemporal graph, while maximizing the information related to the behavior recognition task. In each view, it retains enough information to satisfy the minimum requirement for the behavior recognition task, thus obtaining a multi-view representation.

[0021] The behavior recognition module is used to classify human interaction behavior based on the obtained multi-view representation to obtain the recognition results.

[0022] A third aspect of the present invention provides a computer-readable storage medium.

[0023] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the human interaction behavior recognition method based on multi-view comparison as described above.

[0024] A fourth aspect of the present invention provides a computer device.

[0025] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the human interaction behavior recognition method based on multi-view comparison described above.

[0026] Compared with the prior art, the beneficial effects of the present invention are:

[0027] 1. This invention addresses the issues of noise and the diversity and complexity of skeleton data in interaction graphs. It employs an effective augmentation method to ensure that different views cover the optimal region as much as possible while minimizing useless information. By adaptively deleting nodes and edges to construct an optimized graph structure, the uneven distribution problem can be effectively alleviated. Furthermore, in each view, minimum sufficient information is retained for downstream behavior recognition tasks, while information irrelevant to behavior recognition is discarded to solve the interaction noise problem. This allows for better learning of multi-view representations of interaction behaviors from different perspectives, thereby achieving better recognition of interaction behaviors.

[0028] 2. This invention also takes into account that skeleton data can reduce the influence of unnecessary factors such as viewpoint and background, and graph convolutional neural networks can directly process the topological graph of skeleton data, effectively capturing the spatiotemporal relationships and contextual information in the skeleton data.

[0029] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0030] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0031] Figure 1 This is a flowchart of the human interaction behavior recognition method based on multi-view comparison provided in the embodiments of the present invention;

[0032] Figure 2 This is an example of a spatiotemporal graph established according to an embodiment of the present invention;

[0033] Figure 3 This is the overall structure of the multi-view comparison network provided in the embodiments of the present invention. Detailed Implementation

[0034] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0035] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0036] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0037] To address the technical problems mentioned in the background, this invention proposes a human interaction behavior recognition method based on multi-view comparison. To eliminate unnecessary parameters such as viewpoint and lighting, a method for extracting human pose maps from input video frames is employed. Furthermore, considering the noise in the interaction maps and the diversity and complexity of skeleton data, an effective augmentation method is considered to ensure that different views cover the optimal region as much as possible while minimizing useless information. By adaptively deleting nodes and edges to construct an optimized graph structure, the uneven distribution problem can be effectively alleviated. Moreover, in each view, minimal sufficient information is retained for downstream behavior recognition tasks, while information irrelevant to behavior recognition is discarded to address the interaction noise problem. This allows for better learning of multi-view representations of interaction behaviors from different perspectives, thereby achieving better recognition of interaction behaviors.

[0038] Example 1

[0039] like Figure 1 As shown, this embodiment provides a method for recognizing human interaction behavior based on multi-view comparison, including the following steps:

[0040] Step 1: Obtain the position information of human joints in each frame of the video data;

[0041] In step 1, the pose estimation algorithm is used to estimate the pose based on the video data to obtain the position information of the human joints. In this embodiment, the pose estimation algorithm can be:

[0042] It should be noted that in this embodiment, the attitude estimation algorithm adopts an existing algorithm, such as OpenPose. Those skilled in the art can set it themselves according to the specific working conditions, which will not be described in detail here.

[0043] Step 2: Construct a spatiotemporal graph of the skeleton based on the positional information of the human joints in each frame;

[0044] like Figure 2 As shown, in step 2, constructing a spatiotemporal skeleton map based on the position coordinates of human joints in each frame of the video specifically includes:

[0045] In the spatial dimension, the spatial position of the joints is first determined based on the coordinate information of the joints in each frame. Then, the corresponding edges are drawn according to the natural structure of the human body to obtain the spatial topology of the skeleton sequence. After the spatial topology of the skeleton sequence is constructed, the nodes representing the same joint in adjacent frames are connected to form the spatiotemporal sequence diagram of the skeleton.

[0046] In other words, in the spatial dimension, the skeleton diagram in each frame is defined using the natural structure of the human body, forming a spatial topology diagram of the skeleton sequence; in the temporal dimension, joints representing the same part in adjacent frames are connected to form a spatiotemporal sequence diagram of the skeleton. Spatial relationships are constructed within each frame according to the natural skeletal connections of the human body, while temporal relationships are constructed by connecting the same joints between consecutive frames.

[0047] like Figure 3 As shown, step 3: Based on the skeleton spatiotemporal graph, nodes or edges in the skeleton spatiotemporal graph are adaptively deleted using a graph convolutional neural network, and an enhanced view with deleted nodes or edges is constructed. The skeleton spatiotemporal graph is input into the network in the form of an adjacency matrix, which is n*n in size, where n represents the number of nodes in the graph. If there is a connection between two nodes, the value of the corresponding element in the adjacency matrix is ​​1, otherwise it is 0.

[0048] Specifically, this includes: Step 301: Based on the skeleton spatiotemporal graph, the nodes of the skeleton spatiotemporal graph are adaptively deleted using a graph convolutional neural network to construct an enhanced view of the deleted nodes, which specifically includes:

[0049] Learning deletable nodes in each layer of a graph convolutional network to mask out influential nodes and create a node deletion view can be expressed as:

[0050]

[0051] In the formula, This is a view of node deletion created after blocking those influential nodes, v i It refers to the i-th node in the l-th network layer. From a parameterized The Bernoulli distribution, i.e. It indicates whether to retain node v. i ε means the set of edges.

[0052] In this embodiment, influential nodes refer to nodes that affect the final recognition result. These nodes typically include noisy nodes and nodes that contribute to most actions. Noisy nodes introduce errors into the recognition process; while nodes that contribute to most actions receive excessive attention, thus reducing the differences between different actions.

[0053] Simply deleting the selected node and all its connections will cause a huge change in the graph structure, which will affect information aggregation and make training unstable.

[0054] Therefore, in this embodiment, the nodes of the spatiotemporal graph of the skeleton to be deleted are replaced with the representation of the local subgraph of the node to be deleted, so as to obscure its original representation and retain its corresponding edges.

[0055] Specifically, a multilayer perceptron is used to control whether nodes are masked.

[0056]

[0057] in, This indicates whether the i-th node in the l-th layer needs to be masked. The i-th node in layer l.

[0058] For node v, perform a random walk on graph G with step size k, and then use the mean pooling of the sampled nodes as a representation of the local subgraph of v.

[0059] Step 302: The step of adaptively deleting edges from the skeleton spatiotemporal graph and constructing an enhanced view with deleted edges based on the skeleton spatiotemporal graph through a graph convolutional neural network specifically includes:

[0060] The goal of the edge-removing augmented view is to generate a subgraph that filters out noisy edges and intentionally reduces the impact of certain distributions on the GCN layer.

[0061] Similar to the node deletion view, a view that deletes along the way is created through learnable edge deletion operations:

[0062]

[0063] in, Obey the same Distribution, representing edge e ij Whether to retain it.

[0064] Similarly, for parameters Multilayer perceptrons (MLPs) are also used to control the edges e. ij To block.

[0065]

[0066] In the formula, The meaning is whether edge e is needed. ij To block, Representing edge e ij .

[0067] Step 4: Adopt the information bottleneck principle, retain the minimum sufficient information for the behavior recognition task in each view, increase the difference between the augmented view and the original skeleton spatiotemporal graph, and maximize the information related to the behavior recognition task to obtain a multi-view representation; while encouraging the difference between the augmented view and the original graph, maximize the information related to the behavior recognition task.

[0068] In step 4, minimizing the mutual information between the enhanced view and the original view by using negative contrastive learning loss InfoNCE can remove redundant information in each view, thereby retaining a minimum amount of sufficient information for downstream behavior recognition tasks.

[0069]

[0070]

[0071]

[0072] Where E represents the original view. Indicates an enhanced view. This represents the BPR loss, which aims to maximize the difference between the enhanced view and the original view. This represents the mutual information between two augmented views that have deleted nodes. The augmented views represent the mutual information between two deleted edges, s(·) represents the similarity between two vectors, and E n and For an enhanced view of two different deleted nodes, E e and For two different enhanced views of deleted edges, v i and e is a vector representation of the corresponding node in two different augmented views of the deleted node. i for This is the vector representation of the corresponding edges in two different augmented views where edges have been removed.

[0073] Step 5: Obtain the human interaction behavior recognition results based on the multi-view representation.

[0074] The advantage of the above scheme is that the human interaction behavior recognition method based on multi-view comparison can learn whether to delete edges or nodes, transform the original skeleton graph into related views, and then integrate different views into a compact representation of the downstream behavior recognition task. At the same time, it is jointly optimized with the downstream behavior recognition task in an end-to-end manner, further improving the robustness of the model.

[0075] Example 2

[0076] This embodiment provides a human interaction behavior recognition system based on multi-view comparison, including:

[0077] The joint information acquisition module is used to acquire the position information of human joints in each frame of video data;

[0078] The skeleton spatiotemporal graph construction module is used to construct a skeleton spatiotemporal graph based on the positional information of human joints in each frame.

[0079] An enhanced view building module is used to adaptively delete edges or nodes of a skeleton spatiotemporal graph using a graph convolutional neural network, based on the skeleton spatiotemporal graph, to build an enhanced view with deleted nodes or edges.

[0080] The multi-view representation module is used to adopt the information bottleneck principle to increase the difference between the enhanced view and the original skeleton spatiotemporal graph, while maximizing the information related to the behavior recognition task. In each view, it retains enough information to satisfy the minimum requirement for the behavior recognition task, thus obtaining a multi-view representation.

[0081] The behavior recognition module is used to classify human interaction behavior based on the obtained multi-view representation to obtain the recognition results.

[0082] Example 3

[0083] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a human interaction behavior recognition method based on multi-view comparison as described in Embodiment 1.

[0084] Example 4

[0085] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the human interaction behavior recognition method based on multi-view comparison as described in Embodiment 1.

[0086] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0087] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0088] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0089] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0090] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for recognizing human interaction behavior based on multi-view comparison, characterized in that, Includes the following steps: Obtain the position information of human joints in each frame of video data; Based on the positional information of human joints in each frame, a spatiotemporal graph of the skeleton is constructed. Based on the skeleton spatiotemporal graph, an enhanced view is constructed by adaptively deleting edges or nodes of the skeleton spatiotemporal graph through a graph convolutional neural network. The method based on a skeleton spatiotemporal graph, using a graph convolutional neural network, adaptively deletes edges or nodes from the skeleton spatiotemporal graph to construct an enhanced view with the deleted nodes or edges, specifically including: Learn the removable nodes of each layer of the graph convolutional neural network, and create a node deletion view after masking the nodes that have an impact. Simultaneously, it learns the removable edges of each layer of the graph convolutional neural network, filters out noisy edges, and creates an edge removal view; Replace the nodes of the spatiotemporal graph of the skeleton to be deleted with the representation of the local subgraph of the node to be deleted, so as to blur its original representation and retain its corresponding edges; A multilayer perceptron is used to control whether to mask the nodes based on the parameters. in, Indicates the first l Does the i-th node of the layer need to be masked? No. l The i-th node of the layer; By adopting the information bottleneck principle, the difference between the enhanced view and the original skeleton spatiotemporal graph is increased, while maximizing the information related to the behavior recognition task. In each view, sufficient information that meets the minimum requirement for the behavior recognition task is retained, resulting in a multi-view representation. The human interaction behavior recognition results are obtained by classifying the obtained multi-view representations.

2. The human interaction behavior recognition method based on multi-view comparison as described in claim 1, characterized in that, The construction of the spatiotemporal graph of the skeleton based on the positional information of human joints in each frame specifically includes: In the spatial dimension, the spatial position of the joints is determined based on the coordinate information of the joints in each frame, and then the corresponding edges are drawn according to the natural structure of the human body to obtain the spatial topology map of the skeleton sequence. After constructing the spatial topology graph of the skeleton sequence, connecting the nodes representing the same joint in adjacent frames can form the spatiotemporal sequence graph of the skeleton.

3. The human interaction behavior recognition method based on multi-view comparison as described in claim 1, characterized in that, The expression for creating a node deletion view after masking out influential nodes in each layer of the learning graph convolutional neural network is as follows: In the formula, It refers to the first l The first layer in the network layer i 1 node Derived from a parameterized Bernoulli distribution, which indicates whether to retain nodes. , Let be the set of edges.

4. The human interaction behavior recognition method based on multi-view comparison as described in claim 1, characterized in that, The principle of information bottleneck is adopted to increase the difference between the augmented view and the original skeleton spatiotemporal map, while maximizing the information related to the action recognition task. Specifically, the mutual information between the augmented view and the original map is minimized by using negative contrastive learning loss, redundant information in each view is removed, and the remaining information is retained.

5. The human interaction behavior recognition method based on multi-view comparison as described in claim 1, characterized in that, The spatiotemporal graph input to the skeleton is presented in the form of an adjacency matrix, the size of which is n. n represents the number of nodes in the graph. If there is a connection between two nodes, the element value at the corresponding position in the adjacency matrix is ​​1, otherwise it is 0.

6. The human interaction behavior recognition method based on multi-view comparison as described in claim 1, characterized in that, The expression for minimizing the mutual information between the augmented view and the original image by employing negative contrastive learning loss to remove redundant information in each view and retain the remaining information is as follows: In the formula, Represents the original view. Indicates an enhanced view. This represents the BPR loss, which aims to maximize the difference between the enhanced view and the original view. This represents the mutual information between two augmented views that have deleted nodes. This represents the mutual information between two augmented views that have had their edges deleted. This represents the similarity between two vectors. and An enhanced view for two different deleted nodes. and Enhanced views for two different deleted edges. This is a vector representation of the corresponding node in two different augmented views of the deleted node. for This is the vector representation of the corresponding edges in two different augmented views where edges have been removed.

7. A human interaction behavior recognition system based on multi-view comparison, characterized in that, include: The joint information acquisition module is used to acquire the position information of human joints in each frame of video data; The skeleton spatiotemporal graph construction module is used to construct a skeleton spatiotemporal graph based on the positional information of human joints in each frame. An enhanced view building module is used to adaptively delete edges or nodes of a skeleton spatiotemporal graph using a graph convolutional neural network, based on the skeleton spatiotemporal graph, to build an enhanced view with deleted nodes or edges. The method based on a skeleton spatiotemporal graph, using a graph convolutional neural network, adaptively deletes edges or nodes from the skeleton spatiotemporal graph to construct an enhanced view with the deleted nodes or edges, specifically including: Learn the removable nodes of each layer of the graph convolutional neural network, and create a node deletion view after masking the nodes that have an impact. Simultaneously, it learns the removable edges of each layer of the graph convolutional neural network, filters out noisy edges, and creates an edge removal view; Replace the nodes of the spatiotemporal graph of the skeleton to be deleted with the representation of the local subgraph of the node to be deleted, so as to blur its original representation and retain its corresponding edges; A multilayer perceptron is used to control whether to mask the nodes based on the parameters. in, Indicates the first l Does the i-th node of the layer need to be masked? No. l The i-th node of the layer; The multi-view representation module is used to adopt the information bottleneck principle to increase the difference between the enhanced view and the original skeleton spatiotemporal graph, while maximizing the information related to the behavior recognition task. In each view, it retains enough information to satisfy the minimum requirement for the behavior recognition task, thus obtaining a multi-view representation. The behavior recognition module is used to classify human interaction behavior based on the obtained multi-view representation to obtain the recognition results.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the human interaction behavior recognition method based on multi-view comparison as described in any one of claims 1-6.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the human interaction behavior recognition method based on multi-view comparison as described in any one of claims 1-6.

Images