Multi-object scene touchless interaction detection method

Abstract

The present application belongs to the field of computer vision, and proposes a non-contact interaction category and interaction subject recognition technology, mainly solves the recognition problem of interaction individual and interaction action in the case that interaction individuals do not contact each other and other non-interaction individuals exist in the multi-person scene. For this purpose, the technical scheme adopted by the present application is a multi-object scene non-contact interaction detection method, which uses a relation pair embedding network PRE-Net to obtain a relation pair by aggregating individual and group information, short-term and long-term information in the scene, and to perform interaction prediction and recognition by calculating an interaction relation matrix, an individual interaction category vector and a global interaction category vector. The present application is mainly applied to the recognition occasion of interaction individual and interaction action in the case that interaction individuals do not contact each other and other non-interaction individuals exist in the multi-person scene.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and relates to a novel technology for non-contact interaction subject and interaction category recognition in multi-person scenes. This invention can be used for social relationship analysis, pedestrian trajectory tracking, and abnormal behavior analysis. When individuals interacting in a multi-person scene do not touch each other and there are other individuals not participating in the interaction, the invention identifies the interacting individuals and their actions in the video. Background Technology

[0002] The background technology involved in this invention includes:

[0003] (1) Human-to-human interaction recognition (see reference [1]): Human-to-human interaction recognition is an important step in understanding complex human social activities and plays an important role in surveillance video analysis. In most early datasets, there were only interacting subjects in the scene, without any other prominent subjects, or the prominent interacting subjects could be easily cropped / highlighted from the scene. In addition, for some datasets of multi-person scenes, the interacting subjects in the scene are close to each other and are usually the focus of the camera. Unlike previous studies, this invention aims to solve the problem of non-contact interaction recognition in multi-person scenes, which is more in line with the application of interaction recognition in the real world.

[0004] (2) Group activity / relationship recognition (see reference [2]): Previous studies on group activity / relationship recognition have mainly focused on the spatiotemporal information of individuals and the relationship information between individuals in the scene. The main difference between this invention and group activity recognition is that group activity recognition focuses more on using the behavior of all or most people to obtain the overall activity of the video, while this invention needs to examine all individuals in the scene to identify the interacting subjects. In addition, unlike HOI human-object interaction detection, this invention does not rely on common sense-based inherent priors, and each individual can interact with any other individual in any way. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, this invention aims to propose a non-contact interaction category and interaction subject recognition technology for multi-person scenarios. It primarily addresses the problem of recognizing interacting individuals and their actions in multi-person scenarios where interacting individuals do not interact with each other, and other non-interacting individuals are present. The technical solution adopted by this invention is a non-contact interaction detection method for multi-object scenarios. It utilizes the PRE-Net relation pair embedding network to obtain relation pairs by aggregating individual and group information, as well as short-term and long-term information in the scene. Interaction prediction and recognition are then performed by calculating the interaction relation matrix, individual interaction category vectors, and a global interaction category vector.

[0006] The specific steps are as follows:

[0007] Step S1: Individual Information Extraction

[0008] (a) Short-term video segmentation

[0009] For an input video sequence, it is divided into K equal-length video segments. Each video segment is sampled at short intervals to obtain N video frames. For a video sequence with K segments, K×N video frames are finally obtained.

[0010] (b) Spatial Information Extraction

[0011] For the i-th individual in frame t, its 2D position vector Represented as:

[0012]

[0013] Calculate the direction vector of motion

[0014]

[0015] A single-layer GRU is used to integrate the location information of the same individual across N video frames in a video segment. For individual i in frame t, its location information is calculated.

[0016]

[0017] Ultimately, the spatial characteristics of each individual are obtained.

[0018] (c) Appearance feature extraction

[0019] The Inception-v3 asymmetric convolutional network is used to extract multi-scale feature information within each bounding box. The extracted features are then resized to the same size using RoIAlign, a region of interest feature extraction method. Temporal information is aggregated using multiple 3D convolutional layers with a kernel size of 3×1×1. Finally, a fully connected layer is used to obtain the appearance characteristics of each individual.

[0020] Step S2: Calculate the individual-to-relationship cube

[0021] (a) Node representation of the relationship graph

[0022] The node feature vectors of all M individuals in the scene are calculated by connecting physical characteristics and spatial features. i for:

[0023]

[0024] (b) Relationship graph edge representation

[0025] The computation node is the edge feature E between individuals i and j. i,j for:

[0026] E i,j =fc(concat(F i ,F j ),i,j∈1,2,…,M

[0027] The M×M edge features E between all individuals in the scene i,j Merging into an individual-to-relationship cube

[0028] (c) Long-term feature aggregation

[0029] A single-layer bidirectional GRU is used to construct the relation cube E of video segment k. k Short-term features aggregate into long-term features H k :

[0030] H k =GRU(E k H k-1 )

[0031] in, k = 1, 2, ..., K, where K is the number of segments in the entire video;

[0032] Step S3: Interactive Prediction

[0033] (a) Calculate the interaction matrix

[0034] The long-term characteristic matrix H k Compression is performed in the channel dimension using a two-layer fully connected layer to obtain a matrix. Flatten the matrix fc(H) into a one-dimensional vector v H :

[0035]

[0036] For vector v H Perform the softmax function operation and convert it into the original matrix form to obtain the interaction matrix R:

[0037] R = reshape(softmax(v) H ))

[0038] (b) Calculate the individual interaction category vector

[0039] For matrix H k The maximum value is taken sequentially along the Y dimension for compression:

[0040]

[0041] Input M into the fully connected layer and perform a softmax operation to obtain the individual interaction category vector v. i :

[0042]

[0043] (c) Calculate the global interaction category vector

[0044] For matrix H k The maximum value is taken sequentially along the X and Y dimensions for compression.

[0045]

[0046] Input m into the fully connected layer and perform a softmax operation to obtain the global interaction category vector g:

[0047]

[0048] Perform interactive prediction and recognition.

[0049] The detailed steps are as follows:

[0050] Step 1: Divide the input video sequence into equal-length video segments, sample each video segment at short intervals to obtain several video frames;

[0051] Step 2: For the i-th individual in frame t, use its 2D position vector Calculate the direction vector of motion A single-layer GRU is used to integrate the location information of the same individual across different video frames in the video segment.

[0052] Step 3: Use the asymmetric convolutional network Inception-v3 to extract multi-scale feature information within each bounding box. Adjust the extracted features to the same size using the Region of Interest (ROI) feature extraction RoIAlign. Aggregate temporal information using multiple 3D convolutional layers with a kernel size of 3×1×1. Use fully connected layers to obtain the appearance characteristics of each volume.

[0053] Step 4: Connect physical features and spatial characteristics to obtain the node feature vector F of all M individuals in the scene. i ;

[0054] Step 5: Connect the node feature vectors F of each entity i Obtain edge features E i,j And merge them into an individual-to-relationship cube E;

[0055] Step 6: Use a single-layer bidirectional GRU to aggregate the individual-to-relationship cube E into a long-term feature matrix H;

[0056] Step 7: Compress the long-term feature matrix H in the channel dimension using a two-layer fully connected layer and flatten it into a one-dimensional vector v. Perform a softmax operation on it and convert it into the original matrix form to obtain the interaction relation matrix R;

[0057] Step 8: Compress the long-term feature matrix H by taking the maximum value of each element in the Y dimension. Input the compressed matrix into a fully connected layer and perform a softmax operation to obtain the individual interaction category vector v. i ;

[0058] Step 9: Compress the long-term feature matrix H by taking the maximum value in each of the X and Y dimensions, input the compressed matrix into the fully connected layer, and perform a softmax operation to obtain the global interaction category vector g.

[0059] The features and beneficial effects of this invention are:

[0060] This invention examines all individuals in a scene to identify the interacting agents. Furthermore, unlike HOI (House of Interest), this invention does not rely on inherent prior knowledge based on common sense; each individual can interact with any other individual of any type. Therefore, this invention can be implemented more effectively. Attached image description:

[0061] Figure 1 : Schematic diagram of non-contact interaction categories and interaction subject identification methods. Detailed Implementation

[0062] The purpose of this invention is to propose a non-contact interaction category and interaction subject recognition technology for multi-person scenarios. It mainly solves the problem of recognizing interactive individuals and interactive actions when interactive individuals do not touch each other and there are other non-interactive individuals in the scene.

[0063] The technical solution adopted in this invention is: a method for identifying non-contact interaction categories and interaction subjects in multi-person scenarios. The following mainly introduces the network structure and specific operation steps used in this method.

[0064] I. Relationships embedded in networks:

[0065] This invention uses a relation pair embedding network (PRE-Net) to obtain relation pairs by aggregating individual and group information, as well as short-term and long-term information in a scene, while identifying non-contact interaction objects and their interaction categories.

[0066] II. Specific operating steps:

[0067] Step S1: Individual information extraction.

[0068] (a) Short-term video segmentation

[0069] Given an input video sequence, it is divided into K equal-length video segments. Each video segment is sampled at short intervals to obtain N video frames. For a video sequence with K segments, the final result is K×N video frames.

[0070] (b) Spatial Information Extraction

[0071] For the i-th individual in frame t, its 2D position vector Represented as:

[0072]

[0073] Calculate the direction vector of motion

[0074]

[0075] A single-layer GRU is used to integrate the location information of the same individual across N video frames in a video segment. For individual i in frame t, its location information is calculated.

[0076]

[0077] Ultimately, the spatial characteristics of each individual are obtained.

[0078] (c) Appearance feature extraction.

[0079] The Inception-v3 asymmetric convolutional network was used to extract multi-scale feature information within each bounding box. RoIAlign was used to resize the extracted features to the same size. Multiple 3D convolutional layers with a kernel size of 3×1×1 were used to aggregate temporal information. Finally, a fully connected layer was used to obtain the appearance characteristics of each individual.

[0080] Step S2: Calculate the individual-to-relationship cube

[0081] (d) Relationship graph node representation

[0082] The node feature vector F of all M individuals in the scene is calculated by connecting physical characteristics and spatial features. i for:

[0083]

[0084] (e) Relationship graph edge representation

[0085] Calculate the edge feature E between nodes (individuals) i and j. i,jfor:

[0086] E i,j =fc(concat(F i ,F j ),i,j∈1,2,…,M

[0087] The M×M edge features E between all individuals in the scene i,j Merging into an individual-to-relationship cube

[0088] (f) Long-term feature aggregation

[0089] A single-layer bidirectional GRU is used to construct the relation cube E of video segment k. k Short-term features aggregate into long-term features H k :

[0090] H k =GRU(E k H k-1 )

[0091] in, k = 1, 2, ..., K, where K is the number of segments in the entire video.

[0092] Step S3: Interactive Prediction

[0093] (d) Calculate the interaction matrix

[0094] The long-term characteristic matrix H k Compression is performed in the channel dimension (Z dimension) using a two-layer fully connected layer to obtain a matrix. Flatten the matrix fc(H) into a one-dimensional vector v H :

[0095]

[0096] For vector v H Perform a softmax operation and convert it into the original matrix form to obtain the interaction matrix R:

[0097] R = reshape(softmax(v) H ))

[0098] (e) Calculate the individual interaction category vector

[0099] For matrix H k The maximum value is taken sequentially along the Y dimension for compression:

[0100]

[0101] Input M into the fully connected layer and perform a softmax operation to obtain the individual interaction category vector v.i :

[0102]

[0103] (f) Calculate the global interaction category vector

[0104] For matrix H k The maximum value is taken sequentially along the X and Y dimensions for compression.

[0105]

[0106] Input m into the fully connected layer and perform a softmax operation to obtain the global interaction category vector g:

[0107]

[0108] The non-contact interaction category and interaction subject recognition method studied in this invention is specifically designed for multi-person scenarios, and its specific steps are as follows:

[0109] Step 1: Divide the input video sequence into equal-length video segments, and sample each video segment at short intervals to obtain several video frames.

[0110] Step 2: For the i-th individual in frame t, use its 2D position vector Calculate the direction vector of motion A single-layer GRU is used to integrate the location information of the same individual across different video frames in the video segment.

[0111] Step 3: Use the asymmetric convolutional network Inception-v3 to extract multi-scale feature information within each bounding box. Adjust the extracted features to the same size using the Region of Interest (ROI) feature extraction RoIAlign. Aggregate temporal information using multiple 3D convolutional layers with a kernel size of 3×1×1. Use fully connected layers to obtain the appearance characteristics of each volume.

[0112] Step 4: Connect physical features and spatial characteristics to obtain the node feature vector F of all M individuals in the scene. i .

[0113] Step 5: Connect the node feature vectors F of each entity i Obtain edge features E i,j And merge them into an individual-to-relationship cube E.

[0114] Step 6: Use a single-layer bidirectional GRU to aggregate the individual-to-relationship cube E into a long-term feature matrix H.

[0115] Step 7: Compress the long-term feature matrix H in the channel dimension (Z dimension) using a two-layer fully connected layer, and flatten it into a one-dimensional vector v. Perform a softmax operation on it and convert it into the original matrix form to obtain the interaction relation matrix R.

[0116] Step 8: Compress the long-term feature matrix H by taking the maximum value of each element in the Y dimension. Input the compressed matrix into a fully connected layer and perform a softmax operation to obtain the individual interaction category vector v. i .

[0117] Step 9: Compress the long-term feature matrix H by taking the maximum value in each of the X and Y dimensions, input the compressed matrix into the fully connected layer, and perform a softmax operation to obtain the global interaction category vector g.

[0118] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

[0119] References

[0120] Spchang

[0121] [1]Carreira J,Noland E,Hillier C,et al.A Short Note on the Kinetics-700Human Action Dataset[J].2019.

[0122] [2]Deng Z, Vahdat A, Hu H, et al. Structure Inference Machines: RecurrentNeural Networks for Analyzing Relations in Group Activity Recognition[C] / / 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2016.

Claims

1. A method for non-contact interaction detection in multi-object scenes, characterized in that, By utilizing the PRE-Net relation pair embedding network, relation pairs are obtained by aggregating individual and group information, as well as short-term and long-term information in the scene. Interaction prediction and recognition are then performed by calculating the interaction relation matrix, individual interaction category vectors, and global interaction category vectors. The specific steps are as follows: Step S1: Individual Information Extraction (a) Short-term video segmentation Given an input video sequence, it is divided into... Each video segment is sampled at short intervals to obtain a set of equal-length video clips. Video frames, for those with The video sequence of segments is ultimately obtained as follows: One video frame; (b) Spatial information extraction For the The first frame Each individual, its 2D position vector Represented as: ； Calculate the direction vector of motion : ； In the video segment A single-layer GRU is used to integrate the location information of the same individual within each video frame, for each frame. individuals on Calculate its location information : ； Ultimately, the spatial characteristics of each individual are obtained. ; (c) Appearance feature extraction The Inception-v3 asymmetric convolutional network is used to extract multi-scale feature information within each bounding box. The extracted features are then resized to the same size using RoIAlign, a region of interest feature extraction method. Temporal information is aggregated using multiple 3D convolutional layers with a kernel size of 3×1×1. Finally, a fully connected layer is used to obtain the appearance characteristics of each individual. ; Step S2: Calculate the individual-to-relationship cube (a) Node representation of the relationship graph By connecting physical characteristics and spatial features, all calculations in the scene are performed. The node feature vector of each individual for: ； (b) Relationship diagram edge representation Computation nodes are individuals , Edge features between for: ； Interactions between all individuals in the scene Each edge feature Merging into an individual-to-relationship cube ; (c) Long-term feature aggregation Use a single-layer bidirectional GRU to process video clips Relationship Cube Short-term features aggregate into long-term features : ； in, , This represents the number of segments in the entire video. Step S3: Interactive Prediction (a) Calculate the interaction matrix Long-term characteristic matrix Compression is performed in the channel dimension using a two-layer fully connected layer to obtain a matrix. , matrix Flattened into a one-dimensional vector : ； For vectors Perform a softmax function operation and convert it into the original matrix form to obtain the interaction matrix. : ； (b) Calculate the individual interaction category vector For matrix ,exist Each dimension is compressed by taking its maximum value. ； Will Input the fully connected layer and perform a softmax operation to obtain the individual interaction category vector. : ； (c) Calculate the global interaction category vector For matrix ,exist Each dimension is compressed by taking its maximum value. ； Will Input the fully connected layer and perform a softmax operation to obtain the global interaction category vector. : ； Perform interactive prediction and recognition.

Images