Video human-object interaction detection method based on hierarchical space-time diagram network

Abstract

The invention discloses a video human-object interaction detection method based on a hierarchical space-time diagram network, and the method comprises the steps: extracting the features of human and object instances from a given video, the features comprising human visual features, object visual features and geometric features; designing a time-enhanced recurrence plot network branch module, and obtaining comprehensive enhanced features of each person and object instance; a parallel Transform encoder branch module is designed, and recursive enhanced feature vectors of human and object instances are obtained; fusing the comprehensive enhanced features of each person and object instance and the recursive enhanced feature vectors of the person and object instance to obtain a predicted segmentation boundary; and learning segment-level features according to the segmentation boundary and the time-enhanced features of the person and the object, and predicting the interaction action of the person and the object instance. According to the invention, an effective detection result can be provided in a scene in which an object is operated by a single person without shielding, a plurality of persons with a large number of persons and object shielding, and the object is operated by two hands.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a video human-object interaction detection method based on a hierarchical spatiotemporal graph network. Background Art

[0002] The interaction between humans and objects plays a key role in providing important clues for understanding the surrounding environment. Human-Object Interaction (HOI) detection, as a key basic task for human-machine collaboration and human activity understanding, mainly involves human behavior in object manipulation. Therefore, the research on HOI detection is receiving more and more attention.

[0003] In the relevant research in recent years, HOI detection is mainly divided into image-based methods and video-based methods. Image-based methods only study interactions in static images, which limits the exploration of dynamic human actions, and some dynamic interactions (such as closing and opening doors) may not be distinguished from static images. To solve this problem, some researchers have proposed video-based human-object interaction detection methods, which use the spatiotemporal information in the video to help detect dynamic actions.

[0004] Initial methods for human-object interaction detection based on videos rely on convolutional neural networks or recurrent neural networks to learn spatiotemporal information in videos. However, these methods have difficulty in effectively extracting human-object relationship information within and between video frames. To address this limitation, researchers have proposed methods based on spatiotemporal graph convolutional networks to learn spatiotemporal information. For example, Sunkesula et al. proposed a method using graph convolution and recurrent neural networks to capture hierarchical information of spatiotemporal clues of visual features. They used graph convolutional networks to model spatial interactions and then used recurrent networks to explore temporal correlations. However, compared with the joint model method, this sequential method may not be able to fully capture spatiotemporal information. Therefore, Wang et al. proposed a spatiotemporal graph convolutional network across the spatiotemporal dimensions to learn human-object interaction detection in videos. The above graph convolutional network-based methods usually assume prior knowledge of the temporal structure of the video and focus mainly on the sub-activity labels of the detection clips. To address this problem, Morais et al. proposed ASSIGN, a joint labeling and segmentation method for human-object interaction detection in videos. ASSIGN can not only identify human-object interactions, but also mark the boundaries of each activity clip on the timeline. The method adopts a novel framework to capture the spatiotemporal information of visual features using recursive graph networks.

[0005] Although existing methods use concise and efficient graph network structures to model temporal relationships, they all ignore the hierarchical temporal information in videos. Therefore, it is particularly important to more comprehensively understand the interactive information between people and objects at different time scales. Summary of the invention

[0006] To solve the above technical problems, the present invention proposes a video human-object interaction detection method based on a hierarchical spatiotemporal graph network, which can capture models of temporal relationships of different granularities, including inter-frame and long-sequence time information, for the joint labeling and segmentation tasks of human-object interaction detection in videos, and can more comprehensively understand the interaction information between people and objects at different time scales.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for detecting human-object interaction in a video based on a hierarchical spatiotemporal graph network, the method comprising the following steps: Step 1: extracting features of human and object instances from a given video, wherein the features include human visual features, object visual features, and geometric features; Step 2: Design a temporal enhanced recursive graph network branch module to construct inter-frame and intra-frame connections of human visual features, object visual features, and geometric features to obtain comprehensive enhanced features for each person and object instance; Step 3: Design a parallel Transformer encoder branch module to extract the spatial and long-term temporal information of human visual features, object visual features, and geometric features, and obtain recursively enhanced feature vectors of human and object instances respectively; Step 4: Utilize the comprehensive enhanced features of each person and object instance obtained by fusing the person segment predictor and the object segment predictor, and the recursive enhanced feature vectors of the person and object instances to obtain the predicted segmentation boundary; Step 5: Learn segment-level features based on the segmentation boundaries predicted by the segment predictor of people and objects and the temporal enhancement features of people and objects, and use the obtained segment-level features to predict the interaction actions of people and object instances and perform interaction behavior analysis.

[0008] The beneficial effects of the present invention are: The time-enhanced recursive graph network branch proposed in the present invention is equipped with a grouping fusion mechanism to capture spatial relationships and short-sequence temporal information. This branch not only considers the complementarity of inter-frame information, but also emphasizes the importance of information fusion mechanism. In addition, the time-enhanced recursive graph network TRGN and the parallel Transformer encoder PTE are integrated to extract spatiotemporal features in the video, which can make full use of hierarchical temporal information, including inter-frame correlations and long-sequence temporal contexts, to improve the accuracy of video-based human-object interaction detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a principle block diagram of the method for detecting human-object interaction in video based on a hierarchical spatiotemporal graph network of the present invention; Figure 2 Schematic diagram of inter-frame connections of human nodes. DETAILED DESCRIPTION

[0010] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0011] like Figure 1 As shown in the figure, the video human-object interaction detection method based on hierarchical spatiotemporal graph network proposed in the present invention consists of five parts: video feature extraction, temporal enhancement recursive graph network (TRGN) branch, parallel Transformer encoder (PTE) branch, fragment predictor and fragment-level processing. Among them, the TRGN branch constructs inter-frame and intra-frame connections in the recursive graph, and only the shaded nodes in the graph receive temporal enhancement information through inter-frame connections. The PTE branch contains two parallel spatial and temporal transform encoders for extracting spatial and long-term temporal information in the video. Through these two novel branches, the model makes full use of hierarchical temporal information, including inter-frame and long-term temporal context, to improve the accuracy of video-based human-object interaction detection.

[0012] The specific implementation of each part is introduced below.

[0013] First, in the video feature extraction part, we first extract instance features from a given video, use the target detector Fast R-CNN to detect human and object instances in the video frame, and then associate candidate instances in different frames through multi-target tracking technology. Each candidate instance has its bounding box and category label, which can only be a person or an object; based on the detected bounding box, human visual features and object visual features are extracted from the video frame; and then geometric features are further extracted based on the human skeleton and object bounding box. In this way, human visual features, object visual features, and geometric features are obtained.

[0014] Second, the temporal enhanced recurrent graph network (TRGN) branch part, considering the temporal characteristics of the video, first uses a bidirectional gated recurrent unit (BiGRU) to encode temporal information from human visual features, object visual features and geometric features to obtain temporal features, and then splices the video features extracted in the first part with the temporal features to enrich the spatiotemporal information. In order to further capture the temporal features in adjacent video frames, a temporal enhanced recurrent graph network (TRGN) branch module is designed to introduce inter-frame information into the model. For human node features, only object node features in different frames are considered, while other human node features are ignored, because the task goal is to focus on the relationship between humans and objects. Similarly, for object nodes, only human node features in different frames are considered, while other object node features are ignored.

[0015] For human node features, other human nodes in the current frame pass information to it in the following ways: , in, Indicates Frame The fusion features of the person are obtained by splicing the video features and time features of the individual person nodes. Indicates Frame The fusion features of the person are obtained by splicing the video features and time features of the individual person nodes. Indicates Adjacent in frame The fused features passed to the person by all other human nodes in the frame Frame-level intra-class features. The operation represents calculating the Euclidean distance between the features of two human entities, and then using these distances to calculate the weighted average of the contributions of the message senders.

[0016] Human integration characteristics The time-enhanced information transfer process is as follows Figure 2 As shown, for example, assuming that there are two objects in the video, the fusion feature of the person , it is necessary to consider the adjacent The object node features in the frame include the first object node features And the second object node feature ,First, the information between the object node features and the human node features in different frames is calculated as follows: , in, It is The fused features of all object nodes in the frame. Indicates The object node features in the frame are passed to the Fusion features of people in the frame Frame-level inter-class messages.

[0017] Next, the inter-class messages from different frames are fused to extract the fused features of the person Since the impact of objects in the current frame on detection should take precedence over that in other frames, this paper proposes a group fusion mechanism that first fuses inter-class messages from adjacent frames and uses a multi-layer perceptron (MLP) to project the connected inter-class messages into the original space, and then combines the obtained output with the Frame-level inter-class messages The process can be described as follows: , in, Indicates Frame Individual time-enhanced inter-class features, that is, time-enhanced inter-class features of people.

[0018] Similarly, the frame-level intra-class features of object node features can be calculated as follows: , in, Indicates Frame The fusion features of the objects, Is adjacent The fusion features of all other object node features in the frame are passed to the object The frame-level intra-class features of , that is, the frame-level intra-class features of the object.

[0019] The calculation process of the time-enhanced intra-class message is as follows: , , in, It is Fusion features of all persons in the frame. Indicates The human node features in the frame are passed to the Fusion features of objects in the frame Frame-level inter-class messages. Indicates Frame The temporal enhanced inter-class features of an object, that is, the temporal enhanced inter-class features of the object.

[0020] In addition, since the geometric information is only passed to the object node, further calculation of the fusion passed to the object Geometric information: , in, It is The fused features of all geometric features in the frame. express The fused features passed to the object in the tth frame Finally, by connecting the received geometric information with the enhanced features of human visual features, object visual features and geometric features, the comprehensive enhanced features of each human and object entity can be obtained.

[0021] Third, the parallel Transformer encoder (PTE) branch contains two parallel encoders for extracting spatial and long-term temporal information in the video, which can reduce information loss compared to the sequential structure. For the spatial Transformer encoder, the original input features are first connected in the spatial dimension to obtain the input of the spatial Transformer encoder. Then, the temporal dimension is regarded as a batch dimension, and the encoder is used to learn the spatial information between all video frames.

[0022] In order to further extract the global temporal information of each entity, a temporal Transformer encoder is used in parallel with the spatial Transformer encoder to enrich the long-term temporal context. The operation of the temporal Transformer encoder is similar to that of the spatial Transformer encoder. The temporal dimension is treated as a batch dimension. Then, the human visual features, object visual features, and geometric features are concatenated along the temporal dimension. Next, the concatenation operator is used to fuse the outputs obtained from the two parallel Transformer encoders to obtain the spatiotemporal features, which are then projected into the original space using MLP to obtain the spatiotemporal features of human and object instances, respectively. Finally, we segment along a single dimension to extract the spatiotemporal features of human and object entities, and continue to use BiGRU to further encode the recursive enhanced feature vectors of people and objects. and .

[0023] Fourth, the fragment predictor part, based on the human and object node features obtained from the above outputs, the frame-level intra-class features and time-enhanced inter-class features of humans, the frame-level intra-class features and time-enhanced inter-class features of objects, the recursive enhancement vectors of humans and objects, the geometric transfer features of objects, and the comprehensive enhancement features of human and object entities, fuses them to extract the final spatiotemporal features and uses them to predict the segmentation boundaries, as shown below: , , in, and Respectively represent The probability of the frame being the segmentation boundary of human and object entities. Then, the Gumbel-Softmax (GSM) operator is used to convert these soft outputs of human and object entities into binary outputs. When the binary output is 1, it means that the first Frame is the last frame of the entity's current segment. Conversely, an output of 0 indicates the Frames are not segmentation boundaries. Figure 1In , the working principle of human fragment prediction is basically the same as that of object fragment prediction. The only difference is that the object fragment predictor takes into account the geometric transfer features of the object, while the human fragment prediction does not.

[0024] Fifth, the fragment-level processing part, whose process is used to learn segment-level features for each entity based on the time-enhanced features of the boundary and timing-enhanced RGNN branches provided by the fragment predictor. Using the segment update operation, the segment-level features are sparsely updated according to the boundary information. Finally, the Softmax function is used to use the updated segment-level features to predict the segment label of the entity, and the probability of obtaining the segment label of the human sub-activity or object affordance is calculated to achieve the final prediction. It can accurately identify the interaction between individuals and objects in the scene of single-person operation of objects without occlusion, and provide clear interaction behavior analysis. Even in a multi-person operation environment with a large number of people and objects occluded, the system can use deep learning algorithms to effectively handle occlusion problems, accurately predict and identify the complex interaction relationship between each person and object. In addition, in scenes involving two-handed operation of objects, the system can distinguish and track the actions of both hands and accurately predict the interaction process between both hands and objects.

[0025] In summary, the video human-object interaction detection method based on hierarchical spatiotemporal graph network proposed in the present invention has achieved competitive results in the following three scenarios: 1) single-person operation of objects without occlusion; 2) multi-person operation with a large number of people and objects occluded; 3) scenarios involving two-handed operation of objects, which improves the accuracy of video-based human-object interaction detection.

[0026] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A video human-object interaction detection method based on a hierarchical spatiotemporal graph network, characterized in that: The method comprises the following steps: Step 1: extracting features of human and object instances from a given video, wherein the features include human visual features, object visual features, and geometric features; Step 2: Design a temporal enhanced recursive graph network branch module to construct inter-frame and intra-frame connections of human visual features, object visual features, and geometric features to obtain comprehensive enhanced features for each person and object instance; Step 3: Design a parallel Transformer encoder branch module to extract the spatial and long-term temporal information of human visual features, object visual features, and geometric features, and obtain recursively enhanced feature vectors of human and object instances respectively; Step 4: Utilize the comprehensive enhanced features of each person and object instance obtained by fusing the person segment predictor and the object segment predictor, and the recursive enhanced feature vectors of the person and object instances to obtain the predicted segmentation boundary; Step 5: Learn segment-level features based on the segmentation boundaries predicted by the segment predictor of people and objects and the temporal enhancement features of people and objects, and use the obtained segment-level features to predict the interaction actions of people and object instances and perform interaction behavior analysis.

2. According to claim 1, a method for detecting human-object interaction in a video based on a hierarchical spatiotemporal graph network is characterized in that: The step 1 comprises: using the target detector Fast R-CNN to detect people and object instances in the video frame, associating candidate instances in different frames through multi-target tracking technology, each candidate instance includes a bounding box and a category label, extracting human visual features and object visual features from the video frame according to the detected bounding box; and extracting geometric features according to the human skeleton and the bounding box of the object.

3. The method for detecting human-object interaction in video based on hierarchical spatiotemporal graph network according to claim 1, characterized in that: The step 2 includes: for the human node feature, other human nodes in the current frame transmit information to it in the following manner: ， in, Indicates Frame The fusion features of the person are obtained by splicing the video features and time features of the individual person nodes. Indicates Frame The fusion features of the person are obtained by splicing the video features and time features of the individual person nodes. Indicates Adjacent in frame The fused features passed to the person by all other human nodes in the frame Frame-level intra-class features, The operation represents the calculation of the Euclidean distance between the features of two human entities; The information relationship between object node features and human node features in different frames is: ， in, It is The fusion features of all object nodes in the frame, Indicates The object node features in the frame are passed to the Fusion features of people in the frame The inter-class messages from adjacent frames are fused and the connected inter-class messages are projected into the original space using a multi-layer perceptron (MLP). The resulting output is then combined with the inter-class messages from the first frame. Frame-level inter-class messages To perform the fusion: ， in, Indicates Frame Individual time-enhanced inter-class features, i.e., time-enhanced inter-class features of people; Object node features Features are calculated as follows: ， in, Indicates Frame The fusion features of the objects, Is adjacent The fusion features of all other object node features in the frame are passed to the object The frame-level intra-class features of , that is, the frame-level intra-class features of the object; The calculation process of the temporal enhancement intra-class message of an object is as follows: ， ， in, It is The fused features of all people in the frame, Indicates The human node features in the frame are passed to the Fusion features of objects in the frame Frame-level inter-class messages, Indicates Frame The temporal enhanced inter-class features of an object, that is, the temporal enhanced inter-class features of the object; Fusion passed to object Geometric information: ， in, It is The fusion features of all geometric features in the frame, express The fused features passed to the object in the tth frame The geometric information of the object, that is, the geometric transmission features of the object, is connected with the received geometric information and the enhanced features of human visual features, object visual features and geometric features to obtain the comprehensive enhanced features of each person and object instance.

4. The method for detecting human-object interaction in video based on hierarchical spatiotemporal graph network according to claim 1, characterized in that: The step 3 comprises: for the spatial Transformer encoder, firstly connecting the original features of the input in the spatial dimension to obtain the input of the spatial Transformer encoder, then considering the time dimension as a batch dimension, and using the encoder to learn the spatial information between all video frames; For the temporal Transformer encoder, the time dimension is regarded as the batch dimension. Then, the human visual features, object visual features and geometric features are connected along the time dimension. The outputs obtained from the two parallel Transformer encoders are fused using the connection operator to obtain the spatiotemporal features. The multi-layer perceptron MLP is used to project the spatiotemporal features into the original space to obtain the spatiotemporal features of human and object instances respectively. , segment along a single dimension to extract the spatiotemporal features of human and object entities, and use BiGRU network encoding to obtain recursive enhanced feature vectors of people and objects and .

5. The method for detecting human-object interaction in video based on hierarchical spatiotemporal graph network according to claim 1, characterized in that: The step 4 includes: fusing the obtained node features of humans and objects, the frame-level intra-class features and time-enhanced inter-class features of humans, the frame-level intra-class features and time-enhanced inter-class features of objects, the recursive enhancement vectors of humans and objects, the geometric transfer features of objects, and the comprehensive enhancement features of human and object instances to extract the final spatiotemporal features and predict the segmentation boundary, as shown below: ， ， in, and Respectively represent The probability of the frame being the segmentation boundary of humans and objects is used. The soft output of humans and objects is converted into binary output using the Gumbel-Softmax operator. When the binary output is 1, it means the first Frame is the last frame of the current segment of the corresponding entity. On the contrary, when the output is 0, it means the Frames are not segmentation boundaries.

6. The method for detecting human-object interaction in video based on hierarchical spatiotemporal graph network according to claim 1, characterized in that: The step 5 includes: learning segment-level features for each entity based on the segmentation boundaries provided by the fragment predictor of people and objects and the temporal enhancement features of the temporal enhancement branch, using the segment update operation to sparsely update the segment-level features according to the segmentation boundary information, and then using the Softmax function to use the obtained updated segment-level features to predict the segment label of the entity, and calculating the probability of obtaining the segment label of the human sub-activity or object availability to achieve the final prediction.

Images