Video human-object interaction detection method based on hierarchical spatio-temporal graph network

Abstract

The application discloses a video human-object interaction detection method based on a hierarchical spatiotemporal graph network, and the method comprises the following steps: extracting features of human and object instances from a given video, wherein the features comprise human visual features, object visual features and geometric features; designing a time enhancement recurrent graph network branch module to obtain comprehensive enhancement features of each human and object instance; designing a parallel Transformer encoder branch module to obtain a recurrent enhancement feature vector of the human and object instances; fusing the comprehensive enhancement features of each human and object instance and the recurrent enhancement feature vector of the human and object instances to obtain a predicted segmentation boundary; and learning segment-level features according to the segmentation boundary and the time enhancement features of the human and object to predict interaction actions of the human and object instances. The application can provide effective detection results in the following scenes: single-person operation of an object without occlusion, multi-person operation involving a large number of human and object occlusions, and a scene involving two-hand operation of an object.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a video human-object interaction detection method based on hierarchical spatiotemporal graph networks. Background Technology

[0002] Human-object interactions play a crucial role in providing important clues for understanding the surrounding environment. Human-object interaction (HOI) detection, as a key foundational task for human-computer collaboration and understanding human activities, primarily involves human behavior related to object manipulation. Therefore, research on HOI detection is receiving increasing attention.

[0003] In recent years, HOI detection has been mainly divided into image-based methods and video-based methods. Image-based methods only study interactions in static images, limiting the exploration of dynamic human actions, and some dynamic interactions (such as closing and opening doors) may not be distinguishable from static images. To address this issue, some researchers have proposed video-based human-object interaction detection methods, which utilize spatiotemporal information in videos to help detect dynamic actions.

[0004] Early video-based human-object interaction detection methods relied on convolutional neural networks (CNNs) or recurrent neural networks (RNNs) to learn spatiotemporal information from the video. However, these methods struggled to effectively extract intra- and inter-frame human-object relationship information. To address this limitation, researchers proposed methods based on spatiotemporal graph convolutional networks (SPCNNs) to learn spatiotemporal information. For example, Sunkesula et al. proposed a method utilizing graph convolutions and RNNs to capture hierarchical information from spatiotemporal cues in visual features. They employed a graph convolutional network to model spatial interactions and then used a recurrent network to explore temporal relevance. However, this sequential approach may not adequately capture spatiotemporal information compared to joint modeling methods. Therefore, Wang et al. proposed a spatiotemporal graph convolutional network spanning spatiotemporal dimensions to learn human-object interaction detection in videos. The aforementioned graph convolutional network-based methods typically assume prior knowledge of the video's temporal structure and primarily focus on sub-activity labels for detected segments. To address this issue, Morais et al. proposed ASSIGN, a joint labeling and segmentation method for human-object interaction detection in videos. ASSIGN not only identifies human-object interactions but also marks the boundaries of each activity segment on the timeline. This method employs a novel framework that utilizes recurrent graph networks to capture the spatiotemporal information of visual features.

[0005] While existing methods utilize concise and efficient graph network structures to model temporal relationships, they all neglect the hierarchical temporal information within videos. Therefore, research into a more comprehensive understanding of human-object interaction information across different time scales is particularly important. Summary of the Invention

[0006] To address the aforementioned technical issues, this invention proposes a video human-object interaction detection method based on hierarchical spatiotemporal graph networks. This method can capture models of temporal relationships at different granularities, including inter-frame and long-sequence temporal information, for joint annotation and segmentation tasks in human-object interaction detection in videos. It can provide a more comprehensive understanding of the interaction information between people and objects at different time scales.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A video human-object interaction detection method based on hierarchical spatiotemporal graph networks, the method comprising the following steps: Step 1: Extract features of people and object instances from the given video, including human visual features, object visual features, and geometric features; Step 2: Design the temporal augmentation recursive graph network branch module to construct inter-frame and intra-frame connections for human visual features, object visual features, and geometric features, and obtain the comprehensive augmentation features of each person and object instance; Step 3: Design a parallel Transformer encoder branch module to extract spatial and long-term temporal information of human visual features, object visual features, and geometric features, and obtain recursive enhanced feature vectors for human and object instances respectively. Step 4: Utilize the combined enhanced features of each person and object instance obtained by fusing the fragment predictor of people and the fragment predictor of objects, as well as the recursive enhanced feature vectors of people and object instances, to obtain the predicted segmentation boundary. Step 5: Learn segment-level features based on the segmentation boundaries predicted by the segment predictor for people and objects and the temporal augmentation features for people and objects. Use the obtained segment-level features to predict the interaction actions of people and object instances and perform interaction behavior analysis.

[0008] The beneficial effects of this invention are as follows: The Temporally Augmented Recurrent Graph Network (TRGN) branch proposed in this invention is equipped with a grouping fusion mechanism to capture spatial relationships and short-sequence temporal information. This branch not only considers the complementarity of inter-frame information but also emphasizes the importance of the information fusion mechanism. In addition, the integration of the TRGN and the Parallel Transformer Encoder (PTE) for extracting spatiotemporal features from videos can fully utilize hierarchical temporal information, including inter-frame correlations and long-sequence temporal context, to improve the accuracy of video-based human-object interaction detection. Attached Figure Description

[0009] Figure 1 This is a block diagram illustrating the principle of the video human-object interaction detection method based on hierarchical spatiotemporal graph networks of the present invention. Figure 2 This is a schematic diagram of inter-frame connections for human nodes. Detailed Implementation

[0010] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0011] like Figure 1 As shown, the video human-object interaction detection method based on hierarchical spatiotemporal graph networks proposed in this invention consists of five parts: video feature extraction, a temporally augmented recurrent graph network (TRGN) branch, a parallel Transformer encoder (PTE) branch, a segment predictor, and segment-level processing. The TRGN branch constructs inter-frame and intra-frame connections in the recurrent graph; only shaded nodes receive temporal augmentation information through inter-frame connections. The PTE branch contains two parallel spatial and temporal transformation encoders for extracting spatial and long-term temporal information from the video. Through these two novel branches, the model fully utilizes hierarchical temporal information, including inter-frame and long-term temporal context, to improve the accuracy of video-based human-object interaction detection.

[0012] The specific implementation of each part will be described below.

[0013] First, in the video feature extraction section, instance features are extracted from the given video. The Fast R-CNN object detector is used to detect human and object instances in the video frames. Then, multi-object tracking technology is used to associate candidate instances from different frames. Each candidate instance has its bounding box and class label, which can only be either a person or an object. Based on the detected bounding boxes, human visual features and object visual features are extracted from the video frames. Furthermore, geometric features are extracted based on the human skeleton and object bounding boxes. In this way, human visual features, object visual features, and geometric features are obtained.

[0014] Secondly, in the Temporally Augmented Recurrent Graph Network (TRGN) branch, considering the temporal characteristics of video, a bidirectional gated recurrent unit (BiGRU) is first used to encode temporal information from human visual features, object visual features, and geometric features to obtain temporal features. Then, the video features extracted in the first part are concatenated with the temporal features to enrich the spatiotemporal information. To further capture the temporal features in adjacent video frames, a Temporally Augmented Recurrent Graph Network (TRGN) branch module is designed to introduce inter-frame information into the model. For human node features, only object node features in different frames are considered, while other human node features are ignored, because the task objective is to focus on the relationship between humans and objects. Similarly, for object nodes, only human node features in different frames are considered, while other object node features are ignored.

[0015] For human node features, other human nodes in the current frame pass information to them in the following ways: , in, Indicates the first The first frame The fused human features are obtained by stitching together video and temporal features from individual human nodes. Indicates the first The first frame The fused human features are obtained by stitching together video and temporal features from individual human nodes. Indicates the first Adjacent frames The fusion features passed to the human by all other human nodes in the frame Frame-level intra-class features. The operation represents calculating the Euclidean distance between the features of two human entities, and then using these distances to calculate the weighted average of the information sender's contribution.

[0016] Human integration characteristics Time-enhanced information transmission process, such as Figure 2 As shown, for example, assuming that two objects appear in the video, then for the fusion features of a person... Adjacent The object node features in the frame include the first object node features. and the second object node features First, the information between object node features and human node features in different frames is calculated as follows: , in, It is the first The fusion feature of all object nodes in the frame. Indicates the first The object node features in the frame are passed to the first Frame-by-frame human fusion features Frame-level inter-class messages.

[0017] Next, inter-class messages from different frames will be fused to extract human fusion features. Temporally enhanced inter-class features. Since the impact of objects in the current frame on detection should take precedence over those in other frames, this invention proposes a grouping fusion mechanism. First, inter-class messages from adjacent frames are fused, and then a multilayer perceptron (MLP) is used to project the connected inter-class messages onto the original space. Finally, the resulting output is combined with information from the first frame. Frame-level inter-class messages The fusion process can be described as follows: , in, Indicates the first The first frame Individual time-enhancing inter-class features, i.e., time-enhancing inter-class features of individuals.

[0018] Similarly, the frame-level intra-class features of object node features can be calculated in the following way: , in, Indicates the first The first frame The fusion characteristics of individual objects are adjacent The fused features passed to the object by all other object node features in the frame. The frame-level intra-class features, i.e., the frame-level intra-class features of objects.

[0019] The calculation process for time-enhanced messages within the class is as follows: , , in, It is the first The fused features of all people in the frame. Indicates the first Human node features in the frame are passed to the first Blending features of objects in a frame Frame-level inter-class messages. Indicates the first The first frame Temporal augmentation between classes of an object, i.e., temporal augmentation between classes of an object.

[0020] Furthermore, since geometric information is only passed to object nodes, the fusion passed to the object is further calculated. Geometric information: , in, It is the first The fusion feature of all geometric features in the frame. express Blending features passed to objects in frame t The geometric information, i.e., the geometric transfer characteristics of the object, is then obtained. Finally, by connecting the received geometric information with the enhanced features of human visual features, object visual features, and geometric features, a comprehensive enhanced feature for each human and object entity can be obtained.

[0021] Third, the Parallel Transformer Encoder (PTE) branch contains two parallel encoders for extracting spatial and long-term temporal information from the video, which reduces information loss compared to a sequential structure. For the spatial Transformer encoder, the original features of the input are first concatenated in the spatial dimension to obtain the input of the spatial Transformer encoder. Then, the temporal dimension is treated as a batch processing dimension, and the encoder is used to learn the spatial information between all video frames.

[0022] To further extract global temporal information for each entity, a temporal Transformer encoder, operating in parallel with the spatial Transformer encoder, is employed to enrich the long-order temporal context. The temporal Transformer encoder operates similarly to the spatial Transformer encoder, treating the temporal dimension as a batch processing dimension. Then, it concatenates human visual features, object visual features, and geometric features along the temporal dimension. Next, concatenation operators are used to fuse the outputs from the two parallel Transformer encoders to obtain spatiotemporal features. Finally, an MLP is used to project these spatiotemporal features onto the original space, yielding the spatiotemporal features for both human and object instances. Finally, segmentation along a single dimension is used to extract spatiotemporal features of human and object entities, and BiGRU is further used for encoding to obtain recursively enhanced feature vectors of people and objects. and .

[0023] Fourth, the segment predictor part, based on the human and object node features obtained from the above output, the frame-level intra-class features and temporally enhanced inter-class features of humans, the frame-level intra-class features and temporally enhanced inter-class features of objects, the recursive enhancement vectors of humans and objects, the geometric transitive features of objects, and the comprehensive enhancement features of human and object entities, these are fused to extract the final spatiotemporal features, and they are used to predict the segmentation boundary, as shown below: , , in, and They represent the first The frame represents the probability of segmenting human and object entities. Then, the Gumbel-Softmax (GSM) operator is used to convert these soft outputs of human and object entities into binary outputs. A binary output of 1 indicates that the frame represents the boundary between human and object entities. The frame is the last frame of the current segment of this entity. Conversely, an output of 0 indicates the last frame. The frame is not a segmentation boundary. Among them, Figure 1In this context, human segment prediction works on a similar principle to object segment prediction. The only difference is that object segment predictors consider the geometric transfer characteristics of objects, while human segment predictions do not.

[0024] Fifth, the segment-level processing section learns segment-level features for each entity based on the boundary and temporal augmentation features provided by the segment predictor and the temporal augmentation features of the RGNN branch. Using a segment update operation, the segment-level features are sparsely updated based on boundary information. Finally, the Softmax function is used to predict the segment labels of the entities using the obtained updated segment-level features, calculating the probability of obtaining human sub-activities or object availability segment labels, and achieving the final prediction. This allows for accurate identification of interactions between individuals and objects in unoccluded single-person object manipulation scenarios, providing clear interaction behavior analysis. Even in multi-person operation environments with numerous people and object occlusions, the system can effectively handle occlusion problems using deep learning algorithms, accurately predicting and identifying complex interaction relationships between individuals and objects. Furthermore, in scenarios involving two-handed object manipulation, the system can distinguish and track hand movements, accurately predicting the interaction process between hands and objects.

[0025] In summary, the video human-object interaction detection method based on hierarchical spatiotemporal graph network proposed in this invention has achieved competitive results in the following three scenarios: 1) single-person operation of objects without occlusion; 2) multi-person operation with a large number of people and objects occluding objects; and 3) scenarios involving two-handed operation of objects, thus improving the accuracy of video-based human-object interaction detection.

[0026] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video human-object interaction detection method based on hierarchical spatiotemporal graph networks, characterized in that, The method includes the following steps: Step 1: Extract features of people and object instances from the given video, including human visual features, object visual features, and geometric features; Step 2: Design the temporal augmentation recurrent graph network branch module to construct inter-frame and intra-frame connections for human visual features, object visual features, and geometric features, obtaining comprehensive augmented features for each person and object instance; including: for human node features, other human nodes in the current frame pass information to them in the following ways: ， in, Indicates the first The first frame The fused human features are obtained by stitching together video and temporal features from individual human nodes. Indicates the first The first frame The fused human features are obtained by stitching together video and temporal features from individual human nodes. Indicates the first Adjacent frames The fusion features passed to the human by all other human nodes in the frame Frame-level intra-class features, The operation represents calculating the Euclidean distance between two human entity features; The information relationship between object node features and human node features in different frames is as follows: ， in, It is the first The fusion features of all object nodes in the frame. Indicates the first The object node features in the frame are passed to the first Frame-by-frame human fusion features The frame-level inter-class messages are fused; inter-class messages from adjacent frames are fused, and the connected inter-class messages are projected onto the original space using a multilayer perceptron (MLP). The resulting output is then combined with the messages from the first frame. Frame-level inter-class messages To merge: ， in, Indicates the first The first frame Individual time-enhancing inter-class features, i.e., human time-enhancing inter-class features; Object node features Features are calculated in the following way: ， in, Indicates the first The first frame The fusion characteristics of individual objects are adjacent The fused features passed to the object by all other object node features in the frame. The frame-level intra-class features, i.e., the frame-level intra-class features of objects; The calculation process for the time-enhanced messages within an object's class is as follows: ， ， in, It is the first The fused features of all people in the frame, Indicates the first Human node features in the frame are passed to the first Blending features of objects in a frame Frame-level inter-class messages, Indicates the first The first frame Temporal augmentation inter-class features of an object, i.e., temporal augmentation inter-class features of an object; fusion passed to objects Geometric information: ， in, It is the first The fusion feature of all geometric features in the frame. express Blending features passed to objects in frame t The geometric information, namely the geometric transfer features of the object, is obtained by connecting the received geometric information with human visual features, object visual features and geometric features to obtain the comprehensive enhanced features of each person and object instance. Step 3: Design a parallel Transformer encoder branch module to extract spatial and long-term temporal information of human visual features, object visual features, and geometric features, and obtain recursive enhanced feature vectors for human and object instances respectively. Step 4: Utilize the combined enhanced features of each person and object instance obtained by fusing the fragment predictor of people and the fragment predictor of objects, as well as the recursive enhanced feature vectors of people and object instances, to obtain the predicted segmentation boundary. Step 5: Learn segment-level features based on the segmentation boundaries predicted by the segment predictor for people and objects and the temporal augmentation features for people and objects. Use the obtained segment-level features to predict the interaction actions of people and object instances and perform interaction behavior analysis.

2. The video human-object interaction detection method based on hierarchical spatiotemporal graph networks according to claim 1, characterized in that, Step 1 includes: using the object detector Fast R-CNN to detect human and object instances in video frames; using multi-object tracking technology to associate candidate instances in different frames, each candidate instance including a bounding box and a category label; extracting human visual features and object visual features from the video frames based on the detected bounding boxes; and extracting geometric features based on the human skeleton and the bounding boxes of objects.

3. The video human-object interaction detection method based on hierarchical spatiotemporal graph networks according to claim 1, characterized in that, Step 3 includes: for the spatial Transformer encoder, firstly, the original features of the input are concatenated in the spatial dimension to obtain the input of the spatial Transformer encoder; then, the temporal dimension is treated as the batch processing dimension, and the encoder is used to learn the spatial information between all video frames. For the temporal Transformer encoder, the temporal dimension is treated as a batch processing dimension. Then, human visual features, object visual features, and geometric features are concatenated along the temporal dimension. The outputs obtained from the two parallel Transformer encoders are fused using concatenation operators to obtain spatiotemporal features. A multilayer perceptron (MLP) is then used to project these spatiotemporal features onto the original space to obtain the spatiotemporal features of human and object instances, respectively. The system segments along a single dimension to extract spatiotemporal features of human and object entities, and uses a BiGRU network to encode recursively enhanced feature vectors of people and objects. and .

4. The video human-object interaction detection method based on hierarchical spatiotemporal graph networks according to claim 3, characterized in that, Step 4 includes fusing the obtained human and object node features, human frame-level intra-class features and temporally enhanced inter-class features, object frame-level intra-class features and temporally enhanced inter-class features, human and object recursive enhancement vectors, object geometric transit features, and comprehensive enhancement features of human and object instances to extract the final spatiotemporal features and predict the segmentation boundary, as shown below: ， ， in, and They represent the first The frame, representing the probability of the boundary between human and object entities, is converted from its soft output to binary output using the Gumbel-Softmax operator. A binary output of 1 indicates that the frame represents the boundary between human and object entities. The frame is the last frame of the current segment of the corresponding entity; conversely, an output of 0 indicates the last frame. The frame is not a segmentation boundary.

5. The video human-object interaction detection method based on hierarchical spatiotemporal graph networks according to claim 1, characterized in that, Step 5 includes: learning segment-level features for each entity based on the segmentation boundaries and temporal enhancement features of the temporal enhancement branches provided by the segment predictor for people and objects; using the segment update operation to sparsely update the segment-level features based on the segmentation boundary information; then using the Softmax function to predict the segment labels of the entities using the obtained updated segment-level features; calculating the probability of obtaining human sub-activities or object availability segment labels; and finally achieving the prediction.

Images