Video scene graph generation method based on multi-scale spatiotemporal attention network

Abstract

The present invention discloses a method for generating a video dynamic scene graph based on a multi-scale spatiotemporal attention network. The steps of the present invention are as follows: 1. Data set division, 2. Using a pre-trained target detection network to extract features from video frames and perform target classification, 3. Constructing language features of targets, 4. Constructing comprehensive features of character relationship pairs and storing them as sparse matrices, 5. Constructing a multi-scale spatiotemporal attention network, 6. Constructing a classification network enhanced by a pre-trained model, 7. Loss function, 8. Training model, 9. Calculating network prediction values. The present invention proposes a multi-scale spatiotemporal attention network, which innovatively introduces the "multi-scale" modeling concept based on the classic Transformer architecture to achieve accurate modeling of fine-grained semantics of video dynamics.

Description

Technical Field

[0001] The present invention proposes a video dynamic scene graph generation method (Dynamic Scene Graph Generation) based on a multi-scale spatiotemporal attention network (Multi-scale Spatial-Temporal Transformer, MSTT). Background Art

[0002] The dynamic scene graph generation (DSGG) task aims to simultaneously detect objects appearing in a video and predict the relationships between them, thereby generating a series of triples of the form <subject, predicate, object>. A dynamic scene graph adds a timeline to a static scene graph to represent the changing relationships between objects over time. Generating a dynamic scene graph is a challenging task because it requires considering both spatial and temporal factors. Traditional static image scene graph generation models focus only on the current frame and can usually only capture the static relationships between objects in the image. However, dynamic scene graph generation models require more fine-grained modeling of the semantics in dynamic videos, hoping to generate more specific relationships for the current frame based on other frames in the video. This fine-grained relationship modeling helps to more accurately understand the interactive dynamics in the video. Therefore, with the increasing demand for fine-grainedness, the research on dynamic video scene graph generation is of great significance.

[0003] In recent years, research on video scene graph generation methods can be divided into two branches: coarse-grained and fine-grained. Coarse-grained methods aim to generate scene graphs representing scenes appearing in video clips. These methods have achieved promising results in understanding video clips and predicting relationships between objects, but they fail to capture the dynamic evolution of object relationships across video frames over time. With the increasing demand for video understanding, a growing number of studies are devoted to fine-grained video scene graph generation. However, these methods focus only on the global spatial correlation of objects in the spatial dimension and the short-term temporal correlation of objects in the video in the temporal dimension. This single-scale attention network model cannot capture the local spatial correlation of the relative positions of objects or the long-term temporal correlation of the same pair of objects across the entire video, thus limiting its understanding capabilities. Therefore, designing different modeling scales in the spatial and temporal dimensions to capture more effective information will undoubtedly help deepen the understanding of video scene graphs and enhance the expressive power of the final dynamic video scene graph generation.

[0004] In practical applications, video scene graphs have a wide range of applications. First, they can provide semantic relationships and interaction patterns between objects in a video, helping machines understand video content. This is very useful for tasks such as video content analysis, content recommendation, video retrieval, and video content understanding. Second, video scene graphs can provide information about objects and actions in a video, helping users edit and cut videos. By automatically identifying and analyzing scenes and dynamic elements in a video, video cutting and synthesis can be more convenient. Furthermore, video scene graphs can help computers perform deeper visual reasoning. By understanding the relationships and dynamic changes between objects, computers can better infer future dynamic behavior, thereby providing more intelligent and accurate visual reasoning results. Finally, video scene graphs can be used for video content generation and enhancement. By modifying and adjusting the video scene graph, new video content can be generated or the visual effects of existing videos can be enhanced.

[0005] In summary, video dynamic scene graph generation is a topic worthy of in-depth research. This patent intends to explore several key points in this task, solve the difficulties and key points of the current methods, and form a complete video scene graph generation system.

[0006] Previous methods for generating video scene graphs have focused solely on the global spatial correlation of objects in the spatial dimension and the short-term temporal correlation of objects in the video in the temporal dimension. While these methods are simple to implement and understand, they still suffer from the following two problems:

[0007] 1. It fails to specifically capture local spatial relationships that are highly relevant to pairs of people. This results in the model failing to focus on the spatial proximity between objects, leading to reduced accuracy in relationship modeling when encountering cross-occlusions in dynamic scenes. Therefore, this method adds a local spatial scale to model local spatial relationships, enabling more accurate capture of relationships between people and objects while minimizing interference from irrelevant objects. This module plays a crucial role in accurately modeling relationships between objects in dynamic video scenes that involve dense interactions.

[0008] 2. It is unable to accurately predict and capture the long-term evolution trends and persistent behaviors of relationships between people. This results in the model being unable to account for the persistent behaviors and dynamic evolution between objects, which can lead to incorrect modeling when objects are occluded for long periods of time. Therefore, this method adds a long-term time scale to model long-term temporal relationships. By focusing on the persistent behaviors and dynamic evolution of objects, it can capture long-term relationships such as ongoing interactions and pursuits between objects. This module plays a vital role in modeling long-term dynamic relationships in dynamic video scenes. Summary of the Invention

[0009] To address the above two issues, this paper provides a video scene graph generation method based on a multi-scale spatiotemporal attention network. The core method is to propose a multi-scale spatiotemporal attention network (MSTT) to achieve accurate modeling of the dynamic fine-grained semantics of the video.

[0010] The present invention mainly includes two points:

[0011] 1. Building on the classic Transformer architecture, this method innovatively introduces the concept of "multi-scale" modeling, modeling separately in the spatial and temporal dimensions. In the spatial dimension, while retaining the traditional approach's focus on the global spatial correlation of objects, it also models the local spatial correlation of the relative positions of objects. In the temporal dimension, while retaining the traditional approach's focus on the short-term temporal correlation of objects in a video, it also considers the long-term temporal correlation of the same pair of objects across the entire video.

[0012] 2. The large-scale visual language pre-training model CLIP is used to enhance the model's interactive understanding of visual language. By converting relationship categories into text descriptions and using the CLIP model to generate text embeddings, it can provide better semantic representation and understanding capabilities.

[0013] The technical solution adopted by the present invention to solve the technical problem includes the following steps:

[0014] Step 1: Dataset division

[0015] Divide the dataset.

[0016] Step 2: Use the pre-trained target detection network to extract features from the video frame and perform target classification

[0017] For each video frame in the dataset, a pre-trained object detector is used to identify the object and extract features. The category of each detected object is predicted as a priori condition for the model.

[0018] Step 3: Build the target language features

[0019] The target predicted classification is converted into a word vector containing semantic information according to the pre-trained word vector model.

[0020] Step 4: Construct comprehensive features of person relationship pairs and store them as a sparse matrix

[0021] The features obtained in steps 2 and 3 are concatenated in pairs between the subject and the object, including the visual features of the subject, the visual features of the object, the linguistic features of the subject, the linguistic features of the object, and the joint features between the subject and the object;

[0022] For all frames in a video, the paired features between the subject and object are stored in a sparse matrix. The number of rows in the matrix represents the number of video frames, and the number of columns in the matrix represents the number of object categories. Each column represents the relationship between the same object and a person.

[0023] Step 5: Build a multi-scale spatiotemporal attention network

[0024] The comprehensive features from step 4 are input into the multi-scale spatiotemporal attention network, which has two modules: the spatial encoder and the temporal decoder. The spatial encoder includes local spatial encoding and global spatial encoding, while the temporal decoder includes long-term temporal decoding and short-term temporal decoding.

[0025] Step 6: Build a classification network enhanced by the pre-trained model

[0026] The output of step 5 is fed into the classification network enhanced by the pre-trained model for final relationship classification to enhance interactive understanding of visual language. The final output is the attention relationship category prediction vector, the position relationship category prediction vector, and the contact relationship category prediction vector.

[0027] Step 7: Loss Function

[0028] For object classification in step 2, the prediction vector and target vector are input into the loss function to calculate the loss value. For relation classification in step 6, the three output prediction vectors are input into the corresponding loss function along with the corresponding target vectors, and three loss values ​​are output respectively.

[0029] Step 8: Train the model

[0030] According to the loss value generated by the loss function in step 7, the model parameters of the neural network in step 6 are gradient-backed using the back-propagation algorithm and continuously optimized until the entire network model converges, that is, the loss value of the model approaches 0 and no longer has a downward trend.

[0031] Step 9: Calculate network prediction value

[0032] The prediction vectors output in step 6 are sorted, and the final classification prediction result is determined based on different evaluation criteria. The evaluation strategies are: 1) Constrained strategy, which allows only one predicate between each person pair. This strategy constrains the number of relationships between person pairs in the generated scene graph. 2) Unconstrained strategy, which allows multiple predicates between each person pair. This strategy has no limit on the number of relationships between person pairs in the generated scene graph.

[0033] Step 1 is implemented as follows:

[0034] 70% of the data in the dataset is used for training, and the remaining 30% is used for validation and testing.

[0035] In step 2, the pre-trained object detection network is used to extract features from the video frame and predict the category distribution, as follows:

[0036] For the input video V=[I1,I2,…,I T ] (where T represents the number of frames of the video), each frame I t N can be obtained from the detector t Bounding boxes and their category distribution And the visual features corresponding to each bounding box

[0037] The language characteristics of the build target described in step 3 are as follows:

[0038] The object category labels are mapped to 200-dimensional semantic embedding vectors through pre-trained GloVe-200d. The semantic vector between two objects a and b in the t-th frame is represented as

[0039] Step 4 constructs the comprehensive features of the person relationship pairs and stores them as a sparse matrix, as follows:

[0040] The representation vector between two objects a and b in the tth frame can be expressed as:

[0041]

[0042] Where <,> represents the splicing operation in the channel dimension, represents a flattening operation, and W represents element-wise addition. s ,W o and W u is a linear matrix used to compress visual features into 512 dimensions. represents the feature map of the joint box calculated by RoIAlign, f box is a transformation function used to transform the bounding boxes corresponding to the subject and object into Features of the same shape.

[0043] The features expressed in Equation 1 are stored in a sparse matrix. Specifically, for C object categories and a video V with T frames, the input matrix can be simply expressed as Where D represents the dimension of the input representation. The rows of the sparse matrix represent video frames, while the columns represent pairs of people and objects. Since some objects in the object labels in the video have never appeared, in order to reduce redundancy, the columns of objects that have never appeared are deleted from the sparse matrix, that is, the final input matrix is ​​represented as Where C' represents the number of object categories that actually appear in the current video V.

[0044] The construction of the multi-scale spatiotemporal attention network described in step 5 is as follows:

[0045] 5-1. Constructing a multi-scale spatial encoder:

[0046] This step requires constructing a global spatial encoder and a local spatial encoder.

[0047] 5-1-1. Build a global spatial encoder:

[0048] For the input matrix of video V Take the t-th row (representing the character relationship representation sequence of all characters appearing in the t-th frame) as the input of the encoder as an example, that is, the input sequence is In the global scale spatial encoder, a single-head dot product self-attention mechanism is used. In this operation, Q, K, and V share the same input, and the output after passing through the n-layer encoder is expressed as:

[0049]

[0050] The encoder consists of n stacked MultiHeadAtt global (·), the input of the nth layer is the output of the (n-1)th layer.

[0051] 5-1-2. Constructing a local spatial encoder:

[0052] First, the center point of each object is calculated based on the bounding box of the object in each frame. Then, the nearest object to each object is obtained based on the distance between the center points of the objects, and a mask matrix M∈R is constructed. C'×C' , where the value is 0 or 1. When the value in the matrix M is 1, it means that the object is the closest visible object to the current object; when the value is 0, it means that the object is an invisible object to the current object. The specific method is as follows: After matrix multiplication of Q and K, the affinity matrix A∈R is obtained. C'×C' , calculate the dot product with the mask matrix M, and then perform matrix multiplication with V. The output after the n-layer encoder is expressed as:

[0053]

[0054] The spatial contexts obtained from the above two scales are spliced ​​in the channel dimension as the output of the final multi-scale spatial encoder

[0055] 5-2. Constructing a multi-scale temporal decoder:

[0056] This step requires constructing a long-term temporal decoder and a short-term temporal decoder.

[0057] 5-2-1. Constructing a long-term time decoder:

[0058] For the output matrix of the multi-scale spatial encoder The cth column (representing the relationship between a person and an object with category label c) is selected as the input of the decoder, that is, the input sequence is Q, K, V share the same input, and the output after n layers of decoders is expressed as:

[0059]

[0060] 5-2-2. Construct a short-term time decoder:

[0061] Similar to 5-1-2, a mask matrix M∈R with all values ​​0 or 1 is also set in the decoder T×T , which is used to limit the range of frames that can be focused on at each time instant t. Here, 1 indicates a frame that can be seen at the current moment, and 0 indicates a frame that cannot be seen at the current moment. The input of this module is still the output matrix of the multi-scale spatial encoder, and an additional mask matrix is ​​input. The specific method is similar to that of the multi-scale encoder. The output representation after passing through the n-layer decoder is as follows, which contains information about the evolution of relationships on a short-term time scale:

[0062]

[0063] The output of the multi-scale decoder includes information about captured long-term and short-term temporal dependencies, which are concatenated together in the channel dimension to form the final decoder output.

[0064] The construction of the classification network enhanced by the pre-trained model described in step 6 is as follows:

[0065] 6-1. Design prompts:

[0066] We designed a prompt structure of the form "a photo of a person[relationship]a / an[object]", where the predicate label is filled in the [relationship] position and the object label is filled in the [object] position. For R predicate labels and C object labels, we generated (R × C) sentence text descriptions.

[0067] 6-2. Generate text embeddings for text description sentences:

[0068] The pre-trained CLIP text encoder is used to generate text embeddings for text descriptions offline and use these embeddings to initialize the weights of the learnable classifiers. In this dataset-specific task, different types of relationships need to be predicted for each pair of people, typically including attention relationships, position relationships, and contact relationships. To achieve this goal, the text embeddings are classified into three categories and used to initialize the weights of the three classifiers.

[0069] 6-3. Fine-tuning the classifier:

[0070] To adapt to the specific dataset, the classifier is fine-tuned and the learning rate is set to 2e -5 The fine-tuning process is to optimize the weights of the classifier by performing supervised learning on a specific task dataset so that it can better adapt to the specific relationship classification task.

[0071] The loss function described in step 7 is as follows:

[0072] 7-1. Calculate the predicted distribution of object categories i and the true label The gap between them is calculated using cross entropy (softmax cross entropy), and the specific formula is as follows:

[0073]

[0074] 7-2. Calculate the predicted distribution of attention relationship categories r ai and the true label The gap between them is calculated using cross entropy (softmax cross entropy), and the specific formula is as follows:

[0075]

[0076] 7-3. Calculate the predicted distribution r of position relationship categories si and the true label The gap between them is calculated using the sigmoid binary cross entropy formula:

[0077]

[0078] 7-4. Calculate the predicted distribution r of contact relationship categories ci and the true label The gap between them is calculated using the sigmoid binary cross entropy formula:

[0079]

[0080] 7-5. The total loss of the model is as follows:

[0081]

[0082] The training model described in step 8 is as follows:

[0083] According to the loss value generated by the loss function in step 7, the back propagation algorithm is used to perform gradient backpropagation on the model parameters of the neural network in steps 5 and 6, and continuously optimize until the entire network model converges.

[0084] The model prediction value calculation described in step 9 is as follows:

[0085] The converged model is used to generate and predict the video scene graph. The output results from step 5 are fed into the three classifiers initialized in step 6. The final classification prediction result is determined based on different evaluation criteria. The evaluation strategies are: 1) Constrained strategy, which allows only one predicate between each person pair. This strategy constrains the number of relationships between person pairs in the generated scene graph. 2) Unconstrained strategy, which allows multiple predicates between each person pair. This strategy has no limit on the number of relationships between person pairs in the generated scene graph.

[0086] The essential features of this invention lie in: In the spatial dimension, building on traditional methods, this invention further focuses on the local spatial relationship between two objects with the closest bounding box centers based on object detection. By selecting pairs of people with the closest spatial positions for relationship modeling, their spatial proximity is highlighted. This precise modeling helps better understand the dynamic interactions between people and objects compared to traditional methods, providing more accurate semantic analysis results. In the temporal dimension, this invention, building on traditional methods, also focuses on the long-term temporal relationship between the same pair of objects across all frames. By focusing on the dynamic evolution of the same pair of objects across all frames, long-term temporal modeling can more comprehensively model the long-term relationships between objects, helping to generate more accurate and coherent scene graphs. In the case of occlusion and overlap, objects may gradually become visible across multiple frames or be unblocked through interactive actions. Long-term temporal modeling can capture these persistent behavioral patterns. In summary, the multi-scale modeling concept introduced in this invention in the spatial and temporal dimensions is of great significance in the task of fine-grained semantic modeling of video dynamics. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] Figure 1 : The overall architecture of multi-scale spatiotemporal attention network.

[0088] Figure 2 : Attention design under multi-scale spatial encoder and temporal decoder.

[0089] Figure 3 : CLIP model enhanced classification network. DETAILED DESCRIPTION

[0090] The detailed parameters of the present invention are further described below.

[0091] like Figure 1 , 2, 3, the present invention provides a video scene graph generation method based on a multi-scale spatiotemporal attention network.

[0092] Step 1 is implemented as follows:

[0093] 70% of the data in the dataset is used for training, and the remaining 30% is used for validation and testing.

[0094] In step 2, the pre-trained object detection network is used to extract features from the video frame and predict the category distribution, as follows:

[0095] For the input video V=[I1,I2,…,I T ] (where T represents the number of frames of the video, which is based on the actual data of each video in the dataset), and each frame I t N can be obtained from the detector t Bounding boxes and their category distribution And the visual features corresponding to each bounding box

[0096] The semantic features of the build target described in step 3 are as follows:

[0097] The object category labels are mapped to 200-dimensional semantic embedding vectors through pre-trained GloVe-200d. The semantic vector between two objects a and b in the t-th frame is represented as

[0098] Step 4 constructs the comprehensive features of the person relationship pairs and stores them as a sparse matrix, as follows:

[0099] Take the calculation process of the representation vector between two objects a and b in the tth frame as an example. First, use the linear matrix W s ,W o The visual features of objects a and b are compressed to 512 dimensions. Secondly, the feature map of the joint box of objects a and b is calculated by RoIAlign And transform the bounding boxes corresponding to the subject and object into Features with the same shape, the conversion function is f box , add the joint box feature map to the bounding box transformed features and pass the linear matrix Wu Compress to 512 dimensions. Finally, the semantic features obtained in step 3 are By concatenating the above features, we can obtain the representation vectors of objects a and b in the tth frame, with a total dimension of 512+512+512+200+200=1936.

[0100] Next, the above features are stored in a sparse matrix. Specifically, for C object categories and a video V with T frames, the input matrix can be simply expressed as Where D represents the dimension of the input representation (i.e., 1936). The rows of the sparse matrix represent video frames, while the columns represent pairs of people and objects. Since some objects in the object labels in the video have never appeared, in order to reduce redundancy, the columns of objects that have never appeared are deleted from the sparse matrix, that is, the final input matrix is ​​represented as Where C' represents the number of object categories that actually appear in the current video V.

[0101] The construction of the multi-scale spatiotemporal attention network described in step 5 is as follows:

[0102] 5-1. Constructing a multi-scale spatial encoder:

[0103] This step requires constructing a global spatial encoder and a local spatial encoder.

[0104] 5-1-1. Build a global spatial encoder:

[0105] For the input matrix of video V Take the t-th row (representing the character relationship representation sequence of all characters appearing in the t-th frame) as the input of the encoder as an example, that is, the input sequence is Where D = 1936. In the global scale spatial encoder, a single-head dot product self-attention mechanism is used. In this operation, Q, K, and V share the same input, and the output of the global spatial encoder is obtained after passing through n layers of encoders. The encoder consists of n stacked MultiHeadAtt global (·), the input of the nth layer is the output of the (n-1th) layer. In this method, n is set to 1, that is, only one layer of global spatial encoder is stacked.

[0106] 5-1-2. Constructing a local spatial encoder:

[0107] First, the center point of each object is calculated based on the bounding box of the object in each frame. Then, the nearest object to each object is obtained based on the distance between the center points of the objects, and a mask matrix M∈R is constructed. C'×C', where the value is 0 or 1. When the value in the matrix M is 1, it means that the object is the closest visible object to the current object; when the value is 0, it means that the object is an invisible object to the current object. The specific method is as follows: After matrix multiplication of Q and K, the affinity matrix A∈R is obtained. C'×C' , calculate the dot product with the mask matrix M, and then perform matrix multiplication with V. Finally, after passing through n layers of encoders, the output of the local spatial encoder is obtained. In this method, n is set to 1, which means that only one layer of local spatial encoder is stacked.

[0108] The spatial contexts obtained from the above two scales are spliced ​​in the channel dimension as the output of the final multi-scale spatial encoder

[0109] 5-2. Constructing a multi-scale temporal decoder:

[0110] This step requires constructing a long-term temporal decoder and a short-term temporal decoder.

[0111] 5-2-1. Constructing a long-term time decoder:

[0112] For the output matrix of the multi-scale spatial encoder The cth column (representing the relationship between a person and an object with category label c) is selected as the input of the decoder, that is, the input sequence is Where D = 1936. Q, K, and V share the same input, and after passing through n layers of decoders, the output of the long-term temporal decoder is obtained. In this method, n is set to 1, meaning only one layer of long-term temporal decoders is stacked.

[0113] 5-2-2. Construct a short-term time decoder:

[0114] Similar to 5-1-2, a mask matrix M∈R with all values ​​0 or 1 is also set in the decoder T×T , used to limit the range of frames that can be focused on at each moment t [I t-p ,I t+q ], where p represents the p frames before the current frame, and q represents the q frames after the current frame. The present invention sets p to 1 and q to 0. In the mask matrix, 1 represents the frame that can be seen at the current moment, and 0 represents the frame that cannot be seen at the current moment. The input of this module is still the output matrix of the multi-scale spatial encoder, and the mask matrix is ​​additionally input. The specific approach is similar to that of the multi-scale encoder, and the output of the short-term time decoder is obtained after passing through n layers of decoders. This method sets n to 1, that is, only one layer of short-term time decoder is stacked.

[0115] The output of the multi-scale decoder includes information about captured long-term and short-term temporal dependencies, which are concatenated together in the channel dimension to form the final decoder output.

[0116] The construction of the classification network enhanced by the pre-trained model described in step 6 is as follows:

[0117] 6-1. Design prompts:

[0118] We designed a prompt structure of the form "a photo of a person[relationship]a / an[object]", where the predicate label is filled in the [relationship] position and the object label is filled in the [object] position. For R predicate labels and C object labels, we generated (R × C) sentence text descriptions.

[0119] 6-2. Generate text embeddings for text description sentences:

[0120] The pre-trained CLIP text encoder is used to generate text embeddings (512-d) for the text description sentences offline, and these embeddings are used to initialize the weights of the learnable classifiers. In the task of a specific dataset, different types of relationships need to be predicted for each pair of person relationships, typically including attention relationships, position relationships, and contact relationships. To achieve this goal, the text embeddings are divided into three categories and used to initialize the weights of the three classifiers.

[0121] 6-3. Fine-tuning the classifier:

[0122] To adapt to the specific dataset, the classifier is fine-tuned and the learning rate is set to 2e -5 The fine-tuning process is to optimize the weights of the classifier by performing supervised learning on a specific task dataset so that it can better adapt to the specific relationship classification task.

[0123] The loss function described in step 7 is as follows:

[0124] 7-1. Calculate the predicted distribution of object categories i and the true label The gap between them is calculated using cross entropy (softmax cross entropy).

[0125] 7-2. Calculate the predicted distribution of attention relationship categories r ai and the true label The gap between them, here we use cross entropy (softmax cross entropy).

[0126] 7-3. Calculate the predicted distribution r of position relationship categories si and the true label The gap between them is calculated using sigmoid binary cross entropy.

[0127] 7-4. Calculate the predicted distribution r of contact relationship categories ci and the true label The gap between them is calculated using sigmoid binary cross entropy.

[0128] 7-5. The total loss of the model is the sum of the above losses.

[0129] The training model described in step 8 is as follows:

[0130] According to the loss value generated by the loss function in step 7, the model parameters of the neural network in steps 5 and 6 are gradient-backed using the backpropagation algorithm and continuously optimized until the entire network model converges, that is, the loss value of the model approaches 0 and no longer has a downward trend.

[0131] The model prediction value calculation described in step 9 is as follows:

[0132] The converged model is used to generate and predict the video scene graph. The output results from step 5 are fed into the three classifiers initialized in step 6. The final classification prediction result is determined based on different evaluation criteria. The evaluation strategies are: 1) Constrained strategy, which allows only one predicate between each person pair. This strategy constrains the number of relationships between person pairs in the generated scene graph. 2) Unconstrained strategy, which allows multiple predicates between each person pair. This strategy has no limit on the number of relationships between person pairs in the generated scene graph.

Claims

1. Video scene graph generation method based on multi-scale spatiotemporal attention network, characterized by The steps include: Step 1: Divide the data set; Step 2: Use the pre-trained object detection network to extract features from the video frame and perform object classification; predict the category of each detected object as a priori condition for the model; Step 3: Build the target language features Convert the classification results in step 2 into word vectors containing semantic information according to the pre-trained word vector model; Step 4: Construct comprehensive features of person relationship pairs and store them as a sparse matrix The features obtained in steps 2 and 3 are concatenated in pairs between the subject and the object, including the visual features of the subject, the visual features of the object, the linguistic features of the subject, the linguistic features of the object, and the joint features between the subject and the object; For all frames in a video, the paired features between the subject and object are stored in a sparse matrix. The number of rows in the matrix represents the number of video frames, and the number of columns in the matrix represents the number of object categories. Each column represents the relationship between the same object and a person. Step 5: Build a multi-scale spatiotemporal attention network The comprehensive features of step 4 are input into the multi-scale spatiotemporal attention network, which includes: a spatial encoder and a temporal decoder; the spatial encoder includes local spatial encoding and global spatial encoding, and the temporal decoder includes long-term temporal decoding and short-term temporal decoding; Step 6: Build a classification network enhanced by the pre-trained model The output of step 5 is input into the classification network enhanced by the pre-trained model for final relationship classification to enhance the interactive understanding of visual language; the final output is the attention relationship category prediction vector, the position relationship category prediction vector, and the contact relationship category prediction vector; Step 7: Loss Function For object classification in step 2, the prediction vector and target vector are input into the loss function to calculate the loss value. For relation classification in step 6, the three types of prediction vectors output are input into the corresponding loss function together with the corresponding target vectors, and three loss values ​​are output respectively. Step 8: Train the model Based on the loss value generated by the loss function in step 7, the model parameters of the neural network in step 6 are gradient-backed using the backpropagation algorithm and continuously optimized until the entire network model converges, that is, the training loss of the model drops to a certain range and no longer decreases; Step 9: Calculate network prediction value Sort the prediction vectors output in step 6 and decide the final classification prediction results based on different evaluation criteria.

2. The method for generating a video scene graph based on a multi-scale spatiotemporal attention network according to claim 1, characterized in that: The pre-trained object detection network is used to extract features from the video frame as described in step 2, as follows: For the input video V=[I1,I2,…,I T ], where T represents the number of frames of the video, Each frame I t N can be obtained from the detector t Bounding boxes and their category distribution And the visual features corresponding to each bounding box Where b represents the object bounding box, d represents the object category distribution, and v represents the object visual feature.

3. The video scene graph generation method based on the multi-scale spatiotemporal attention network according to claim 2 is characterized in that The language characteristics of the build target described in step 3 are as follows: The object category label is mapped to a 200-dimensional semantic embedding vector through the pre-trained GloVe-200d. The semantic vector between two objects a and b in the t-th frame is represented as 4. A video scene graph generation method based on a multi-scale spatiotemporal attention network according to claim 3, characterized in that Step 4 constructs the comprehensive features of the person relationship pairs and stores them as a sparse matrix, as follows: The representation vector between two objects a and b in the tth frame can be expressed as: Where <,> represents the splicing operation in the channel dimension, represents the flattening operation, and ⊕ represents element-wise addition; W s ,W o and W u It is a linear matrix used to compress visual features into 512 dimensions; represents the feature map of the joint box calculated by RoIAlign, f box is a transformation function used to transform the bounding boxes corresponding to the subject and object into Features of the same shape; The features expressed in Equation 1 are stored in a sparse matrix. For C object categories and a video V with T frames, the input matrix is ​​expressed as Where D represents the dimension of the input representation; the rows of the sparse matrix represent video frames, and the columns represent pairs of people and objects; the input matrix is ​​represented as Where C' represents the number of object categories that actually appear in the current video V.

5. A video scene graph generation method based on a multi-scale spatiotemporal attention network according to claim 4, characterized in that The construction of the multi-scale spatiotemporal attention network described in step 5 includes the following steps: 5-1. Constructing a multi-scale spatial encoder: 5-1-1. Build a global spatial encoder: For the input matrix of video V The input sequence is Where t represents the tth frame of the video. In the spatial encoder at the global scale, a single-head dot product self-attention mechanism is adopted. In this operation, Q, K, and V share the same input, and the output after passing through the n-layer encoder is expressed as: The encoder consists of n stacked MultiHeadAtt global (·), the input of the nth layer is the output of the (n-1)th layer; 5-1-2. Constructing a local spatial encoder: Calculate the center point of the object based on the bounding box of the object in each frame; According to the distance between the center points of the objects, the nearest object of each object is obtained, and a mask matrix M∈R is constructed C '×C' , where the value is 0 or 1; when the value in the matrix M is 1, it means that the object is the closest visible object to the current object; when the value is 0, it means that the object is an invisible object to the current object; When Q and K are multiplied by the matrix, the affinity matrix A∈R is obtained. C'×C' , calculate the dot product with the mask matrix M, and then perform matrix multiplication with V on this basis; the output after the n-layer encoder is expressed as: The spatial contexts obtained from the above two scales are spliced ​​in the channel dimension as the output of the final multi-scale spatial encoder 5-2. Constructing a multi-scale temporal decoder: 5-2-1. Constructing a long-term time decoder: For the output matrix of the multi-scale spatial encoder To select the cth column as the input of the decoder, the input sequence is Q, K, V share the same input, and the output after n layers of decoders is expressed as: 5-2-2. Construct a short-term time decoder: In the decoder, a mask matrix M∈R with all values ​​0 or 1 is also set. T×T , which is used to limit the range of frames that can be focused on at each time t; where 1 indicates a frame that can be seen at the current time, and 0 indicates a frame that cannot be seen at the current time; the input of this module is the output matrix of the multi-scale spatial encoder, and an additional mask matrix is ​​input; the output representation after the n-layer decoder is as follows, which contains the relationship evolution information on the short-term time scale: The output of the multi-scale decoder includes information about captured long-term and short-term temporal dependencies; this information is concatenated together in the channel dimension to form the final decoder output.

6. A video scene graph generation method based on a multi-scale spatiotemporal attention network according to claim 5, characterized in that The construction of the classification network enhanced by the pre-trained model described in step 6 includes the following steps: 6-1. Design Tips: A text prompt structure with [relationship] and [object] is designed, where the predicate label is filled in the [relationship] position and the object label is filled in the [object] position; for R predicate labels and C object labels, (R×C) sentence text descriptions are generated; 6-2. Generate text embeddings for text description sentences: Use the pre-trained CLIP text encoder to generate text embeddings of text description sentences offline, and use these embeddings as weights to initialize the learnable classifier; 6-3. Fine-tuning the classifier: Fine-tune the classifier and set the learning rate to 2e -5 The process of fine-tuning is to optimize the weights of the classifier by performing supervised learning on a specific task dataset.

7. A video scene graph generation method based on a multi-scale spatiotemporal attention network according to claim 6, characterized in that The loss function described in step 7 includes the following steps: 7-1. Calculate the predicted distribution of object categories O i and the true label The gap between them is calculated using cross entropy softmax crossentropy, and the formula is as follows: 7-2. Calculate the predicted distribution of attention relationship categories r ai and the true label The gap between them is calculated using cross entropy softmaxcross entropy, and the formula is as follows: 7-3. Calculate the predicted distribution r of position relationship categories si and the true label The gap between them is calculated using the binary cross entropy sigmoid binary cross entropy, with the following formula: 7-4. Calculate the predicted distribution r of contact relationship categories ci and the true label The gap between them is calculated using the binary cross entropy sigmoid binary cross entropy, with the following formula: 7-5. The total loss of the model is as follows: Loss=Loss obj +Loss rel_a +Loss rel_s +Loss rel_c (Formula 12).

8. The method for generating a video scene graph based on a multi-scale spatiotemporal attention network according to claim 1, characterized in that: Judging criteria include: With the constraint strategy, only one predicate is allowed between each character pair; this strategy constrains the number of relationships between character pairs in the generated scene graph; The unconstrained strategy allows multiple predicates between each character pair; under this strategy, there is no limit on the number of relationships between character pairs in the scene graph generated.

Images