An end-to-end dynamic scene graph generation method and system

Abstract

The application relates to an end-to-end dynamic scene graph generation method and system, which comprises the following steps: extracting instance feature maps of each instance in a target video; extracting fine-grained features of the instance feature maps of each instance to obtain fine-grained feature maps of each instance; predicting instance description texts corresponding to the fine-grained feature maps of each instance to obtain instance attribute information of each instance; querying pair instance representations between each instance from a pair instance query dataset to obtain boundary box coordinate information and category information of the pair instance; predicting dynamic visual relationships of the pair instance to obtain dynamic visual relationship information of the pair instance; and generating a dynamic scene graph according to the instance attribute information, the boundary box coordinate information, the category information and the dynamic visual relationship information of the pair instance. The application improves the prediction accuracy, realizes end-to-end training, provides a new paradigm for dynamic scene graph generation, and simplifies the training steps of the model.

Description

Technical Field

[0001] This invention relates to the field of computer vision dynamic scene graph generation technology, specifically to an end-to-end dynamic scene graph generation method and system. Background Technology

[0002] Scene Graph Generation (SGG) is an important task in computer vision, aiming to automatically parse structured scene representations from images. SGG aims to generate complete structured representations of image or video content, bridging the gap between computer vision and language. Scene graphs contain rich visual and semantic information and can be used for various vision tasks, such as image retrieval, image captioning, and visual question answering. A scene graph consists of a series of "subject-predicate-object" triples. While significant progress has been made in image scene graph generation, video scene graph generation is still in its early stages.

[0003] Johnson (Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li,

[0004] David Shamma, Michael Bernstein, and Li Fei-Fei. Im-

[0005] age retrieval using scene graphs. In Proceedings of the

[0006] IEEE / CVF Conference on Computer Vision and Pattern

[0007] Recognition (CVPR), pages 3668, et al. first proposed using scene graphs to describe the detailed semantic information of a scene. With the rapid development of convolutional neural networks (CNNs), researchers have proposed various deep learning-based scene graph generation methods. Image scene graph methods typically follow a two-stage paradigm and focus on exploring spatial contextual features. Lin (LinLin, Chong Shi, Yibing Zhan, Zuopeng Yang, Yaqi Wu, ...) et al. first proposed using scene graphs to describe the detailed semantic information of a scene.

[0008] and Dacheng Tao. Td2-net: Toward denoising and debi-

[0009] asing for video scene graph generation. In Proceedings of

[0010] the AAAI Conference on Artificial Intelligence (AAAI), pages

[0011] 3495AAAI Confere et al. proposed a direction-aware message passing module that focuses on the predicate construction direction of the scene graph and generates a predicate attention map to more accurately identify predicates. (Chaofan Zheng, Xinyu Lyu, Yuyu Guo, Pengpeng Zeng,

[0012] Jingkuan Song, and Lianli Gao. Learning to generate scene

[0013] graph from head to tail. In 2022 IEEE International Con-

[0014] References on Multimedia and Expo (ICME), pages 1, 1999, and others proposed a course reweighting mechanism that passes the robust features of the head predicate to the tail predicate to alleviate the long-tail distribution phenomenon in the scene graph.

[0015] Dynamic scene graphs, compared to image-based scene graphs, contain richer spatiotemporal contextual information and more complex visual relationships. (Cong, Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo)

[0016] Rosenhahn, and Michael Ying Yang. Spatial-temporal trans-

[0017] former for dynamic scene graph generation. In Proceedings

[0018] of the IEEE / CVF International Conference on Computer Vi-

[0019] The paper, published in ICCV (pages 16372), proposes a two-stage dynamic scene graph baseline based on a transformer. It learns the temporal dependencies between subject-object pairs through an encoder-decoder structure. The effectiveness of temporal dependencies in dynamic scene graph generation is demonstrated. (Tao Pu, Tianshui Chen, Hefeng Wu, Yongyi Lu, and Liang...)

[0020] Lin. Spatial-temporal knowledge-embedded transformer for

[0021] video scene graph generation. IEEE Transactions on Image

[0022] Processing (TIP), 33:556tion. IEEE T) et al. further introduced temporal prior information to guide the aggregation of spatial context information and temporal relevance. These two-stage methods emphasize the role of temporal relevance in dynamic scene graph generation and focus on designing modules to learn the spatiotemporal context in videos. However, these two-stage methods hinder end-to-end learning, and erroneous detection results can affect the training of prediction models, resulting in biased scene graphs.

[0023] The long-tailed distribution of visual relationships is a significant challenge in dynamic scene graph generation. (Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-)

[0024] Fei. Visual relationship detection with language priors. In

[0025] European Conference on Computer Vision (ECCV), pages

[0026] 852opean Confe et al. used linguistic priors in semantic word embeddings to identify tail predicates from a global perspective. Nag (Nayak Nag, Kyle Min, Subarna Tripathi, and Amit K Roy-

[0027] Chowdhury. Unbiased scene graph generation in videos. In

[0028] Proceedings of the IEEE / CVF Conference on Computer Vi-

[0029] (See *Symne and Pattern Recognition (CVPR), pages 22803–22813, 2023*). This paper employs a memory-guided training strategy to transfer knowledge from head predicates rich in data samples to tail predicates to address long-tail problems. However, these methods are limited by the contextual information of local region features and rely only on a limited number of tail predicate samples, resulting in suboptimal performance. Summary of the Invention

[0030] To address the problem that existing technologies for generating dynamic scene graphs are limited by the contextual information of local region features and rely only on a limited number of tail predicate samples, resulting in suboptimal performance, this invention provides an end-to-end dynamic scene graph generation method and system.

[0031] The technical solution of the present invention to solve the above-mentioned technical problems is as follows:

[0032] An end-to-end dynamic scene graph generation method includes the following steps:

[0033] A deep convolutional neural network model is used to extract instance feature maps of each instance in the target video;

[0034] A visual encoder is used to extract fine-grained features from the instance feature maps of each instance to obtain fine-grained feature maps of each instance.

[0035] A visual language pre-trained model is used to predict the instance description text corresponding to the fine-grained feature map of each instance, thereby obtaining the instance attribute information of each instance.

[0036] Initialize a set of paired instance query sets, use a multi-head cross-attention mechanism to aggregate the spatial features of the image to update the query, obtain paired instance representations, and regress to obtain the bounding box coordinates and category information of paired instances;

[0037] Based on the fine-grained feature maps of paired instances, predict the dynamic visual relationship of paired instances to obtain dynamic visual relationship information of paired instances.

[0038] A dynamic scene graph is generated based on the instance attribute information, bounding box coordinate information, category information, and dynamic visual relationship information of paired instances.

[0039] The beneficial effects of this invention are as follows: It predicts feature vectors using a deep convolutional neural network model, then predicts instance subjects using a visual language pre-trained model, then predicts paired instance subjects separately, and finally generates a dynamic scene graph based on the positional information and dynamic visual relationships between paired instance subjects. This invention improves prediction accuracy while enabling end-to-end training, providing a new paradigm for dynamic scene graph generation and simplifying the model training steps.

[0040] Based on the above technical solution, the present invention can be further improved as follows.

[0041] Furthermore, a deep convolutional neural network model is used to extract instance feature maps of each instance in the target video, including the following steps:

[0042] Construct the deep convolutional neural network model;

[0043] Collect dynamic scene graphs to generate a dataset;

[0044] The data in the dynamic scene graph generation dataset is preprocessed to obtain the preprocessed dynamic scene graph generation dataset.

[0045] The deep convolutional neural network model is trained using the dataset generated from the preprocessed dynamic scene graph to obtain a feature extraction backbone network model.

[0046] The target video is segmented into a sequence of multiple video frames to obtain a video frame sequence dataset;

[0047] The video frame sequence dataset is input into the feature extraction backbone network model to obtain the instance feature map for each instance.

[0048] Furthermore, the dynamic scene graph generation dataset includes multiple video data sets; the preprocessed dynamic scene graph generation dataset includes multiple video frame datasets, and each of the multiple video frame datasets corresponds one-to-one with the multiple video data sets.

[0049] The data in the dynamic scene graph generation dataset is preprocessed to obtain a preprocessed dynamic scene graph generation dataset, specifically including the following steps:

[0050] Each video data in the dynamic scene graph generation dataset is divided into multiple video frame time-series data;

[0051] When the number of video frame time-series data corresponding to the segmented video data is less than a preset video frame threshold, the video frame time-series data is supplemented by the last video frame time-series data in the multiple video frame time-series data, so that the number of video frame time-series data in the video frame dataset is equal to the preset video frame threshold, and the video frame dataset corresponding to the video data is obtained.

[0052] When the number of video frame time-series data corresponding to the segmented video data is equal to a preset video frame threshold, the video frame dataset corresponding to the video data includes all the video frame time-series data segmented from the video data.

[0053] When the number of video frame time-series data corresponding to the segmented video data is greater than the preset video frame threshold, the video data is sequentially divided into multiple time-series video segments, and each time-series video segment is divided into video frame time-series data with a number equal to the preset video frame threshold, to obtain the video frame dataset corresponding to the video data.

[0054] The video frame data in all the video frame datasets are normalized to obtain the preprocessed dynamic scene graph generation dataset.

[0055] Furthermore, the visual encoder is a DETR visual encoder, which includes a convolutional layer and a transformer encoder;

[0056] A visual encoder is used to extract fine-grained features from the instance feature maps of each instance, resulting in fine-grained feature maps for each instance. This process includes the following steps:

[0057] The convolutional layer is used to reduce the number of image channels of the instance feature map of each instance to obtain the low-channel instance feature map of each instance;

[0058] The transformer encoder is used to extract features from the low-channel instance feature maps of each instance to obtain the fine-grained feature maps of each instance.

[0059] Furthermore, the transformer encoder includes a multi-head self-attention layer, and the self-attention calculation formula for the multi-head self-attention layer in the encoder layer is as follows:

[0060] ;

[0061] in, This represents the output value of the multi-head self-attention layer. This represents the feature map of the low-channel instance. Indicates position code, This represents the number of image channels in the low-channel instance feature map. This is represented as a transpose operation. Indicates to Perform the transpose operation. This represents the feature map of the instance.

[0062] Furthermore, a visual language pre-trained model is used to predict the instance description text corresponding to the fine-grained feature maps of each instance, thereby obtaining the instance attribute information of each instance. The specific steps are as follows:

[0063] Construct the CLIP model;

[0064] The data in the dynamic scene graph dataset is used to generate a visual triplet dataset;

[0065] Text prompts are generated using the aforementioned visual triplet dataset;

[0066] The text prompt information is input into the text encoder of the CLIP model to generate text embeddings, thereby obtaining the visual language pre-trained model;

[0067] The fine-grained feature maps of each instance are input into the visual language pre-trained model to obtain the instance attribute information of each instance.

[0068] Furthermore, a set of paired instance query sets is initialized, and the spatial features of the image are aggregated using a multi-head cross-attention mechanism to update the queries, obtaining paired instance representations. The bounding box coordinates and category information of the paired instances are then obtained through regression, including the following steps:

[0069] Construct the pairwise instance query dataset;

[0070] An instance parsing decoder is constructed based on a multi-head cross-attention mechanism;

[0071] Based on the fine-grained feature maps of each instance, the pairwise representations between each instance are retrieved from the pairwise instance query dataset to obtain the pairwise subject representations;

[0072] Based on the paired subject representation, the instance parser decoder is used to predict the bounding box coordinates and category information of the paired instances in the paired subject representation.

[0073] Furthermore, predicting the dynamic visual relationship between paired instances based on the fine-grained feature maps of the paired instances yields dynamic visual relationship information for the paired instances, including the following steps:

[0074] Construct a grouped predicate decoder;

[0075] The grouped predicate decoder is used to extract the temporal dependencies of the fine-grained feature maps of paired instances through a temporal attention mechanism to obtain the spatiotemporal context features of paired instances;

[0076] The dynamic visual relationship information of the paired instances is obtained by using the grouped predicate decoder to predict the dynamic visual relationship of the paired instances based on the spatiotemporal context features of the paired instances.

[0077] To address the aforementioned technical problems, this invention provides an end-to-end dynamic scene graph generation system, the details of which are as follows:

[0078] An end-to-end dynamic scene graph generation system, comprising:

[0079] Feature extraction module: used to extract instance feature maps of each instance in the target video using a deep convolutional neural network model; and to extract fine-grained features of the instance feature maps of each instance using a visual encoder to obtain fine-grained feature maps of each instance.

[0080] Attribute recognition module: used to predict the instance description text corresponding to the fine-grained feature map of each instance using a visual language pre-trained model, and obtain the instance attribute information of each instance;

[0081] The pairwise instance recognition module is used to initialize a set of pairwise instance query sets, use a multi-head cross-attention mechanism to aggregate the spatial features of the image to update the query, obtain the pairwise instance representation, and regress to obtain the bounding box coordinates and category information of the pairwise instances.

[0082] Visual Relationship Recognition Module: Used to predict the dynamic visual relationship of paired instances based on the fine-grained feature map of paired instances, and obtain the dynamic visual relationship information of paired instances;

[0083] Scene graph construction module: used to generate dynamic scene graphs based on instance attribute information, bounding box coordinate information, category information, and dynamic visual relationship information of paired instances.

[0084] To address the aforementioned technical problems, the present invention provides a computing device, the specific technical solution of which is as follows:

[0085] A computing device includes a memory and one or more processors, wherein executable code is stored in the memory, and when the one or more processors execute the executable code, the steps of the above-described end-to-end dynamic scene graph generation method are implemented. Attached Figure Description

[0086] Figure 1 This is a flowchart of an end-to-end dynamic scene graph generation method according to an embodiment of the present invention;

[0087] Figure 2 This is a schematic diagram of the deep convolutional neural network model structure in an embodiment of the present invention;

[0088] Figure 3 This is a dynamic visual relationship structure diagram in an embodiment of the present invention;

[0089] Figure 4 This is a schematic diagram of the structure of the dynamic scene diagram in an embodiment of the present invention;

[0090] Figure 5 This is a flowchart of an end-to-end dynamic scene graph generation system according to an embodiment of the present invention. Detailed Implementation

[0091] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0092] like Figure 1 As shown, this embodiment provides an end-to-end dynamic scene graph generation method, including the following steps:

[0093] S1. Use a deep convolutional neural network model to extract instance feature maps of each instance in the target video;

[0094] like Figure 2 As shown, a deep convolutional neural network model is used to extract instance feature maps of each instance in the target video, including the following steps:

[0095] S101. Construct a deep convolutional neural network model;

[0096] S102. Collect dynamic scene graphs and generate datasets;

[0097] S103. Preprocess the data of the dynamic scene graph generation dataset to obtain the preprocessed dynamic scene graph generation dataset.

[0098] S104. Use the preprocessed dynamic scene graph to generate a dataset to train a deep convolutional neural network model and obtain a feature extraction backbone network model.

[0099] S105. Divide the target video into sequence data of multiple video frames to obtain a video frame sequence dataset;

[0100] S106. Input the video frame sequence dataset into the feature extraction backbone network model to obtain the instance feature map of each instance.

[0101] The dynamic scene graph generation dataset includes multiple video data; the preprocessed dynamic scene graph generation dataset includes multiple video frame datasets, and each video frame dataset corresponds one-to-one with the multiple video data.

[0102] The data in the dynamic scene graph generation dataset is preprocessed to obtain the preprocessed dynamic scene graph generation dataset. The specific steps include the following:

[0103] Each video data in the dynamic scene graph generation dataset is divided into multiple video frame time-series data;

[0104] When the number of video frame time-series data corresponding to the segmented video data is less than the preset video frame threshold, the video frame time-series data is supplemented by the last video frame time-series data in multiple video frame time-series data, so that the number of video frame time-series data in the video frame dataset is equal to the preset video frame threshold, and the video frame dataset corresponding to the video data is obtained.

[0105] When the number of video frame time series data corresponding to the segmented video data is equal to the preset video frame threshold, the video frame dataset corresponding to the video data includes all video frame time series data segmented from the video data.

[0106] When the number of video frame time-series data corresponding to the segmented video data is greater than the preset video frame threshold, the video data is sequentially divided into multiple time-series video segments, and each time-series video segment is divided into video frame time-series data with a number equal to the preset video frame threshold, thus obtaining the video frame dataset corresponding to the video data.

[0107] The video frame data in all video frame datasets are normalized to obtain the preprocessed dynamic scene graph generation dataset.

[0108] Specifically, the ActionGenome dataset was chosen for dynamic scene graph generation. It is the only publicly available large-scale dynamic scene graph generation dataset, containing 470,000 bounding boxes across 35 categories, 230,000 labeled frames, and 26 predicates representing visual relations. The evaluation metric used was the average recall (Recall@K) for scene graph detection. Experiments were conducted using the PyTorch framework and an NVIDIA 4090D GPU. The model was trained using the Adam optimizer, with the learning rates for the backbone network, encoder, and decoder all set to... The learning rate of the classifier in the entity parsing decoder and the grouped predicate decoder is The batch size was set to 8, and a total of 20 training rounds were conducted.

[0109] The data preprocessing process is as follows: First, a batch size of b is set. b video frames are sampled from each video in the dataset. When the number of video frames is less than the set batch size, we pad the video by copying the last frame. When the number of video frames is greater than the set batch size, we crop the video into multiple segments according to the batch size. When the last segment is smaller than the batch size, we select b frames in reverse order as the last segment. Simultaneously, we label all video frames trained using the copy-padding method and remove their results during evaluation to ensure fair comparison with other methods. Then, we scale the input video frames to a fixed size and normalize their pixel values ​​to obtain the final input.

[0110] The ResNet50 network was chosen as the deep convolutional neural network model. ResNet50 is a deep convolutional neural network based on the ResNet architecture. Its core innovation is the introduction of residual blocks, which solves the gradient vanishing and degradation problems in deep networks through skip connections. It is applied to image classification neural networks within deep convolutional neural networks. Image features are extracted using the ResNet50 network for a given video frame sequence. 3 indicates that the number of color channels is 3. and Indicates the height and width of the image. This represents the first video frame. This indicates the second video frame. Indicates the first One video frame, Indicates the first One video frame, Less than or equal to ,when equal hour, equal For each video frame After passing through the backbone network, low-resolution feature maps are generated. ,in Indicates the number of channels. and These are the height and width of the scaled feature map.

[0111] S2. Use a visual encoder to extract fine-grained features from the instance feature maps of each instance to obtain fine-grained feature maps of each instance; where the visual encoder is a DETR visual encoder, which includes a convolutional layer and a transformer encoder.

[0112] A visual encoder is used to extract fine-grained features from the instance feature maps of each instance, resulting in fine-grained feature maps for each instance. This process includes the following steps:

[0113] S201. Use convolutional layers to reduce the number of image channels in the instance feature maps of each instance to obtain low-channel instance feature maps of each instance.

[0114] S202. Use a transformer encoder to extract features from the low-channel instance feature maps of each instance to obtain fine-grained feature maps of each instance.

[0115] DETR-based visual encoders use Convolutional layers transfer the channels of the feature map from Reduce to And adjust the shape of the feature map to a sequence form. To adapt to DETR's transformer encoder processing. The encoder consists of multiple encoder layers, each containing a multi-head self-attention layer and a forward propagation network (FFN) to serialize feature maps. and position encoding As input.

[0116] The transformer encoder includes a multi-head self-attention layer. The self-attention calculation formula for the multi-head self-attention layer in the encoder layer is as follows:

[0117] ;

[0118] in, This represents the output value of the multi-head self-attention layer. Represents the low-channel instance feature map. Indicates position code, The number of image channels representing the low-channel instance feature map. This is represented as a transpose operation. Indicates to Perform the transpose operation. Represents the instance feature map.

[0119] S3. Use a visual language pre-trained model to predict the instance description text corresponding to the fine-grained feature map of each instance, and obtain the instance attribute information of each instance.

[0120] A visual language pre-trained model is used to predict the instance description text corresponding to the fine-grained feature maps of each instance, thereby obtaining the instance attribute information of each instance. The specific steps are as follows:

[0121] S301. Construct the CLIP model;

[0122] S302. Generate a visual triplet dataset using data from the dynamic scene graph dataset;

[0123] S303. Generate text prompt information using the visual triplet dataset;

[0124] S304. Input the text prompt information into the text encoder of the CLIP model to generate text embedding, and obtain a visual language pre-trained model;

[0125] S305. Input the fine-grained feature maps of each instance into the visual language pre-trained model to obtain the instance attribute information of each instance.

[0126] S4. Initialize a set of paired instance query sets, use a multi-head cross-attention mechanism to aggregate the spatial features of the image to update the query, obtain the paired instance representation, and regress to obtain the bounding box coordinates and category information of the paired instances;

[0127] Initialize a set of paired instance query sets, use a multi-head cross-attention mechanism to aggregate spatial features of the image to update the queries, obtain paired instance representations, and regress to obtain the bounding box coordinates and category information of paired instances, including the following steps:

[0128] S401, Construct a pairwise instance query dataset;

[0129] S402. Constructing an instance parsing decoder based on a multi-head cross-attention mechanism;

[0130] S403. Based on the fine-grained feature maps of each instance, retrieve the pairwise representations between each instance from the pairwise instance query dataset to obtain the pairwise subject representations;

[0131] S404. Based on the paired subject representation, use an instance parsing decoder to predict the bounding box coordinates and category information of paired instances in the paired subject representation.

[0132] The instance parser decoder is used to locate paired subjects in a video. It first initializes a paired subject query set. ,in Used to represent a pair of subjects, Objects used to represent paired entities Indicates the number of queries defined. This represents the feature embedding dimension for each query. Simultaneously, two learnable query embeddings, i.e., initial paired entities, are initialized. and the initial paired subject object , The instance parsing decoder consists of multiple decoder layers. Each decoder layer comprises a self-attention mechanism, a multi-head cross-attention mechanism, and a forward propagation layer, which obtains instance features by progressively updating the instance query.

[0133] Taking the main query set as an example, the self-attention mechanism in the decoder layer is used to update the query set. ,key ,value The calculation formula is as follows:

[0134] ;

[0135] Multi-head cross-attention convergence visual spatial features Update query to obtain main feature representation , among which query ,key ,value The calculation formula is as follows:

[0136] ;

[0137] Subsequently, the main features are represented through a forward propagation layer. Performing nonlinear transformations, where the linear layers can... A non-sequence association mapping is performed to form a fine-grained entity representation, and the calculation formula is as follows.

[0138] ;

[0139] The final subject feature representation is obtained. Then, the bounding box coordinates of the subject are predicted using a multilayer perceptron (MLP) structure. Since the main subjects in the dataset are usually people, there is no need to predict their categories. Following the subject-based query refinement approach can yield a fine-grained representation of the objects. and the bounding box coordinates of the object In addition, Input a linear classifier and a softmax function to predict the category to which an object belongs. ,in This indicates the number of object categories in the dataset. It implements the function of locating the subjects and objects to be predicted in video frames. Pairs of subjects and objects are paired instances.

[0140] During training, Hungarian matching is used to filter queries that match real-world scenario labels. Specifically, Hungarian matching filters queries by calculating a cost function (including instance bounding box cost, instance class cost, and predicate class cost). In the instance parsing decoder, we mainly calculate the instance bounding box and predicate class costs. For the query set... A query, where the instance bounding box cost function of the query is... The calculation formulas are as follows, using cross-entropy loss and IoU loss:

[0141] ;

[0142] in , It is a coefficient of the IoU (Intersection over Union) loss, used to adjust the proportion of the loss function. Indicates the number of ground-based real-world instances. This represents the predicted bounding box coordinates for the current query. Indicates the first The bounding box coordinates of a ground reality. This is a generalized IoU (Intersection over Union) calculation, with the following formula:

[0143] ;

[0144] Indicates the area covered, therefore The area covered express The area covered express The area covered express The area covered Indicates inclusion and The maximum region. For instance category cost, the cross-entropy loss function is used to calculate the cost function, and its formula is as follows:

[0145] ;

[0146] in Indicates prediction For the real category The negative logarithmic probability, This is a piecewise indicator function, which is 1 when the ground truth class label is not the background and 0 otherwise. For each query, the cost is calculated relative to all ground truth object classes in the current image. Furthermore, we optimize the model by calculating the cross-entropy loss and generalized IoU loss between the predicted query and the ground truth bounding box. The loss function for instance localization is calculated using the same formula as above.

[0147] S5. Predict the dynamic visual relationship of paired instances based on the fine-grained feature maps of paired instances to obtain the dynamic visual relationship information of paired instances.

[0148] like Figure 3 As shown, the dynamic visual relationship between pairs of instances is predicted based on the fine-grained feature maps of these pairs, thus obtaining dynamic visual relationship information for the pairs of instances. This process includes the following steps:

[0149] S501, Construct a grouped predicate decoder;

[0150] S502. Using a grouped predicate decoder, the fine-grained feature maps of paired instances are used to extract temporal dependencies through a temporal attention mechanism to obtain the spatiotemporal context features of paired instances.

[0151] S503. Using a grouped predicate decoder, predict the dynamic visual relationship of paired instances based on the spatiotemporal context features of paired instances to obtain dynamic visual relationship information of paired instances.

[0152] The grouped predicate decoder is used to predict dynamic visual relationships in videos. Specifically, dynamic visual relationships are semantically divided into three subsets: attentional relationships, spatial relationships, and contact relationships. We initialize a query set for each predicate subclass. Queries with the same index in each query set correspond to the same subject-object pair and contain visual triple information of 'subject-predicate-object'. The grouped predicate decoder is a three-branch structure, with each branch containing multiple decoder layers and a forward propagation layer for decoding a specific predicate subclass.

[0153] For each branch, visual spatial features are first introduced. The spatiotemporal attention mechanism is used to extract the spatiotemporal dependencies of the video and update the spatiotemporal context features of the video. Next, a self-attention mechanism is used to capture global context information within the decoder and adjusted according to different queries. We then embed the target-guided embedding in the instance parsing decoder. Queries added to each branch are used to align the representation consistency of the same query index across different decoders. ,key ,value ,in Indicates the first The formula for the self-attention mechanism is as follows: (The formula is incomplete and requires further context.)

[0154] ;

[0155] Subsequently, we aggregated spatiotemporal context features through a multi-head cross-attention mechanism, mined the dynamic visual relationships between different subject-object pairs, and updated the query results. + ,key ,value The formula for cross-attention is as follows:

[0156] ;

[0157] Finally, the current branch is processed through a forward propagation layer. A nonlinear transformation is performed to form a fine-grained visual triple representation, and the calculation formula is as follows.

[0158] ;

[0159] To obtain the final subject representation Then, a linear classifier and activation function are used to obtain the visual triple prediction category for the current branch. Since a subject-object pair may contain multiple spatial and contact class predicates, the sigmoid is used as the activation function here.

[0160] During training, focal loss is used to optimize the model's visual relationship prediction. Focal loss has a balance factor. and regulatory factors This allows for assigning more weight to samples that are difficult to classify. The specific formula is as follows:

[0161]

[0162] in The model represents the first The predicted probabilities of each predicate category, plus the cost function of the instance decoder, yield the overall cost function for scene graph generation. :

[0163] ;

[0164] The above cost function yields a value of... The cost matrix, elements Indicates the actual ground situation Matched query The cost of Hungarian matching can be obtained by selecting the index with the minimum cost. Each value represents an index in the query set, and then the overall cost function is calculated. Update the model parameters.

[0165] S6. Generate, based on the instance attribute information, bounding box coordinate information, category information, and dynamic visual relationship information of paired instances, such as... Figure 4 The dynamic scene diagram shown.

[0166] This invention predicts feature vectors using a deep convolutional neural network model, then predicts instance subjects using a visual language pre-trained model, and finally predicts paired instance subjects. Based on the positional information and dynamic visual relationships between paired instance subjects, a dynamic scene graph is generated. This invention improves prediction accuracy while enabling end-to-end training, providing a new paradigm for dynamic scene graph generation and simplifying the model training process.

[0167] The algorithm provided in this invention was compared with state-of-the-art methods. The comparison methods selected representative dynamic scene graph generation methods: STTran, TEMPURA, APT, STKET, and TD^2. STTran is a classic baseline method for dynamic scene graph generation, TEMPURA is a method for handling long-tailed distributions, and TD^2 is currently the best-performing dynamic scene graph generation method. All of these methods are two-stage scene graph generation algorithms. Experimental results are shown in Table 1.

[0168] Table 1

[0169]

[0170] As can be seen from the experimental comparison results in Table 1, this algorithm has achieved significant advantages over the other five methods. Furthermore, this algorithm can achieve end-to-end training, providing a new paradigm for dynamic scene graph generation and simplifying the model training pipeline.

[0171] like Figure 5 As shown, in some other embodiments, an end-to-end dynamic scene graph generation system is also provided, including:

[0172] Feature extraction module: Used to extract instance feature maps of each instance in the target video using a deep convolutional neural network model; and to extract fine-grained features of the instance feature maps of each instance using a visual encoder, thus obtaining fine-grained feature maps of each instance.

[0173] Attribute recognition module: Used to predict the instance description text corresponding to the fine-grained feature map of each instance using a visual language pre-trained model, and obtain the instance attribute information of each instance;

[0174] Paired instance recognition module: Initialize a set of paired instance query sets, use a multi-head cross-attention mechanism to aggregate the spatial features of the image to update the query, obtain the paired instance representation, and regress to obtain the bounding box coordinates and category information of the paired instances;

[0175] Visual Relationship Recognition Module: Used to predict the dynamic visual relationship between pairs of instances based on the fine-grained feature maps of the pairs of instances, and obtain the dynamic visual relationship information of the pairs of instances;

[0176] Scene graph construction module: used to generate dynamic scene graphs based on instance attribute information, bounding box coordinate information, category information, and dynamic visual relationship information of paired instances.

[0177] Based on Embodiment 1, this embodiment provides a storage medium that stores a computer program or computer instructions. When the computer program or computer instructions are executed by the processor of a computer, the steps of the above-described end-to-end dynamic scene graph generation method are implemented.

[0178] The storage medium can be an internal storage unit of any data processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart memory card, SD card, flash memory card, etc., mounted on the device. Furthermore, the storage medium can include both internal storage units of any data processing device and external storage devices. The computer-readable storage medium is used to store computer programs and other programs and data required by any data processing device, and can also be used to temporarily store data that has been output or will be output.

[0179] Example 3

[0180] Based on Embodiment 1, this embodiment provides a computing device, including a memory and one or more processors. The memory stores executable code. When the one or more processors execute the executable code, they implement the steps of the end-to-end dynamic scene graph generation method in Embodiment 1.

[0181] The memory can be an internal storage unit of any data processing device in any of the foregoing embodiments, such as a hard disk or RAM. The memory can also be an external storage device of any data processing device, such as a plug-in hard disk, smart memory card, SD card, flash memory card, etc., mounted on the device. Furthermore, the memory can include both internal storage units of any data processing device and external storage devices. The memory is used to store computer programs and other programs and data required by any data processing device, and can also be used to temporarily store data that has been output or will be output.

[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the concept and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An end-to-end dynamic scene graph generation method, characterized in that, Includes the following steps: A deep convolutional neural network model is used to extract instance feature maps of each instance in the target video; A visual encoder is used to extract fine-grained features from the instance feature maps of each instance to obtain fine-grained feature maps of each instance. A visual language pre-trained model is used to predict the instance description text corresponding to the fine-grained feature map of each instance, thereby obtaining the instance attribute information of each instance. Initialize a set of paired instance query sets, use a multi-head cross-attention mechanism to aggregate the spatial features of the image to update the query, obtain paired instance representations, and regress to obtain the bounding box coordinates and category information of paired instances; Based on the fine-grained feature maps of paired instances, predict the dynamic visual relationship of paired instances to obtain dynamic visual relationship information of paired instances. A dynamic scene graph is generated based on the instance attribute information, bounding box coordinate information, category information, and dynamic visual relationship information of paired instances. Predicting the dynamic visual relationship of paired instances based on the fine-grained feature maps of paired instances to obtain dynamic visual relationship information of paired instances includes the following steps: Construct a grouped predicate decoder; The grouped predicate decoder is used to extract the temporal dependencies of the fine-grained feature maps of paired instances through a temporal attention mechanism to obtain the spatiotemporal context features of paired instances; The dynamic visual relationship between pairs of instances is predicted by the grouped predicate decoder based on the spatiotemporal context features of the pairs of instances, thereby obtaining the dynamic visual relationship information of the pairs of instances. Dynamic visual relationships are semantically divided into three subsets: attentional relationships, spatial relationships, and contact relationships. A query set is initialized for each predicate subclass. Queries with the same index in each query set correspond to the same subject-object pair and contain visual triple information of 'subject-predicate-object'. The grouped predicate decoder is a three-branch structure, with each branch containing multiple decoder layers and a forward propagation layer, used to decode a specific predicate subclass; For each branch, visual spatial features are first introduced. The spatiotemporal attention mechanism is used to extract the spatiotemporal dependencies of the video and update the spatiotemporal context features of the video. Next, a self-attention mechanism is used to capture global context information within the decoder, and adjustments are made based on different queries to guide the embedding of the target in the instance parsing decoder. Add queries to each branch to align the representation consistency of the same query index across different decoders; Subsequently, a multi-head cross-attention mechanism is used to aggregate spatiotemporal context features, mine the dynamic visual relationships between different subject-object pairs, and update the subject representation of the current branch obtained from the query. , Finally, the main body of the current branch is represented through a forward propagation layer. A nonlinear transformation is performed to form a fine-grained visual triple representation. To obtain the final subject representation Then, the visual triplet prediction category of the current branch is obtained by using a linear classifier and activation function.

2. The end-to-end dynamic scene graph generation method according to claim 1, characterized in that, The deep convolutional neural network model is used to extract instance feature maps of each instance in the target video, including the following steps: Construct the deep convolutional neural network model; Collect dynamic scene graphs to generate a dataset; The data in the dynamic scene graph generation dataset is preprocessed to obtain the preprocessed dynamic scene graph generation dataset. The deep convolutional neural network model is trained using the dataset generated from the preprocessed dynamic scene graph to obtain a feature extraction backbone network model. The target video is segmented into a sequence of multiple video frames to obtain a video frame sequence dataset; The video frame sequence dataset is input into the feature extraction backbone network model to obtain the instance feature map for each instance.

3. The end-to-end dynamic scene graph generation method according to claim 2, characterized in that, The dynamic scene graph generation dataset includes multiple video data; the preprocessed dynamic scene graph generation dataset includes multiple video frame datasets, and the multiple video frame datasets correspond one-to-one with the multiple video data. The data in the dynamic scene graph generation dataset is preprocessed to obtain a preprocessed dynamic scene graph generation dataset, specifically including the following steps: Each video data in the dynamic scene graph generation dataset is divided into multiple video frame time-series data; When the number of video frame time-series data corresponding to the segmented video data is less than a preset video frame threshold, the video frame time-series data is supplemented by the last video frame time-series data in the multiple video frame time-series data, so that the number of video frame time-series data in the video frame dataset is equal to the preset video frame threshold, and the video frame dataset corresponding to the video data is obtained. When the number of video frame time-series data corresponding to the segmented video data is equal to a preset video frame threshold, the video frame dataset corresponding to the video data includes all the video frame time-series data segmented from the video data. When the number of video frame time-series data corresponding to the segmented video data is greater than the preset video frame threshold, the video data is sequentially divided into multiple time-series video segments, and each time-series video segment is divided into video frame time-series data with a number equal to the preset video frame threshold, to obtain the video frame dataset corresponding to the video data. The video frame data in all the video frame datasets are normalized to obtain the preprocessed dynamic scene graph generation dataset.

4. The end-to-end dynamic scene graph generation method according to claim 1, characterized in that, The visual encoder is a DETR visual encoder, which includes a convolutional layer and a transformer encoder; A visual encoder is used to extract fine-grained features from the instance feature maps of each instance, resulting in fine-grained feature maps for each instance. This process includes the following steps: The convolutional layer is used to reduce the number of image channels of the instance feature map of each instance to obtain the low-channel instance feature map of each instance; The transformer encoder is used to extract features from the low-channel instance feature maps of each instance to obtain the fine-grained feature maps of each instance.

5. The end-to-end dynamic scene graph generation method according to claim 4, characterized in that, The transformer encoder includes a multi-head self-attention layer, and the self-attention calculation formula for the multi-head self-attention layer in the encoder layer is as follows: ； in, This represents the output value of the multi-head self-attention layer. This represents the feature map of the low-channel instance. Indicates position code, This represents the number of image channels in the low-channel instance feature map. This is represented as a transpose operation. Indicates to Perform the transpose operation. This represents the feature map of the instance.

6. The end-to-end dynamic scene graph generation method according to claim 2, characterized in that, The visual language pre-trained model is used to predict the instance description text corresponding to the fine-grained feature maps of each instance, thereby obtaining the instance attribute information of each instance. The specific steps are as follows: Construct the CLIP model; The data in the dynamic scene graph dataset is used to generate a visual triplet dataset; Text prompts are generated using the aforementioned visual triplet dataset; The text prompt information is input into the text encoder of the CLIP model to generate text embeddings, thereby obtaining the visual language pre-trained model; The fine-grained feature maps of each instance are input into the visual language pre-trained model to obtain the instance attribute information of each instance.

7. The end-to-end dynamic scene graph generation method according to claim 1, characterized in that, Initialize a set of paired instance query sets, use a multi-head cross-attention mechanism to aggregate spatial features of the image to update the queries, obtain paired instance representations, and regress to obtain the bounding box coordinates and category information of paired instances, including the following steps: Construct the pairwise instance query dataset; An instance parsing decoder is constructed based on a multi-head cross-attention mechanism; Based on the fine-grained feature maps of each instance, the pairwise representations between each instance are retrieved from the pairwise instance query dataset to obtain the pairwise subject representations; Based on the paired subject representation, the instance parser decoder is used to predict the bounding box coordinates and category information of the paired instances in the paired subject representation.

8. An end-to-end dynamic scene graph generation system, characterized in that, include: Feature extraction module: Used to extract instance feature maps of each instance in the target video using a deep convolutional neural network model; A visual encoder is used to extract fine-grained features from the instance feature maps of each instance to obtain fine-grained feature maps of each instance. Attribute recognition module: used to predict the instance description text corresponding to the fine-grained feature map of each instance using a visual language pre-trained model, and obtain the instance attribute information of each instance; The pairwise instance recognition module is used to initialize a set of pairwise instance query sets, use a multi-head cross-attention mechanism to aggregate the spatial features of the image to update the query, obtain the pairwise instance representation, and regress to obtain the bounding box coordinates and category information of the pairwise instances. Visual Relationship Recognition Module: Used to predict the dynamic visual relationship of paired instances based on the fine-grained feature map of paired instances, and obtain the dynamic visual relationship information of paired instances; Scene graph construction module: used to generate dynamic scene graphs based on instance attribute information, bounding box coordinate information, category information, and dynamic visual relationship information of paired instances; Predicting the dynamic visual relationship of paired instances based on the fine-grained feature maps of paired instances to obtain dynamic visual relationship information of paired instances includes the following steps: Construct a grouped predicate decoder; The grouped predicate decoder is used to extract the temporal dependencies of the fine-grained feature maps of paired instances through a temporal attention mechanism to obtain the spatiotemporal context features of paired instances; The dynamic visual relationship between pairs of instances is predicted by the grouped predicate decoder based on the spatiotemporal context features of the pairs of instances, thereby obtaining the dynamic visual relationship information of the pairs of instances. Dynamic visual relationships are semantically divided into three subsets: attentional relationships, spatial relationships, and contact relationships. A query set is initialized for each predicate subclass. Queries with the same index in each query set correspond to the same subject-object pair and contain visual triple information of 'subject-predicate-object'. The grouped predicate decoder is a three-branch structure, with each branch containing multiple decoder layers and a forward propagation layer, used to decode a specific predicate subclass; For each branch, visual spatial features are first introduced. The spatiotemporal attention mechanism is used to extract the spatiotemporal dependencies of the video and update the spatiotemporal context features of the video. Next, a self-attention mechanism is used to capture global context information within the decoder, and adjustments are made based on different queries to guide the embedding of the target in the instance parsing decoder. Add queries to each branch to align the representation consistency of the same query index across different decoders; Subsequently, a multi-head cross-attention mechanism is used to aggregate spatiotemporal context features, mine the dynamic visual relationships between different subject-object pairs, and update the subject representation of the current branch obtained from the query. , Finally, the main body of the current branch is represented through a forward propagation layer. A nonlinear transformation is performed to form a fine-grained visual triple representation. To obtain the final subject representation Then, the visual triplet prediction category of the current branch is obtained by using a linear classifier and activation function.

9. A computing device, characterized in that, The method includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, they implement the steps of the end-to-end dynamic scene graph generation method as described in any one of claims 1 to 7.

Images