End-to-end dynamic scene graph generation method and system

Abstract

The invention relates to an end-to-end dynamic scene graph generation method and system. The method comprises the following steps: extracting an instance feature graph of each instance in a target video; extracting fine-grained features of the instance feature map of each instance to obtain a fine-grained feature map of each instance; predicting an instance description text corresponding to the fine-grained feature map of each instance, and obtaining instance attribute information of each instance; pairwise representations among the instances are inquired from the pairwise instance inquiry data set, and bounding box coordinate information and category information of the pairwise instances are obtained; predicting a dynamic visual relationship of the paired instances to obtain dynamic visual relationship information of the paired instances; generating a dynamic scene graph according to the instance attribute information, the bounding box coordinate information, the category information and the dynamic visual relationship information of the paired instances; according to the method, end-to-end training can be realized while the prediction precision is improved, a new normal form is provided for dynamic scene graph generation, and the training steps of the model are simplified.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computing visual dynamic scene graph generation, and particularly relates to an end-to-end dynamic scene graph generation method and system. BACKGROUND

[0002] Scene Graph Generation (SGG) is an important task in computer vision, aiming to automatically parse structured scene representations from images. Scene graph generation aims to generate a complete structured representation for image or video content, which can bridge the gap between computer vision and language. Scene graphs contain rich visual and semantic information and can be used for various vision tasks such as image retrieval, image captioning, and visual question answering. Scene graphs are composed of a series of triples of "subject-predicate-object". Image scene graph generation has made great progress, but video scene graph generation is still in its infancy.

[0003] Johnson (Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Im- age retrieval using scene graphs. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR),pages 3668)first proposed to use scene graphs to describe detailed semantic information of scenes. With the rapid development of convolutional neural networks (CNN), researchers have proposed various scene graph generation methods based on deep learning. Image scene graph methods usually follow a two-stage paradigm and focus on exploring spatial context features. Lin (LinLin, Chong Shi, Yibing Zhan, Zuopeng Yang, Yaqi Wu, and Dacheng Tao. Td2-net: Toward denoising and debi- asing for video scene graph generation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pages 3495AAAI Confere) et al. proposed a direction-aware message passing module that focuses on the predicate construction direction of the scene graph and generates a predicate attention map to more accurately recognize predicates. Jingkuan Song, and Lianli Gao. Learning to generate scene graph from head to tail. In 2022 IEEE International Con- ference on Multimedia and Expo (ICME), pages 1ional Con) et al. proposed a course reweighting mechanism to transfer the robust features of the head predicate to the tail predicate to alleviate the long-tail distribution phenomenon in the scene graph.

[0004] Compared with image scene graphs, dynamic scene graphs contain richer spatiotemporal context information and more complex visual relationships. Rosenhahn, and Michael Ying Yang. Spatial-temporal trans- former for dynamic scene graph generation. In Proceedings of the IEEE / CVF International Conference on Computer Vi- sion (ICCV), pages 16372ional Confer) et al. proposed a two-stage dynamic scene graph baseline based on transformer, which learns the temporal dependency between "subject-object" pairs through an encoder-decoder structure. They proved the effectiveness of temporal dependency for dynamic scene graph generation. Lin. Spatial-temporal knowledge-embedded transformer for video scene graph generation. IEEE Transactions on Image Processing (TIP), 33:556. IEEE T) et al. further introduced temporal prior information to guide the aggregation of spatial context and temporal correlation. These two-stage methods emphasize the role of temporal correlation in generating dynamic scene graphs and focus on designing modules to learn spatiotemporal context in videos. However, these two-stage methods hinder end-to-end learning, as incorrect detection results can affect the training of prediction models and produce biased scene graphs.

[0005] The long-tail distribution of visual relationships is a major challenge in dynamic scene graph generation. Lu et al. (Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In European Conference on Computer Vision (ECCV), pages 852, European Conference) leveraged language priors in semantic word embeddings to identify tail predicates from a global perspective. Nag et al. (Nayak Nag, Kyle Min, Subarna Tripathi, and Amit K Roy-Chowdhury. Unbiased scene graph generation in videos. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22803–22813, 2023) adopted a memory-guided training strategy to transfer knowledge from the abundant head predicates to the tail predicates to address the long-tail problem. However, these methods are limited by the contextual information of local region features and rely only on a limited number of tail predicate samples, resulting in suboptimal performance. Summary of the Invention

[0006] In order to solve the technical problem that the dynamic scene graph generation in the prior art is limited to the context information of local area features and only depends on limited tail predicate samples, resulting in suboptimal performance, the application provides an end-to-end dynamic scene graph generation method and system.

[0007] The technical solution of the application for solving the above technical problem is as follows: An end-to-end dynamic scene graph generation method comprises the following steps: An instance feature map of each instance in a target video is extracted by using a deep convolutional neural network model; Fine-grained features of the instance feature map of each instance are extracted by using a visual encoder to obtain a fine-grained feature map of each instance; Instance description texts corresponding to the fine-grained feature map of each instance are predicted by using a visual language pre-training model to obtain instance attribute information of each instance; A set of paired instance query sets is initialized, spatial features of an image are aggregated and updated by using a multi-head cross attention mechanism to obtain paired instance representations, and the boundary box coordinates and category information of the paired instances are regressed; Dynamic visual relationships of the paired instances are predicted according to the fine-grained feature map of the paired instances to obtain dynamic visual relationship information of the paired instances; A dynamic scene graph is generated according to the instance attribute information, boundary box coordinate information, category information and dynamic visual relationship information of the paired instances.

[0008] The application has the beneficial effects that the feature vector is predicted by the deep convolutional neural network model, the instance subject is predicted by the visual language pre-training model, the paired instance subjects are predicted respectively, and finally the dynamic scene graph is generated according to the position information and dynamic visual relationships between the paired instance subjects. The application improves the prediction accuracy, realizes end-to-end training, provides a new paradigm for dynamic scene graph generation, and simplifies the training steps of the model.

[0009] On the basis of the above technical solution, the application can be further improved as follows.

[0010] Further, the instance feature map of each instance in a target video is extracted by using a deep convolutional neural network model, comprising the following steps: The deep convolutional neural network model is constructed; A dynamic scene graph generation dataset is collected; The data of the dynamic scene graph generation dataset is preprocessed to obtain a preprocessed dynamic scene graph generation dataset; The deep convolutional neural network model is trained by using the preprocessed dynamic scene graph generation dataset to obtain a feature extraction backbone network model; segmenting the target video into a plurality of video frame sequence data to obtain a video frame sequence data set; inputting the video frame sequence data set into the feature extraction backbone network model to obtain the instance feature map of each instance.

[0011] Further, the dynamic scene graph generation dataset includes a plurality of video data; the pre-processed dynamic scene graph generation dataset includes a plurality of video frame data sets, and each of the plurality of video frame data sets corresponds to one of the plurality of video data; The data of the dynamic scene graph generation dataset is preprocessed to obtain a pre-processed dynamic scene graph generation dataset, which includes the following steps: Each of the video data in the dynamic scene graph generation dataset is segmented into a plurality of video frame time sequence data; When the number of video frame time sequence data corresponding to the segmented video data is less than a preset video frame threshold, the last video frame time sequence data in the plurality of video frame time sequence data is used to supplement the video frame time sequence data, so that the number of video frame time sequence data in the video frame data set is equal to the preset video frame threshold, and the video frame data set corresponding to the video data is obtained; When the number of video frame time sequence data corresponding to the segmented video data is equal to the preset video frame threshold, the video frame data set corresponding to the video data includes all the video frame time sequence data segmented from the video data; When the number of video frame time sequence data corresponding to the segmented video data is greater than the preset video frame threshold, the video data is sequentially segmented into a plurality of time sequence video segments, and each of the time sequence video segments is segmented into a plurality of video frame time sequence data with a number equal to the preset video frame threshold, and the video frame data set corresponding to the video data is obtained; The video frame data in all the video frame data sets is normalized to obtain the pre-processed dynamic scene graph generation dataset.

[0012] Further, the visual encoder is a DETR visual encoder, and the DETR visual encoder includes a convolution layer and a transformer encoder. The visual encoder is used to extract the fine-grained features of the instance feature map of each instance to obtain a fine-grained feature map of each instance, including the following steps: The convolution layer is used to reduce the image channel number of the instance feature map of each instance to obtain a low-channel instance feature map of each instance; The transformer encoder is used to extract features from the low-channel instance feature map of each instance to obtain the fine-grained feature map of each instance.

[0013] Further, the transformer encoder comprises a multi-head self-attention layer, and a self-attention calculation formula of the multi-head self-attention layer in the encoder layer is as follows: ; Wherein, represents an output value of the multi-head self-attention layer, represents the low-channel instance feature map, represents position encoding, represents the number of image channels of the low-channel instance feature map, represents a transpose operation, represents a transpose operation on , represents the instance feature map.

[0014] Further, an instance attribute information of each instance is obtained by predicting an instance description text corresponding to the fine-grained feature map of each instance by using a visual language pre-training model, and the specific steps are as follows: Constructing a CLIP model; Generating a visual triple data set by using data in a dynamic scene graph generation data set; Generating text prompt information by using the visual triple data set; Inputting the text prompt information into a text encoder of the CLIP model to generate a text embedding, thereby obtaining the visual language pre-training model; Inputting the fine-grained feature map of each instance into the visual language pre-training model to obtain the instance attribute information of each instance.

[0015] Further, a set of paired instance query sets is initialized, a multi-head cross-attention mechanism is used to aggregate spatial features of an image to update a query, a paired instance representation is obtained, and a bounding box coordinate and a class information of a paired instance are regressed, including the following steps: Constructing the paired instance query data set; Constructing an instance parsing decoder based on a multi-head cross-attention mechanism; Obtaining a paired representation between each instance from the paired instance query data set according to the fine-grained feature map of each instance, thereby obtaining a paired subject representation; Predicting the bounding box coordinate information and the class information of the paired instance in the paired subject representation by using the instance parsing decoder according to the paired subject representation.

[0016] Further, a dynamic visual relationship of a paired instance is predicted according to the fine-grained feature map of the paired instance, thereby obtaining dynamic visual relationship information of the paired instance, including the following steps: Constructing a grouping predicate decoder; Using the grouped predicate decoder, the fine-grained feature maps of the paired instances are subjected to a temporal attention mechanism to extract temporal dependencies, thereby obtaining spatiotemporal context features of the paired instances; The grouped predicate decoder is used to predict the dynamic visual relationship between pairs of instances according to the spatiotemporal context features of the pairs of instances, thereby obtaining the dynamic visual relationship information of the pairs of instances.

[0017] In order to solve the above technical problems, the present invention provides an end-to-end dynamic scene graph generation system, the specific contents of which are as follows: An end-to-end dynamic scene graph generation system, comprising: Feature extraction module: used to extract instance feature maps of each instance in the target video using a deep convolutional neural network model; use a visual encoder to extract fine-grained features of the instance feature maps of each instance to obtain fine-grained feature maps of each instance; Attribute recognition module: used to predict the instance description text corresponding to the fine-grained feature map of each instance using a visual language pre-training model to obtain instance attribute information of each instance; Paired Instance Recognition Module: This module initializes a set of paired instance queries, aggregates the spatial features of the image using a multi-head cross-attention mechanism to update the query, obtains paired instance representations, and regresses the bounding box coordinates and category information of the paired instances. A visual relationship recognition module is configured to predict the dynamic visual relationship between paired instances based on the fine-grained feature maps of the paired instances, thereby obtaining dynamic visual relationship information of the paired instances; Scene graph construction module: used to generate a dynamic scene graph based on instance attribute information, bounding box coordinate information, category information and dynamic visual relationship information of paired instances.

[0018] In order to solve the above technical problems, the present invention provides a computing device, and its specific technical solutions are as follows: A computing device includes a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the steps of the above-mentioned end-to-end dynamic scene graph generation method are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of an end-to-end dynamic scene graph generation method according to an embodiment of the present invention; Figure 2 Schematic diagram of the deep convolutional neural network model structure in an embodiment of the present invention; Figure 3 This is a dynamic visual relationship structure diagram in an embodiment of the present invention; Figure 4A schematic diagram of the structure of a dynamic scene graph in an embodiment of the present invention; Figure 5 The figure is a flow chart of an end-to-end dynamic scene graph generation system in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0021] like Figure 1 As shown, this embodiment provides an end-to-end dynamic scene graph generation method, including the following steps: S1. Use a deep convolutional neural network model to extract instance feature maps of each instance in the target video; like Figure 2 As shown in FIG, a deep convolutional neural network model is used to extract instance feature maps of each instance in the target video, including the following steps: S101, build a deep convolutional neural network model; S102, collecting dynamic scene graphs to generate data sets; S103, pre-processing the data of the dynamic scene graph generation data set to obtain a pre-processed dynamic scene graph generation data set; S104, using the pre-processed dynamic scene graph to generate a data set to train a deep convolutional neural network model to obtain a feature extraction backbone network model; S105, dividing the target video into sequence data of multiple video frames to obtain a video frame sequence data set; S106: Input the video frame sequence dataset into the feature extraction backbone network model to obtain an instance feature map of each instance.

[0022] The dynamic scene graph generation data set includes a plurality of video data; the pre-processed dynamic scene graph generation data set includes a plurality of video frame data sets, and the plurality of video frame data sets correspond one to one with the plurality of video data; Preprocessing the data of the dynamic scene graph generation dataset to obtain a pre-processed dynamic scene graph generation dataset specifically includes the following steps: Each video data in the dynamic scene graph generation dataset is divided into multiple video frame time series data; When the amount of video frame timing data corresponding to the segmented video data is less than a preset video frame threshold, the video frame timing data is supplemented with the last frame of video frame timing data in the plurality of video frame timing data so that the amount of video frame timing data in the video frame data set is equal to the preset video frame threshold, thereby obtaining a video frame data set corresponding to the video data; When the number of video frame time sequence data corresponding to the segmented video data is equal to the preset video frame threshold, the video frame data set corresponding to the video data includes all video frame time sequence data segmented by the video data; When the number of video frame time sequence data corresponding to the segmented video data is greater than the preset video frame threshold, the video data is sequentially segmented into a plurality of time sequence video segments, and each time sequence video segment is segmented into video frame time sequence data equal in number to the preset video frame threshold, to obtain a video frame data set corresponding to the video data; The video frame data in all video frame data sets is normalized to obtain a preset dynamic scene graph generation data set.

[0023] Specifically, the dynamic scene graph generation data set selects the ActionGenome data set, which is the only public large dynamic scene graph generation data set, contains 470,000 bounding boxes of 35 categories, a total of 230,000 labeled frames, and contains 26 predicates as visual relationships. The evaluation index used is the average recall Recall@K under the scene graph detection. The experiment is carried out under the Pytorch framework, using an NVIDIA 4090D GPU. The model is trained by the Adam optimizer, the learning rate of the backbone network, the encoder, and the decoder , and the learning rate of the classifier in the entity parsing decoder and the grouped predicate decoder is . The batch size is set to 8, and a total of 20 rounds are trained.

[0024] The data preprocessing process is as follows: first, set a batch size value b, sample b video frames from each video in the data set, when the number of video frames is less than the set batch size value, we fill it by copying the last frame of the video. When the number of video frames is greater than the set batch size value, we cut the video into multiple segments according to the batch size value, and when the last segment is less than the batch size value, we select b frames in reverse order as the last segment. At the same time, we will mark all the video frames that have been filled by copying for training, and delete their results during evaluation to facilitate fair comparison with other methods. Subsequently, we scale the input video frames to a fixed size and normalize the pixel values to obtain the input.

[0025] The deep convolutional neural network model selects the Resnet50 network, which is a deep convolutional neural network based on the residual network (ResNet) architecture. Its core innovation is the introduction of residual blocks (Residual Blocks), which solves the gradient disappearance and degradation problem in deep networks through skip connections (Skip Connections); the deep convolutional neural network is applied to image classification neural networks. The Resnet50 network is used to extract image features. For a given video frame sequence , 3 represents the number of color channels is 3, and represents the height and width of the image, represents the first video frame, represents the second video frame, represents the first video frame, represents the first video frame, is less than or equal to , when is equal to , is equal to , for each video frame , a low-resolution feature map is generated after the backbone network , wherein represents the number of channels, and is the height and width of the scaled feature map.

[0026] S2, a visual encoder is used to extract the fine-grained features of the instance feature map of each instance, to obtain the fine-grained feature map of each instance; wherein the visual encoder is a DETR visual encoder, and the DETR visual encoder includes a convolutional layer and a transformer encoder; The visual encoder is used to extract the fine-grained features of the instance feature map of each instance, to obtain the fine-grained feature map of each instance, including the following steps: S201, a convolutional layer is used to reduce the image channel number of the instance feature map of each instance, to obtain a low-channel instance feature map of each instance; S202, a transformer encoder is used to extract features from the low-channel instance feature map of each instance, to obtain a fine-grained feature map of each instance.

[0027] The DETR-based visual encoder uses a convolutional layer to reduce the channel of the feature map from to , and adjusts the shape of the feature map to a sequence form to adapt to the transformer encoder of the DETR for processing. The encoder is composed of multiple encoder layers, and each encoder layer contains a multi-head self-attention layer and a forward propagation network (FFN), taking the serialized feature map and position encoding as input.

[0028] The transformer encoder includes a multi-head self-attention layer, and the self-attention calculation formula of the multi-head self-attention layer in the encoder layer is as follows: ; wherein, represents an output value of the multi-head self-attention layer, represents a low-channel instance feature map, represents a position encoding, represents a number of image channels of the low-channel instance feature map, represents a transposition operation, represents a transposition operation on , represents an instance feature map.

[0029] S3, predicting instance description texts corresponding to the fine-grained feature maps of each instance by using the visual language pre-training model to obtain instance attribute information of each instance; predicting instance description texts corresponding to the fine-grained feature maps of each instance by using the visual language pre-training model to obtain instance attribute information of each instance, and the specific steps are as follows: S301, constructing a CLIP model; S302, generating a visual triple data set by using dynamic scene graph generation data in a data set; S303, generating text prompt information by using the visual triple data set; S304, inputting the text prompt information into a text encoder of the CLIP model to generate a text embedding to obtain a visual language pre-training model; S305, inputting the fine-grained feature maps of each instance into the visual language pre-training model to obtain instance attribute information of each instance.

[0030] S4, initializing a set of paired instance query sets, aggregating spatial features of an image by using a multi-head cross-attention mechanism to update queries, obtaining paired instance representations, and regressing to obtain boundary box coordinates and class information of the paired instances; initializing a set of paired instance query sets, aggregating spatial features of an image by using a multi-head cross-attention mechanism to update queries, obtaining paired instance representations, and regressing to obtain boundary box coordinates and class information of the paired instances, including the following steps: S401, constructing a paired instance query data set; S402, constructing an instance parsing decoder based on a multi-head cross-attention mechanism; S403, obtaining paired representations between each instance from the paired instance query data set according to the fine-grained feature maps of each instance to obtain paired subject representations; S404, predicting boundary box coordinate information and class information of paired instances in the paired subject representations by using the instance parsing decoder according to the paired subject representations.

[0031] The instance parsing decoder is used to locate paired subjects in the video. It first initializes a paired subject query set. ,in Used to represent the subject of a pair of subjects, An object representing a pair of entities, Indicates the number of queries defined, Represents the feature embedding dimension of each query. At the same time, two learnable query embeddings, namely the initial paired entities, are initialized and the object of the initial paired subject , The instance parsing decoder consists of multiple decoder layers, each of which consists of a self-attention mechanism, a multi-head cross-attention mechanism, and a forward propagation layer, which obtains instance features by gradually updating the instance query.

[0032] Taking the subject query set as an example, the self-attention mechanism in the decoder layer is used to update the query set. ,key ,value , the calculation formula is as follows: ; Aggregating visual spatial features using multi-head cross attention Update query to get the main feature representation , where the query ,key ,value , the calculation formula is as follows: ; Then, the subject feature representation is processed through a forward propagation layer. Perform nonlinear transformation, in which the linear layer can Perform non-sequential association mapping to form a fine-grained entity representation. The calculation formula is as follows.

[0033] ; Get the final subject feature representation Finally, the bounding box coordinates of the subject are predicted through the multi-layer perceptron (MLP) structure The subject of the dataset is usually a person, so there is no need to predict its category. Following the subject query refinement method, a fine-grained representation of the object can be obtained. and the object's bounding box coordinates In addition, Input linear classifier and Softmax function to predict the category of the object ,in Represents the number of object categories in the dataset. It implements the localization function of the subject and object to be predicted in the video frame. Pairs of subjects and objects are called paired instances.

[0034] During the training process, Hungarian matching is used to filter queries that match the actual scene labels. Specifically, Hungarian matching filters queries by calculating the cost function (including instance bounding box cost, instance category cost, and predicate category cost). In the instance parsing decoder, we mainly calculate the instance bounding box and predicate category costs. queries, where the instance bounding box cost function of the query is Calculated by cross entropy loss and IoU loss, the calculation formula is as follows: ; in , Is the coefficient of IoU (intersection over union) loss, which is used to adjust the ratio of the loss function. represents the number of ground truth instances, Represents the bounding box coordinate prediction value of the current query, Indicates the The bounding box coordinates of the ground truth. It is a generalized IoU (intersection over union) calculation, and the formula is as follows: ; represents the area covered, so The area covered, express The area covered, express The area covered, express The area covered, Indicates inclusion and For the instance category cost, the cross entropy loss function is used to calculate the cost function, and its formula is as follows: ; in Represents prediction For real categories The negative log probability of is a piecewise indicator function that is 1 when the true class label is not background and 0 otherwise. For each query, the cost value is calculated for all true object classes in the current image. In addition, we optimize the model by calculating the cross entropy loss and generalized IoU loss between the predicted query and the true ground truth bounding box. The loss function for instance localization is calculated in the same way as above.

[0035] S5, predicting the dynamic visual relationship of the pair instance according to the fine-grained feature map of the pair instance, to obtain dynamic visual relationship information of the pair instance; As Figure 3 shown, predicting the dynamic visual relationship of the pair instance according to the fine-grained feature map of the pair instance, to obtain dynamic visual relationship information of the pair instance, includes the following steps: S501, constructing a grouping predicate decoder; S502, using the grouping predicate decoder to extract the time dependence of the fine-grained feature map of the pair instance through the time attention mechanism, to obtain the spatio-temporal context feature of the pair instance; S503, using the grouping predicate decoder to predict the dynamic visual relationship of the pair instance according to the spatio-temporal context feature of the pair instance, to obtain the dynamic visual relationship information of the pair instance.

[0036] The grouping predicate decoder is used to predict the dynamic visual relationship of the video. Specifically, the dynamic visual relationship is divided into three subsets according to semantics, namely attention relationship, spatial relationship and contact relationship. We initialize a query set for each predicate subclass. The queries with the same index in each query set correspond to the same subject-object pair and contain visual triple information of'subject-predicate-object'. The grouping predicate decoder is a three-branch structure, each branch contains multiple decoder layers and a forward propagation layer, which is used to decode a specific predicate subclass.

[0037] For each branch, first introduce the visual spatial feature , extract the spatio-temporal dependence of the video using the spatio-temporal attention mechanism, and update the spatio-temporal context feature of the video. Then, use the self-attention mechanism to capture the global context information inside the decoder and adjust according to different queries. We add the target guide embedding of the instance parsing decoder to the query of each branch to align the consistency of the representation of the same query index in different decoders. The query , key , value , where represents the th branch, and the formula of the self-attention mechanism is as follows: ; Subsequently, we aggregate the spatio-temporal context features through the multi-head cross-attention mechanism, mine the dynamic visual relationship between different subject-object pairs and update the query to obtain + , key , value . The formula of cross-attention is as follows: ; Finally, a forward propagation layer is used to perform a nonlinear transformation on the current branch to form a fine-grained visual triple representation, and the calculation formula is as follows.

[0038] ; to obtain the final subject representation , and then a linear classifier and an activation function are used to obtain the visual triple prediction category of the current branch. Since a pair of subject-object pairs may contain multiple spatial and contact predicate categories, sigmoid is used as the activation function.

[0039] In the training process, focal loss is used to optimize the visual relationship prediction of the model, and focal loss has a balance factor and a regulation factor , which can give more weight to difficult-to-classify samples, and the specific formula is as follows: Wherein represents the prediction probability of the model for the first predicate category, and the cost function of the instance decoder is added to obtain the overall cost function of the scene graph generation : ; The above cost function can obtain a cost matrix of , and the element represents the cost of matching the ground truth to the query , and the result of the Hungarian matching can be obtained by selecting the index of the minimum value , and each value represents the index in the query set, and then the overall cost function is calculated to update the model parameters.

[0040] S6, according to the instance attribute information, the bounding box coordinate information, the category information and the dynamic visual relationship information of the pair of instances, a dynamic scene graph as shown in Figure 4 is generated.

[0041] The embodiment of the application predicts a feature vector through a deep convolutional neural network model, then predicts an instance subject through a visual language pre-training model, then predicts a pair of instance subjects respectively, and finally generates a dynamic scene graph according to the position information and the dynamic visual relationship between the pair of instance subjects. The application improves the prediction accuracy and can also realize end-to-end training, provides a new paradigm for dynamic scene graph generation, and simplifies the training steps of the model.

[0042] The algorithm provided by the present application is compared with the most advanced method, and some representative dynamic scene graph generation methods STTran, TEMPURA, APT, STKET, TD^2 are selected as comparison methods. Among them, STTran is a classic dynamic scene graph generation baseline method, TEMPURA is a dynamic scene graph generation method for solving long-tail distribution, TD^2 is the best dynamic scene graph generation method at present, and the above methods are all two-stage scene graph generation algorithms. The experimental results are shown in Table 1.

[0043] Table 1 As can be seen from the experimental comparison results in Table 1, the algorithm has obvious advantages compared with the other five methods, and the algorithm can realize end-to-end training, providing a new paradigm for dynamic scene graph generation and simplifying the training pipeline of the model.

[0044] As Figure 5 shown, in some other embodiments, an end-to-end dynamic scene graph generation system is also provided, comprising: a feature extraction module: configured to extract instance feature maps of each instance in the target video by using a deep convolutional neural network model; and extract fine-grained features of the instance feature maps of each instance by using a visual encoder, to obtain fine-grained feature maps of each instance; an attribute recognition module: configured to predict instance description texts corresponding to the fine-grained feature maps of each instance by using a visual language pre-training model, to obtain instance attribute information of each instance; a pair instance recognition module: configured to initialize a set of pair instance query sets, aggregate spatial features of the image to update the query by using a multi-head cross attention mechanism, to obtain pair instance representations, and regress to obtain boundary box coordinates and class information of the pair instances; a visual relationship recognition module: configured to predict dynamic visual relationships of the pair instances according to the fine-grained feature maps of the pair instances, to obtain dynamic visual relationship information of the pair instances; a scene graph construction module: configured to generate a dynamic scene graph according to the instance attribute information, the boundary box coordinate information, the class information and the dynamic visual relationship information of the pair instances.

[0045] Based on Embodiment 1, the present embodiment provides a storage medium storing a computer program or computer instructions, which, when executed by a processor of a computer, implements the steps of the above-mentioned end-to-end dynamic scene graph generation method.

[0046] The storage medium can be an internal storage unit of any data processing-capable device described in any of the preceding embodiments, such as a hard disk or a memory. The storage medium can also be an external storage device of any data processing-capable device, such as a plug-in hard disk, a smart storage card, an SD card, a flash memory card, and the like. Further, the storage medium can include both an internal storage unit and an external storage device of any data processing-capable device. The computer-readable storage medium is used to store a computer program and other programs and data required by any data processing-capable device, and can also be used to temporarily store data that has been output or is to be output.

[0047] Embodiment 3 Based on Embodiment 1, this embodiment provides a computing device including a memory and one or more processors, the memory storing executable code, and the one or more processors implementing the steps of the end-to-end dynamic scene graph generation method in Embodiment 1 when executing the executable code.

[0048] The memory can be an internal storage unit of any data processing-capable device described in any of the preceding embodiments, such as a hard disk or a memory. The memory can also be an external storage device of any data processing-capable device, such as a plug-in hard disk, a smart storage card, an SD card, a flash memory card, and the like. Further, the memory can include both an internal storage unit and an external storage device of any data processing-capable device. The memory is used to store a computer program and other programs and data required by any data processing-capable device, and can also be used to temporarily store data that has been output or is to be output.

[0049] The above merely describes preferred embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, and the like within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An end-to-end dynamic scene graph generation method, characterized in that: The steps include: A deep convolutional neural network model is used to extract instance feature maps of each instance in the target video; Extracting fine-grained features of the instance feature map of each instance using a visual encoder to obtain a fine-grained feature map of each instance; Using a visual language pre-training model to predict instance description text corresponding to the fine-grained feature map of each instance, to obtain instance attribute information of each instance; Initialize a set of paired instance queries, use a multi-head cross-attention mechanism to aggregate the spatial features of the image to update the query, obtain paired instance representations, and regress the bounding box coordinates and category information of the paired instances; Predicting a dynamic visual relationship between paired instances based on the fine-grained feature maps of the paired instances to obtain dynamic visual relationship information of the paired instances; A dynamic scene graph is generated based on instance attribute information, bounding box coordinate information, category information and dynamic visual relationship information of paired instances.

2. The end-to-end dynamic scene graph generation method according to claim 1, characterized in that A deep convolutional neural network model is used to extract instance feature maps of each instance in the target video, including the following steps: Constructing the deep convolutional neural network model; Collect dynamic scene graphs to generate datasets; Preprocessing the data of the dynamic scene graph generation data set to obtain a pre-processed dynamic scene graph generation data set; Using the preprocessed dynamic scene graph to generate a data set to train the deep convolutional neural network model to obtain a feature extraction backbone network model; Segmenting the target video into sequence data of a plurality of video frames to obtain a video frame sequence data set; The video frame sequence data set is input into the feature extraction backbone network model to obtain the instance feature map of each instance.

3. The end-to-end dynamic scene graph generation method according to claim 2, characterized in that: The dynamic scene graph generation data set includes a plurality of video data; the pre-processed dynamic scene graph generation data set includes a plurality of video frame data sets, and the plurality of video frame data sets correspond one to one to the plurality of video data; Preprocessing the data of the dynamic scene graph generation data set to obtain a corresponding pre-processed dynamic scene graph generation data set specifically includes the following steps: Each of the video data in the dynamic scene graph generation data set is divided into a plurality of video frame time series data; When the amount of video frame timing data corresponding to the segmented video data is less than a preset video frame threshold, supplement the video frame timing data with the last frame of the plurality of video frame timing data so that the amount of the video frame timing data in the video frame data set is equal to the preset video frame threshold, thereby obtaining the video frame data set corresponding to the video data; When the number of video frame timing data corresponding to the segmented video data is equal to a preset video frame threshold, the video frame data set corresponding to the video data includes all the video frame timing data segmented from the video data; When the number of video frame time-series data corresponding to the segmented video data is greater than a preset video frame threshold, segmenting the video data into a plurality of time-series video segments in sequence, and segmenting each of the time-series video segments into the video frame time-series data having a number equal to the preset video frame threshold, to obtain the video frame data set corresponding to the video data; The video frame data in all the video frame data sets are normalized to obtain the pre-processed dynamic scene graph generation data set.

4. The end-to-end dynamic scene graph generation method according to claim 1, characterized in that The visual encoder is a DETR visual encoder, and the DETR visual encoder includes a convolutional layer and a transformer encoder; Extracting fine-grained features of the instance feature map of each instance using a visual encoder to obtain a fine-grained feature map of each instance includes the following steps: Using the convolutional layer to reduce the number of image channels of the instance feature map of each instance to obtain a low-channel instance feature map of each instance; The transformer encoder is used to perform feature extraction on the low-channel instance feature map of each instance to obtain the fine-grained feature map of each instance.

5. The end-to-end dynamic scene graph generation method according to claim 4, characterized in that: The transformer encoder includes a multi-head self-attention layer. The self-attention calculation formula of the multi-head self-attention layer in the encoder layer is as follows: ； in, represents the output value of the multi-head self-attention layer, represents the low-channel instance feature map, represents the position code, represents the number of image channels of the low-channel instance feature map, Represented as a transpose operation, Express Perform a transpose operation, Represents the instance feature map.

6. The end-to-end dynamic scene graph generation method according to claim 2, characterized in that: The visual language pre-training model is used to predict the instance description text corresponding to the fine-grained feature map of each instance to obtain the instance attribute information of each instance. The specific steps are as follows: Construct CLIP model; Generate a visual triplet dataset using data in the dynamic scene graph generation dataset; Generating text prompt information using the visual triplet dataset; Inputting the text prompt information into the text encoder of the CLIP model to generate text embedding to obtain the visual language pre-training model; The fine-grained feature map of each instance is input into the visual language pre-training model to obtain instance attribute information of each instance.

7. The end-to-end dynamic scene graph generation method according to claim 1, characterized in that: Initialize a set of paired instance queries, use the multi-head cross-attention mechanism to aggregate the spatial features of the image to update the query, obtain paired instance representations, and regress the bounding box coordinates and category information of the paired instances, including the following steps: Constructing the paired instance query dataset; Build an instance parsing decoder based on a multi-head cross-attention mechanism; Extracting paired representations between each instance from the paired instance query dataset according to the fine-grained feature map of each instance to obtain paired subject representations; The instance parsing decoder is used to predict the bounding box coordinate information and the category information of the paired instances in the paired subject representation according to the paired subject representation.

8. The end-to-end dynamic scene graph generation method according to claim 1, characterized in that: Predicting the dynamic visual relationship of the paired instances based on the fine-grained feature maps of the paired instances to obtain dynamic visual relationship information of the paired instances includes the following steps: Build a grouped predicate decoder; Using the grouped predicate decoder, the fine-grained feature maps of the paired instances are subjected to a temporal attention mechanism to extract temporal dependencies, thereby obtaining spatiotemporal context features of the paired instances; The grouped predicate decoder is used to predict the dynamic visual relationship between pairs of instances according to the spatiotemporal context features of the pairs of instances, thereby obtaining the dynamic visual relationship information of the pairs of instances.

9. An end-to-end dynamic scene graph generation system, characterized in that include: Feature extraction module: used to extract instance feature maps of each instance in the target video using a deep convolutional neural network model; Extracting fine-grained features of the instance feature map of each instance using a visual encoder to obtain a fine-grained feature map of each instance; Attribute recognition module: used to predict the instance description text corresponding to the fine-grained feature map of each instance using a visual language pre-training model to obtain instance attribute information of each instance; Paired Instance Recognition Module: This module initializes a set of paired instance queries, aggregates the spatial features of the image using a multi-head cross-attention mechanism to update the query, obtains paired instance representations, and regresses the bounding box coordinates and category information of the paired instances. A visual relationship recognition module is configured to predict the dynamic visual relationship between paired instances based on the fine-grained feature maps of the paired instances, thereby obtaining dynamic visual relationship information of the paired instances; Scene graph construction module: used to generate a dynamic scene graph based on instance attribute information, bounding box coordinate information, category information and dynamic visual relationship information of paired instances.

10. A computing device, characterized in that: The method comprises a memory and one or more processors, wherein the memory stores executable code, and when the one or more processors execute the executable code, the steps of the end-to-end dynamic scene graph generation method according to any one of claims 1 to 8 are implemented.

Images