Dynamic scene graph generation method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a dynamic scene graph generation method and device, electronic equipment, a storage medium and a program product, and the generation method comprises the steps: obtaining a video sequence which comprises a plurality of image frames arranged according to a time sequence; using the dynamic scene understanding network to process the plurality of image frames to obtain an object relationship feature of each image frame, the object relationship feature being used for representing an association relationship between different objects in the corresponding image frame; extracting an image frame with content representativeness from the plurality of image frames as an important frame; processing the object relation characteristics of the important frames by using a relation network to obtain relation information among different objects in the video sequence; and generating a dynamic scene graph of the video sequence according to the relation information among different objects in the video sequence. According to the method, the detection and modeling performance of rare objects and relationships can be improved, and the integrity and generalization ability of the generated dynamic scene graph can be improved.

Description

Technical Field

[0001] The present disclosure generally relates to the field of classification technology, and more specifically, to a method, device, electronic device, storage medium and program product for generating a dynamic scene graph. Background Art

[0002] A scene graph is a graph that represents objects in an image or video and the relationships between them. Scene graph generation methods can be roughly divided into two categories: static scene graph generation, that is, generating a scene graph from a single image; and dynamic scene graph generation, that is, generating a scene graph from a video. For static scene graph generation, popular object detectors such as Faster R-CNN and Mask R-CNN are generally used to extract objects, and then predict the relationships between objects based on visual and semantic features. Dynamic scene graph generation methods mainly capture temporal information through 3D convolutional models and transformers, mainly mining temporal structure information at the feature level, and modeling dynamic scene graph generation as a classification task.

[0003] One of the core challenges of dynamic scene graph generation is how to effectively integrate information in the spatiotemporal dimensions. Generating dynamic scene graphs requires not only the detection of objects in the scene, but also the effective modeling and reasoning of the semantic relationships between objects. Existing methods have limitations in the recognition and reasoning of object relationships, especially when faced with complex, dynamic, and long-distance object interactions, which are prone to semantic ambiguity or misrecognition, and lack the ability to accurately model and reason about fine-grained relationships. In video scenes, the distribution of objects and relationships exhibits a significant long-tail characteristic, that is, a few high-frequency objects and relationships are frequently observed, while a large number of low-frequency objects and rare relationships are difficult to fully learn. Existing technical solutions are difficult to effectively deal with this long-tail distribution, resulting in poor performance in the detection and modeling of rare objects and relationships, thereby affecting the integrity and generalization ability of dynamic scene graphs. Summary of the invention

[0004] The present disclosure provides a method, device, electronic device, storage medium and program product for generating a dynamic scene graph, which are used to solve at least one of the above problems.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for generating a dynamic scene graph is provided, the generating method comprising: acquiring a video sequence, wherein the video sequence comprises a plurality of image frames arranged in time sequence; using a dynamic scene understanding network to process the plurality of image frames to obtain object relationship features of each image frame, wherein the object relationship features are used to represent the association relationship between different objects in the corresponding image frames; extracting image frames representative of content from the plurality of image frames as important frames; using a relationship network to process the object relationship features of the important frames to obtain relationship information between different objects in the video sequence; and generating a dynamic scene graph of the video sequence based on the relationship information between different objects in the video sequence.

[0006] Optionally, the dynamic scene understanding network includes a space-time Transformer network, wherein the use of the dynamic scene understanding network to process the multiple image frames to obtain object relationship features of each image frame includes: performing object detection processing for each image frame, extracting visual features of each object in the image frame, and determining original object relationship features of the image frame based on the visual features; obtaining a temporal code of the video sequence, wherein the temporal code is used to represent the temporal order of each image frame in the video sequence; inputting the temporal code and the original object relationship features of the multiple image frames into the space-time Transformer network to obtain object relationship features of each image frame of the video sequence, wherein the temporal code is input into a temporal decoder of the space-time Transformer network.

[0007] Optionally, the use of a relational network to process object relation features of the important frame to obtain relation information between different objects in the video sequence includes: for any of the important frames, when there are multiple objects of the same category in the important frame, determining irrelevant objects from the multiple objects of the same category based on reference objects of corresponding categories in adjacent image frames of the important frame; removing features related to the irrelevant objects from the object relation features of the important frame to obtain revised object relation features of the important frame; and using a relational network to process revised object relation features of the important frame to obtain relation information between different objects in the video sequence.

[0008] Optionally, the relationship network includes a spatial encoder, a temporal encoder and a classifier, wherein the use of the relationship network to process the object relationship features of the important frame to obtain the relationship information between different objects in the video sequence includes: inputting the object relationship features of the important frame into the spatial encoder to obtain a first feature; inputting the object relationship features of the multiple image frames into the long-term encoder to obtain a second feature; inputting the object relationship features of the historical image frames of the important frame into the temporal encoder to obtain a third feature, the historical image frames are image frames that are earlier than the important frame in time sequence; inputting the first feature, the second feature and the third feature into the temporal encoder to obtain a fourth feature; inputting the fourth feature into the classifier to obtain the relationship information between different objects in the important frame.

[0009] Optionally, the relational network is trained by the following steps: adjusting the structure of the relational network to obtain the relational network to be trained, wherein the relational network to be trained includes a spatial encoder, a temporal encoder and a classifier connected in sequence, and the spatial encoders are at least two connected in parallel; pre-training the relational network to be trained by a supervised training method to obtain a pre-trained relational network, wherein each pre-training inputs object relationship features of at least two image frames in a sample video sequence into the relational network to be trained; adjusting the structure of the pre-trained relational network with reference to the structure of the relational network to obtain the relational network to be fine-tuned; fine-tuning the relational network to be fine-tuned by a supervised training method to obtain the relational network.

[0010] Optionally, the loss function used in the relational network training includes at least one of the following: time continuity loss, multi-label margin loss, and cross entropy loss.

[0011] According to a second aspect of an embodiment of the present disclosure, a device for generating a dynamic scene graph is provided, the generating device comprising: an acquisition unit, configured to acquire a video sequence, wherein the video sequence comprises a plurality of image frames arranged in time sequence; a first processing unit, configured to use a dynamic scene understanding network to process the plurality of image frames to obtain object relationship features of each image frame, wherein the object relationship features are used to represent the association relationship between different objects in the corresponding image frames; an extraction unit, configured to extract image frames representative of content from the plurality of image frames as important frames; a second processing unit, configured to use a relationship network to process the object relationship features of the important frames to obtain relationship information between different objects in the video sequence; and a generating unit, configured to generate a dynamic scene graph of the video sequence based on the relationship information between different objects in the video sequence.

[0012] Optionally, the dynamic scene understanding network includes a space-time Transformer network, and the first processing unit is further configured to: perform object detection processing for each image frame, extract visual features of each object in the image frame, and determine the original object relationship features of the image frame based on the visual features; obtain the timing code of the video sequence, wherein the timing code is used to represent the time order of each image frame in the video sequence; input the timing code and the original object relationship features of the multiple image frames into the space-time Transformer network to obtain the object relationship features of each image frame of the video sequence, wherein the timing code is input into the timing decoder of the space-time Transformer network.

[0013] Optionally, the second processing unit is further configured to: for any of the important frames, when there are multiple objects of the same category in the important frame, determine irrelevant objects from the multiple objects of the same category based on reference objects of corresponding categories in adjacent image frames of the important frame; remove features related to the irrelevant objects from the object relationship features of the important frame to obtain corrected object relationship features of the important frame; and use a relationship network to process the corrected object relationship features of the important frame to obtain relationship information between different objects in the video sequence.

[0014] Optionally, the relationship network includes a spatial encoder, a temporal encoder and a classifier, and the second processing unit is further configured to: input the object relationship feature of the important frame into the spatial encoder to obtain a first feature; input the object relationship features of the multiple image frames into the long-term encoder to obtain a second feature; input the object relationship features of the historical image frames of the important frame into the temporal encoder to obtain a third feature, the historical image frames are image frames that are earlier than the important frame in time sequence; input the first feature, the second feature and the third feature into the temporal encoder to obtain a fourth feature; input the fourth feature into the classifier to obtain relationship information between different objects in the important frame.

[0015] Optionally, the relational network is trained by the following steps: adjusting the structure of the relational network to obtain the relational network to be trained, wherein the relational network to be trained includes a spatial encoder, a temporal encoder and a classifier connected in sequence, and the spatial encoders are at least two connected in parallel; pre-training the relational network to be trained by a supervised training method to obtain a pre-trained relational network, wherein each pre-training inputs object relationship features of at least two image frames in a sample video sequence into the relational network to be trained; adjusting the structure of the pre-trained relational network with reference to the structure of the relational network to obtain the relational network to be fine-tuned; fine-tuning the relational network to be fine-tuned by a supervised training method to obtain the relational network.

[0016] Optionally, the loss function used in the relational network training includes at least one of the following: time continuity loss, multi-label margin loss, and cross entropy loss.

[0017] According to a third aspect of an embodiment of the present disclosure, there is provided an electronic device, comprising: at least one processor; and at least one memory storing computer executable instructions, wherein the computer executable instructions, when executed by the at least one processor, prompt the at least one processor to execute a method for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute a method for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure.

[0019] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, prompt the at least one processor to execute a method for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure.

[0020] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0021] According to the method and device for generating a dynamic scene graph, an electronic device, and a storage medium disclosed in the present invention, by first using an existing dynamic scene understanding network to extract object relationship features, the object relationship features of each image frame can fully include the association relationship between different image frames, but the amount of information is large and the focus is not clear. On this basis, by first extracting representative important frames, further processing only the important frames, and extracting the object relationship information therein, the extracted object and relationship information can be made more representative, thereby effectively coping with long-tail distribution, improving the detection and modeling performance of rare objects and relationships, and helping to improve the integrity and generalization ability of the generated dynamic scene graph.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0024] Figure 1 is a flow chart of a method for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure;

[0025] Figure 2 is a schematic diagram of a framework of an execution system of a method for generating a dynamic scene graph according to a specific embodiment of the present disclosure;

[0026] Figure 3 is a schematic diagram of a training architecture of a relationship network according to a specific embodiment of the present disclosure;

[0027] Figure 4 is a block diagram of a device for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure;

[0028] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the attached claims.

[0031] It should be noted that the phrase "at least one of the items" in the present disclosure includes three types of parallel situations: "any one of the items", "a combination of any number of the items", and "all of the items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. Another example is "executing at least one of step 1 and step 2" which means the following three parallel situations: (1) executing step 1; (2) executing step 2; (3) executing step 1 and step 2.

[0032] Hereinafter, a method, an apparatus, an electronic device, a storage medium, and a program product for generating a dynamic scene graph according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0033] Figure 1 is a flow chart of a method for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure. The method for generating a dynamic scene graph can be executed by an electronic device with sufficient computing power.

[0034] Reference Figure 1 In step S101, a video sequence is obtained, wherein the video sequence includes a plurality of image frames arranged in time sequence.

[0035] The purpose of the method of the exemplary embodiment of the present disclosure is to generate a dynamic scene graph for the video sequence obtained here. In other words, the video sequence obtained here is the processing object of the method of the exemplary embodiment of the present disclosure. This step can obtain the video sequence from the memory of the electronic device executing the method, or can obtain the video sequence from other electronic devices through a communication network.

[0036] In step S102, a dynamic scene understanding network is used to process multiple image frames to obtain object relationship features of each image frame.

[0037] The dynamic scene understanding network can be a network used in related technologies to capture the temporal information and / or spatial information of a video. The object relationship features obtained by the dynamic scene understanding network are used to represent the association relationship between different objects in the corresponding image frames (objects usually refer to concrete objects, such as people, cups, tables, etc.). They are detailed features obtained after a comprehensive analysis of the objects in each image frame. They not only reflect the relationship between the objects in each image frame, but also can refer to objects in other image frames, so that the object relationship features of a single image frame contain information in the time dimension.

[0038] In step S103, image frames with representative contents are extracted from the plurality of image frames as important frames.

[0039] Important frames are image frames in a video sequence that are informative, representative, or critical to the understanding of the overall content. Important frames are usually obtained through motion analysis, visual content detection, time sampling, or a model-based selection algorithm to identify and extract frames that are informative or critical. As an example, for a model-based selection algorithm, a CNN (Convolutional Neural Network) can be used to extract feature vectors from each image frame, and the similarity between different image frames can be calculated (for example, using cosine similarity) to form a similarity matrix, and then a cluster analysis is performed. Each cluster represents a group of similar image frames. From each cluster, a central frame or a representative frame is selected as an important frame according to the mean of the feature vector. Important frames can be selected based on the position of the image frame in the cluster or the amount of change between frames. After that, redundant frames can be removed to find representative image frames. The process can also use unsupervised indicators to evaluate the image frames as a reference for selecting important frames.

[0040] In step S104, the object relationship features of the important frames are processed using a relationship network to obtain relationship information between different objects in the video sequence.

[0041] On the one hand, using a relational network to further process the object relationship features of the image frames can complete information integration and extract the relationship information between different objects in the video sequence. On the other hand, the object relationship features of the important frames are processed specifically. Although the object relationship features of the multiple image frames obtained in step S102 contain comprehensive and detailed information, the amount of information is large, the focus is not clear, and there is a long-tail distribution. By screening these features and only retaining the object relationship features of the important frames extracted in step S103, the amount of data processed can be significantly reduced, and the data quality can be improved, so that the objects and their relationship information extracted thereby are more representative.

[0042] In step S105, a dynamic scene graph of the video sequence is generated according to the relationship information between different objects in the video sequence.

[0043] This step uses explicit object relationship information to achieve image generation and obtain a dynamic scene graph.

[0044] According to the method for generating a dynamic scene graph of an exemplary embodiment of the present disclosure, by first using an existing dynamic scene understanding network to extract object relationship features, the object relationship features of each image frame can fully include the association relationship between different image frames, but the amount of information is large and the focus is not clear. On this basis, by first extracting representative important frames, further processing only the important frames, and extracting the object relationship information therein, the extracted objects and relationship information can be made more representative, thereby effectively coping with long-tail distribution, improving the detection and modeling performance of rare objects and relationships, and helping to improve the integrity and generalization ability of the generated dynamic scene graph.

[0045] Next, a method for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure is further introduced.

[0046] Regarding step S102, optionally, the dynamic scene understanding network includes a space-time Transformer network. The space-time Transformer network effectively captures information in the time dimension by combining a space encoder and a time decoder through an encoder-decoder structure. Accordingly, step S102 includes: performing object detection processing for each image frame, extracting visual features of each object in the image frame, and determining the original object relationship features of the image frame based on the visual features; obtaining the temporal coding of the video sequence, wherein the temporal coding is used to represent the temporal order of each image frame in the video sequence; inputting the temporal coding and the original object relationship features of multiple image frames into the space-time Transformer network to obtain the object relationship features of each image frame of the video sequence, wherein the temporal coding is input into the temporal decoder of the space-time Transformer network. By first determining the original object relationship features of the image frame, the original relationship representation can be established for different objects in the image frame. By obtaining the temporal coding, the description of the temporal relationship between different image frames can be compensated, allowing the model to consider the temporal order of multiple image frames when processing the object relationship features, thereby enhancing the modeling of temporal dependencies. As an example, an object detection network (such as but not limited to Faster R-CNN) may be used when extracting visual features of various objects in an image frame.

[0047] Further optionally, the operation of determining the original object relationship features of the image frame based on the visual features includes: for each image frame, determining multiple object pairs from all objects in the image frame, determining the relationship features of the object pairs according to the visual features of two objects in the object pairs, and using the relationship features of the multiple object pairs as the original object relationship features of the image frame. Among them, t represents the time corresponding to the image frame, represents the relationship feature of the kth object pair, and K(t) represents the number of object pairs in the image frame. When determining the object pairs, for example, each object can be made to form an object pair with other objects, so as to achieve a complete object pair construction. In other words, if the number of objects identified in the image frame is N(t), then

[0048] As an example, when the spatial-temporal Transformer network processes temporal encoding and raw object relationship features of multiple image frames, it first converts the raw object relationship features of a single image frame into Input the spatial encoder, use the encoders with the same multi-layer structure and connected in sequence in the spatial encoder to iteratively calculate the original object relationship features of each image frame, and then summarize the calculation results of the spatial encoder for each image frame to obtain the intermediate object relationship features of multiple image frames, which are input into the temporal decoder together with the temporal code. The temporal decoder can use a sliding window to batch multiple image frames so that messages are transmitted between adjacent frames to avoid interference with distant frames. Due to the sliding window, the object relationship features in the same image frame have different representations in different batches. For example, for an image frame, the object relationship feature that appears earliest in the window can be selected as the output result of step S102.

[0049] Regarding step S104, optionally, step S104 includes: for any important frame, when there are multiple objects of the same category in the important frame, determining irrelevant objects from multiple objects of the same category according to reference objects of corresponding categories in adjacent image frames of the important frame; removing features related to irrelevant objects from object relationship features of the important frame to obtain corrected object relationship features of the important frame; using a relationship network to process the corrected object relationship features of the important frame to obtain relationship information between different objects in the video sequence. By determining objects that are irrelevant to the reference object as irrelevant objects based on reference objects of the same category, and cleaning up some features in the object relationship features of the important frame, specifically cleaning up features related to irrelevant objects, the amount of data processed can be further reduced, the interference of irrelevant objects can be reduced, the data quality can be improved, and the corrected object relationship features determined thereby are more representative, further improving the detection and modeling performance of rare objects and relationships, and helping to improve the integrity and generalization ability of the dynamic scene graph finally generated.

[0050] It should be understood that the reference object is a representative object in the image frame, and thus can be used as a reference for cleaning irrelevant objects. Specifically, in a video, objects of the same category usually maintain a certain consistency in consecutive frames. If the bounding box of a reference object is detected in a certain frame, it can usually be assumed that the same type of object will also exist in its previous frame and the next frame. Therefore, through this motion consistency, the model can use the information of the previous and next frames to infer and select, determine the object related to the reference object in the current image frame, and treat other objects of the same category in the current image frame as irrelevant objects. At the same time, the selection of objects related to the reference object in the previous and next frames can be based on contextual relationships. Even if there is no clear reference object in the adjacent image frames of the current image frame, the model can also make inferences based on the detected visual features and motion trends. As an example, in the training stage, the reference object can be an object with real annotations, or an object inferred based on the real annotated object and the contextual relationship. To this end, a sample data set with high annotation quality and good temporal continuity can be used during training to improve the reliability of the model in obtaining real annotations between different frames. In the reasoning stage, the reference object may be a manually annotated object or an object determined by other analysis methods, such as an object obtained by semantic analysis of an important frame, which is not limited in the present disclosure.

[0051] It should be noted that step S104 uses the object relationship features of important frames, that is, feature cleaning is performed based on important frames, which can be called a frame enhancement strategy. For the object relationship features of important frames, the features of objects with higher correlation are further screened based on the reference objects, which can be called an irrelevant object cleaning strategy. Both strategies are based on prior rules to streamline the features of image frames, which can be collectively referred to as spatiotemporal prior reasoning mechanisms. When processing spatiotemporal dynamic relationships in videos, the prior art usually only focuses on the independent features between static frames, but fails to fully model the dynamic changes of objects and their relationships in the time dimension. The present disclosure introduces a spatiotemporal prior reasoning mechanism to effectively capture the changes in object relationships in videos over time and establish a more accurate spatiotemporal model. The subsequent relationship network can not only process the object relationship of the current image frame, but also combine the information of historical image frames to predict the changes in object relationships in future image frames, thereby realizing spatiotemporal consistency modeling over a long span of time, and can refer to prior knowledge when reasoning about object relationships, thereby enhancing the rationality and versatility of reasoning.

[0052] Regarding the relationship network used in step S104, optionally, the relationship network includes a spatial encoder (which may be the same structure as the spatial encoder in the aforementioned space-time Transformer network, or a different structure, which is not limited by the present disclosure), a temporal encoder, and a classifier. At this time, step S104 includes: inputting the object relationship feature of the important frame into the spatial encoder to obtain a first feature; inputting the object relationship features of multiple image frames into a long-term encoder (which may be an encoder obtained by separate training in advance, or a temporal encoder may be used, which is not limited by the present disclosure) to obtain a second feature; inputting the object relationship features of the historical image frames of the important frame (the historical image frames are specifically image frames that are earlier than the important frame in time and are adjacent to the important frame, and only at least two historical image frames may be used, or all historical image frames of the important frame may be used) into the temporal encoder to obtain a third feature; inputting the first feature, the second feature, and the third feature into the temporal encoder to obtain a fourth feature; inputting the fourth feature into the classifier to obtain the relationship information between different objects in the important frame. The relational network is used to process each important frame one by one. The first feature (representing the spatial information of the current important frame) obtained by extracting spatial information from the current important frame, the second feature (representing the global temporal information) obtained by extracting temporal information from all image frames, and the third feature (representing the historical information of the current important frame) obtained by extracting temporal information from at least two historical image frames are jointly input into the temporal encoder. This can take into account both the pertinence and comprehensiveness of the input information, and improve the accuracy and completeness of the object relationship information ultimately obtained.

[0053] Further optionally, the relational network is trained by the following steps: adjusting the structure of the relational network to obtain the relational network to be trained, wherein the relational network to be trained includes a spatial encoder, a temporal encoder and a classifier connected in sequence, and the spatial encoders are at least two connected in parallel; pre-training the relational network to be trained by a supervised training method to obtain a pre-trained relational network, wherein each pre-training inputs object relationship features of at least two image frames in a sample video sequence into the relational network to be trained; adjusting the structure of the pre-trained relational network with reference to the structure of the relational network to obtain the relational network to be fine-tuned; and fine-tuning the relational network to be fine-tuned by a supervised training method.

[0054] Generally speaking, the training of the relational network includes two stages: pre-training and fine-tuning. That is, the network structure is first adjusted to obtain the relational network to be trained, and pre-training is performed on it. Then, the structure of the pre-trained relational network is adjusted and the parameters are fine-tuned to obtain the relational network.

[0055] In the pre-training stage, unlike the inference stage where image frames (specifically important frames) are processed one by one, this is equivalent to processing the object relationship features of at least two historical image frames each time, extracting the third feature and inputting it into the temporal encoder, but not inputting the first feature and the second feature, so that object relationship extraction can be performed only for at least two input image frames in the pre-training stage, thereby improving the training efficiency of the pre-training stage. It should be understood that at this time, the number of spatial encoders can be consistent with the number of input image frames, so that each spatial encoder is used to process the object relationship features of an image frame, ensuring that the processing object of a spatial encoder is always the object relationship features of an image frame, which is consistent with the inference stage and the fine-tuning stage, and each spatial encoder is exactly the same, that is, during pre-training, it is necessary to keep the same update on the parameters of each spatial encoder.

[0056] In the fine-tuning stage, the network structure is readjusted so that the relationship network to be fine-tuned can input image frames one by one according to the operations in the reasoning stage, and input the first feature, the second feature and the third feature into the temporal encoder, maintaining consistency with the reasoning stage.

[0057] Optionally, the loss function used in relational network training includes at least one of the following: temporal continuity loss, multi-label margin loss, and cross entropy loss. Temporal continuity loss can ensure consistency in the analysis of object relationships in image frames at different times. Multi-label margin loss takes into account that object relationships can be described by multiple predicates. For example, a person-object relationship can be described by multiple predicates as "person-holding-bottle" and "person-grasping-bottle". Accordingly, multiple classifiers can be used to infer different types of relationships. At this time, using multi-label margin loss can achieve coordinated training of different classifiers. Cross entropy loss, as a commonly used loss function for classification models, can enhance the learning ability of relational models for object relationships. The coordinated use of these three loss functions can improve the training effect of relational networks.

[0058] As an example, the temporal continuity loss L d It can be calculated by a distance metric (such as Euclidean distance) between two objects in an image frame, as follows:

[0059]

[0060] Among them, N f is the number of image frames currently being processed, Indicates the current processing N f The object distance in the i-th image frame among the image frames, represents the corresponding object distance in the related frame of the i-th image frame, and compares it with By comparing, the temporal continuity can be evaluated. For the current i-th frame, for example, the annotations of the i-1th and i+1th frames can be used as references.

[0061] Multi-label margin loss L p The formula is, for example:

[0062]

[0063] For an object pair (specifically a subject-object pair) r, p + is a set of labeled positive samples, which includes multiple predicates and annotates several correct predicates. - is the negative sample set, which includes multiple wrong predicates. is the confidence score of the p-th predicate. These scores reflect the model's confidence in each predicate in the current context. Similarly, is the confidence score of the qth predicate. The above multi-label margin loss L p The specific representation is based on p + and q - Construct multiple positive and negative sample pairs and calculate for each sample pair Then the max values ​​of all positive and negative sample pairs are summed up as the multi-label marginal loss L p The relational network predicts object relations by calculating the confidence scores of multiple predicates of a subject-object pair r, and then taking the predicates with scores exceeding the threshold as the prediction results. This ensures the validity of the prediction and avoids misjudgments with low confidence. This structural design aims to enhance the model's ability to capture fine-grained relations in spatiotemporal scenarios and ensure the accuracy of predictions.

[0064] As an example, the total loss function (spatiotemporal prior continuous loss function) used in relational network training is:

[0065] L total =L p +L d +L o .

[0066] Among them, L o is the standard cross entropy loss.

[0067] As an example, the same loss function may be used in the pre-training stage and the fine-tuning stage to ensure consistency of the training objectives. Of course, different loss functions may also be used as needed, and the present disclosure does not impose any limitation on this.

[0068] Next, a method for generating a dynamic scene graph according to a specific embodiment of the present disclosure is introduced.

[0069] Reference Figure 2The system for executing this specific embodiment mainly includes a video sequence processing module, a space-time Transformer architecture, a space-time prior reasoning module (also known as a frame enhancement and object cleaning module) and a relationship network module.

[0070] Firstly, the video sequence processing module is used to extract image frames from the video sequence to obtain input data.

[0071] Then, the spatial-temporal Transformer architecture is used to process the spatiotemporal information in the video sequence, using a multi-head self-attention mechanism to model objects and relationships in the image frames of the video sequence.

[0072] In addition, the spatiotemporal prior reasoning module introduces a spatiotemporal prior reasoning mechanism to capture the dynamic changes of object relationships in videos. By modeling on the temporal sequence, the model is able to predict the evolution of objects and relationships between different image frames, thereby enhancing the understanding of the temporal dependence of video content.

[0073] Finally, the relation network module improves its ability to predict object relations in future image frames by pre-training and fine-tuning the relation network. During the pre-training stage, the relation network can learn spatiotemporal features on large-scale unlabeled video data, which enables it to better adapt to downstream tasks, thereby improving overall prediction accuracy and generalization ability.

[0074] Specifically, the operations of the space-time Transformer architecture to process the spatiotemporal information in video sequences mainly include relational representation and the processing of the space-time Transformer network.

[0075] Relationship representation:

[0076] Faster R-CNN is used to extract the visual features of each object in image frame t. And extract the bounding box of each object And the object category distribution of the target candidate box Represents the category of each extracted object and the relationship between the i-th and j-th target candidate boxes Contains visual appearance, spatial information, and semantic embedding and can be expressed as:

[0077]

[0078] Where <,> represents the concat operation. Indicates a flattening operation. represents element-by-element addition, f s 、f o Each of them is implemented by a fully connected layer to map the 2048-dimensional vector to a 512-dimensional vector, f uIt is implemented by a fully connected layer, which maps a 12544-dimensional vector to a 512-dimensional vector. The feature map of the corresponding joint box generated for RoIAlign combines the features of the two target candidate boxes, so that the subsequent decoder can better understand the interaction between the two. Through this design, the model's understanding of the relationship between objects can be improved, thereby enhancing the overall performance of object detection. box In order to convert the bounding boxes of the subject and object into a function that can be used for feature fusion or further calculation, this conversion usually involves mapping the information of the bounding box (such as coordinates, size) to a specific feature space to facilitate combination with other features. Determined by the object categories of the subject and the object, it is obtained by directly processing the object category text of the subject and the object. The 200-dimensional semantic embedding vector can capture the information of the object category to a certain extent. If the task requires richer feature representation, the dimension of the semantic embedding vector can also be increased. Ability to exchange spatial and temporal information in space-time Transformer networks.

[0079] Processing of the spatial-temporal Transformer network:

[0080] The spatial-temporal Transformer network maintains the original encoder-decoder architecture. The difference is that the encoder and decoder are given more specific tasks. The spatial encoder input is a single The query Q, key K and value V share the same input, and the output of the n-th layer encoder is represented as:

[0081]

[0082] The encoder consists of N identical Att enc. The encoder consists of layers, which are stacked in sequence, and the input of the (n)th layer is equal to the output of the (n-1)th layer. Among them, the spatial information hidden in the relational representation plays a vital role in the self-attention mechanism. The final output of the encoder stack is sent to the temporal decoder.

[0083] Introducing frame encoding as temporal encoding in the temporal decoder. In the absence of convolution and recurrence, by customizing the frame encoding as sequence order knowledge, it is possible to t The temporal position is injected into the frame, so that the temporal information is explicitly conveyed through the frame encoding without convolution and recurrence, helping the model to better capture the temporal dynamics. In addition, since the number of embedding vectors in the temporal decoder that depends on the window size η is fixed and relatively short, the frame encoding is constructed using the learned embedding parameters: E f =[e1,…,eη ], where e1,…,e η ∈R 1936 , is the same as The learned vectors of the same length can be used to train the frame encoding while training the model to achieve its learning.

[0084] The temporal decoder captures the temporal dependencies between image frames, which not only requires a lot of computation and memory consumption, but also the useful information is easily overwhelmed by a large number of irrelevant representations. By using a sliding window to batch image frames, messages are passed between adjacent frames to avoid interference with distant frames. Since the frame encoding provides sequence order knowledge, the frame encoding used in each batch remains unchanged.

[0085] Self-attention layer of temporal decoder and spatial encoder Att enc. () is the same, that is, the masked multi-head self-attention layer is removed. The sliding window of size η represents the sequence [X1,…,X T ], the i-th generated input batch is represented as:

[0086] Z i =[X i ,…,X i+η-1 ],i∈{1,…,T-η+1}

[0087] where the window size η≤T, and T is the length of the video. The decoder consists of N layers of identical self-attention layers Att stacked together. dec. () is composed of a structure similar to that of an encoder. Consider the first layer:

[0088] Q=K=Z i +E f ,

[0089] V=Z i ,

[0090]

[0091] The same encoding is added to the relation representation in the same frame as the query and key, and the final prediction uses the output from the last decoder layer. Due to the sliding window, the relations in the same frame have different representations in different batches. Here, the earliest representation in the window is selected, which can more effectively capture temporal dependencies and reduce redundant information, thereby improving performance in video understanding or time series prediction tasks. This strategy helps ensure that the model can dynamically adapt to changing contexts and improve the accuracy of its predictions.

[0092] It should be noted that Figure 2The spatial-temporal Transformer network in has a noisy video inference part, which means that the influence of noise can be considered in the inference stage. This part is performed during network training and does not directly affect the structure of the encoder or decoder.

[0093] The operations of the spatiotemporal prior reasoning module mainly include frame enhancement and object cleaning.

[0094] Regarding frame enhancement, a temporal prior frame enhancement strategy is designed to automatically obtain more credible samples as important frames in the video. Specifically, important frames are selected through an unsupervised method and task-related object information is ensured in these frames, thereby improving the performance and accuracy of the model in time series tasks. This strategy enables the model to effectively capture valuable temporal information without manual annotation, allowing the model to capture fine-grained temporal continuity. Frame enhancement is performed based on the results of detected unannotated frames without the need for additional manual annotation.

[0095] At the same time, object cleaning is required, that is, deleting irrelevant objects. If multiple objects of the same category are detected in a certain frame, irrelevant objects need to be removed, and objects related to the true annotated objects in the previous and next frames need to be retained. Specifically, the bounding boxes of objects of the same category in the true annotations of the previous frame and the next frame of the current frame are used as references, and the objects most related to them are selected and retained, and other objects are cleaned up as irrelevant objects. Where (x i ,y i ) is the center point of the bounding box, (w i ,h i ) represents the width and height of the bounding box. i∈{1,2,…,N V}, where N V is the number of objects of the same category detected in the important frame. The object bounding box in the current frame The same category labeled object B in the previous frame p =[x p ,y p ,w p ,h p The correlation between ] is calculated by the following formula:

[0096]

[0097] The smaller the value, the stronger the correlation between objects. If the annotation of the previous frame does not contain the category of the i-th detected object, then is 0, and the sequence Similarly, the correlation for the next frame is Use calculation formula To represent the correlation between the i-th detected object and the previous and next frames, where i∈{1,2,…,N v The u-th object with the smallest value in the sequence is selected as the final result, indicating the object most relevant to the previous and next frames. It is retained, and other objects are removed as irrelevant objects. Specifically, the relationship representation X of the important frame is removed. t Elements in the object that are not related to the object.

[0098] The relational network used in the relational network module is used to realize object relationship prediction (because it is used to generate dynamic scene graphs, it can also be called dynamic relationship prediction). It needs to be pre-trained and fine-tuned first. Figure 3 , the strategies for pre-training and fine-tuning are as follows:

[0099] Instead of using only one classifier, multiple linear classifiers are used to predict different types of object relations. t,ij Represents object O t,i and O t,j The relationship category distribution between , the prediction formula in the pre-training stage is Where t represents the time index of the current image frame, which refers to the position of the image frame being processed in the entire video sequence. l represents the level or number of layers of feature representation. Usually it is obtained after multi-layer network processing about object O t,i and O t,j The features can help the model extract information related to relationship prediction at different levels, y t,ij Represents the predicted object category. Since there may be multiple correct relationships between two objects in practice, such as "a person touches food" and "a person eats food", a multi-label edge loss is used in pre-training, and its formula is as follows:

[0100]

[0101] where Y + represents the set of predicates containing the true labels, Y - represents the set of unlabeled negative predicates, and is the prediction confidence score of the p-th predicate, is the prediction confidence score of the qth predicate.

[0102] During fine-tuning, the spatial encoder mentioned above is reused to capture the spatial information of the current frame. Then a global temporal encoder is used to capture temporal correlations based on the output of the long-term encoder (the input of the long-term encoder is the object relationship features of multiple image frames of the video sequence output by the spatial-temporal Transformer network). This process may involve aggregating information at different time points in the sequence to better understand the temporal relationship between objects and actions. The global temporal encoder shares parameters with the progressive temporal encoder in the pre-training phase (which can also be called a short-term encoder relative to the long-term encoder). Both are temporal encoders in the relational network introduced in the previous article, but they receive different input information and can focus on different time scales, thereby jointly improving the performance of the model in spatiotemporal tasks. Therefore, different names are used here to distinguish their functions. The formula is as follows:

[0103]

[0104] The subscript g indicates that this feature is generated by the global temporal encoder and is used to capture temporal correlations over a long time range. When the pre-trained relational network is used to process the object relational features of the t-2th image frame and the t-1th image frame, the lth layer feature output by the pre-trained progressive temporal encoder corresponds to Figure 3 The global temporal encoder in the middle is input from above. t,ij Represents time-related coding, which can be output by a long-term encoder, corresponding to Figure 3 The global timing encoder in Z comes from the input below. f It is a frame encoding, mainly used for global temporal encoders, focusing on the specific features and temporal context of the current frame, such as the feature representation generated from the spatial features of the current frame, which can be output by the spatial encoder, corresponding to Figure 3 The global temporal encoder in the figure is input from the left, so it is similar to the frame encoding E f The final output of the global temporal encoder is expressed as Similar to pre-training, multiple linear classifiers are also used in fine-tuning:

[0105]

[0106] in, yes The last element in the fine-tuning process uses the same loss function as the pre-training. In the inference phase, only the classifiers in the fine-tuning phase are used. fin , while the classifiers in the pre-training stage are pre It will no longer be used to obtain the predicted object relationships and then generate a dynamic scene graph based on them.

[0107] Existing technologies for dynamic scene graph generation include: a method based on weak supervision, in which the target video is input into a trained weakly supervised dynamic scene graph generation model to obtain a dynamic scene graph corresponding to the target video. A method based on temporal transition regularization, in which a transformation matrix is ​​created based on the category and context of the image to capture statistical correlations based on each category and fine-grained correlations based on each context image, and these correlations are explicitly incorporated into the current VidSGG algorithm to provide stronger and more fine-grained regularization. A method based on a tracking algorithm, in which an object element tracking method is used to obtain element information in each frame, and then the scene relationship between the object elements in each frame is fitted in conjunction with the temporal context information, and the corresponding scene graph is generated based on the scene relationship.

[0108] The specific embodiments of the present disclosure are mainly to realize the generation of dynamic scene graphs from video scenes, extract key objects and their relationships in video sequences, so that the generated dynamic scene graphs can accurately express the video content, improve the efficiency of video understanding and analysis, and provide more comprehensive data support for video-related applications.

[0109] Compared with weak supervision-based methods, weak supervision methods rely on pre-trained weak supervision models. Although such methods do not require a large amount of labeled data, they fail to effectively combine domain knowledge or prior information in the reasoning process, resulting in the model lacking sufficient semantic information support when facing complex scenes. The method of this specific embodiment solves the problem of inaccurate reasoning caused by lack of knowledge in existing methods by introducing prior knowledge into the model, and significantly improves the model's reasoning ability and understanding of object relationships in complex scenes.

[0110] Compared with the method based on temporal transition regularization, it captures the statistical correlation between categories and contexts through an explicit transition matrix, and regularizes the temporal transition. Although this method captures some temporal correlations at a fine-grained level, it relies on a fixed statistical transition matrix and lacks flexibility, especially in dealing with dynamic changes. The method of this specific embodiment introduces a temporal prior reasoning mechanism, which predicts the dynamic changes of object relationships by analyzing the temporal patterns in the video sequence, and further improves the prediction ability of future frame relationships through a pre-trained model. This method not only takes into account the dependency between the current frame and the historical frames, but also predicts the evolution of the relationship between future frames, and has stronger temporal modeling capabilities.

[0111] Compared with the method based on tracking algorithm, this method obtains scene relationships by tracking object elements. It relies on tracking objects frame by frame, resulting in that the temporal information can only be captured locally, the modeling of spatiotemporal information is relatively limited, and it is easy to have spatiotemporal context breaks, and it is impossible to fully utilize the dynamic changes of objects in the video. The method of this specific embodiment introduces a space-time Transformer network, which can perform global modeling in the time and space dimensions of the entire video sequence at the same time through a multi-head self-attention mechanism, and can capture richer spatiotemporal information between image frames, ensuring that the changes of objects and their relationships at different times are accurately modeled, thereby improving the spatiotemporal consistency of dynamic scene graph generation.

[0112] Figure 4 is a block diagram of a device for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure. Figure 4 The generating device 400 includes an acquiring unit 401, a first processing unit 402, an extracting unit 403, a second processing unit 404, and a generating unit 405.

[0113] The acquisition unit 401 may acquire a video sequence, wherein the video sequence includes a plurality of image frames arranged in time sequence.

[0114] The first processing unit 402 may use a dynamic scene understanding network to process multiple image frames to obtain object relationship features of each image frame, wherein the object relationship features are used to represent the association relationship between different objects in the corresponding image frames.

[0115] The extraction unit 403 may extract image frames having representative contents from the plurality of image frames as important frames.

[0116] The second processing unit 404 may use a relational network to process object relational features of important frames to obtain relational information between different objects in the video sequence.

[0117] The generating unit 405 may generate a dynamic scene graph of the video sequence according to the relationship information between different objects in the video sequence.

[0118] Optionally, the dynamic scene understanding network includes a space-time Transformer network, and the first processing unit 402 can also: perform object detection processing for each image frame, extract visual features of each object in the image frame, and determine the original object relationship features of the image frame based on the visual features; obtain the timing code of the video sequence, wherein the timing code is used to represent the time order of each image frame in the video sequence; input the timing code and the original object relationship features of multiple image frames into the space-time Transformer network to obtain the object relationship features of each image frame of the video sequence, wherein the timing code is input into the timing decoder of the space-time Transformer network.

[0119] Optionally, the second processing unit 404 may also: for any important frame, when there are multiple objects of the same category in the important frame, determine irrelevant objects from the multiple objects of the same category based on reference objects of corresponding categories in adjacent image frames of the important frame; remove features related to irrelevant objects from object relationship features of the important frame to obtain corrected object relationship features of the important frame; use a relationship network to process the corrected object relationship features of the important frame to obtain relationship information between different objects in the video sequence.

[0120] Optionally, the relationship network includes a spatial encoder, a temporal encoder and a classifier, and the second processing unit 404 can also: input the object relationship features of the important frame into the spatial encoder to obtain a first feature; input the object relationship features of multiple image frames into the long-term encoder to obtain a second feature; input the object relationship features of the historical image frames of the important frame into the temporal encoder to obtain a third feature, the historical image frames are image frames that are earlier than the important frames in time; input the first feature, the second feature and the third feature into the temporal encoder to obtain a fourth feature; input the fourth feature into the classifier to obtain the relationship information between different objects in the important frame.

[0121] Optionally, the relational network is trained by the following steps: adjusting the structure of the relational network to obtain the relational network to be trained, wherein the relational network to be trained includes a spatial encoder, a temporal encoder and a classifier connected in sequence, and the spatial encoders are at least two connected in parallel; pre-training the relational network to be trained by a supervised training method to obtain a pre-trained relational network, wherein each pre-training inputs object relationship features of at least two image frames in a sample video sequence into the relational network to be trained; adjusting the structure of the pre-trained relational network with reference to the structure of the relational network to obtain the relational network to be fine-tuned; fine-tuning the relational network to be fine-tuned by a supervised training method to obtain the relational network.

[0122] Optionally, the loss function used in relational network training includes at least one of the following: temporal continuity loss, multi-label margin loss, and cross entropy loss.

[0123] According to an embodiment of the present disclosure, an electronic device may be provided. Figure 5 It is a block diagram of an electronic device according to an embodiment of the present disclosure, the electronic device 500 includes at least one memory 501 and at least one processor 502, the at least one memory stores a computer executable instruction set 5011 and an operating system 5012, when the computer executable instruction set 5011 is executed by the at least one processor 502, the method for generating a dynamic scene graph according to the embodiment of the present disclosure is executed.

[0124] As an example, the electronic device 500 may be a PC, a tablet device, a personal digital assistant, a smart phone, or other device capable of executing the above-mentioned instruction set. Here, the electronic device 500 is not necessarily a single electronic device, but may also be any device or circuit collection capable of executing the above-mentioned instructions (or instruction sets) individually or in combination. The electronic device 500 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device interconnected with a local or remote (e.g., via wireless transmission) interface.

[0125] In the electronic device 500, the processor 502 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not limitation, the processor 502 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0126] The processor 502 can execute instructions or codes stored in the memory, wherein the memory 501 can also store data. Instructions and data can also be sent and received over the network via the network interface device, wherein the network interface device can use any known transmission protocol.

[0127] The memory 501 may be integrated with the processor 502, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. In addition, the memory 501 may include a separate device, such as an external disk drive, a storage array, or any other storage device that can be used by a database system. The memory 501 and the processor 502 may be operationally coupled, or may communicate with each other, such as through an I / O port, a network connection, etc., so that the processor 502 can read files stored in the memory 501.

[0128] In addition, the electronic device 500 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device may be connected to each other via a bus and / or a network.

[0129] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute the method for generating a dynamic scene graph of the embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0130] According to an embodiment of the present disclosure, a computer program product is provided, including computer instructions, which implement the method for generating a dynamic scene graph of the embodiment of the present disclosure when the computer instructions are executed by a processor.

[0131] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

[0132] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for generating a dynamic scene graph, characterized in that: The generation method comprises: Acquire a video sequence, wherein the video sequence includes a plurality of image frames arranged in time sequence; Using a dynamic scene understanding network, processing the multiple image frames to obtain object relationship features of each image frame, wherein the object relationship features are used to represent association relationships between different objects in the corresponding image frames; Extracting image frames with representative contents from the multiple image frames as important frames; Using a relational network, processing the object relational features of the important frames to obtain relational information between different objects in the video sequence; A dynamic scene graph of the video sequence is generated according to the relationship information between different objects in the video sequence.

2. The generation method according to claim 1, characterized in that: The dynamic scene understanding network includes a space-time Transformer network, wherein the use of the dynamic scene understanding network to process the multiple image frames to obtain the object relationship feature of each image frame includes: For each image frame, performing object detection processing to extract visual features of each object in the image frame, and determining original object relationship features of the image frame based on the visual features; Acquire a time sequence code of the video sequence, wherein the time sequence code is used to represent a time sequence of each image frame in the video sequence; The temporal code and the original object relationship features of the multiple image frames are input into the space-time Transformer network to obtain the object relationship features of each image frame of the video sequence, wherein the temporal code is input into the temporal decoder of the space-time Transformer network.

3. The generation method according to claim 1, characterized in that: The using relationship network to process the object relationship features of the important frames to obtain the relationship information between different objects in the video sequence includes: For any of the important frames, when there are multiple objects of the same category in the important frame, determining irrelevant objects from the multiple objects of the same category according to reference objects of corresponding categories in image frames adjacent to the important frame; From the object relationship features of the important frame, removing features related to the irrelevant object to obtain a modified object relationship feature of the important frame; The modified object relationship features of the important frames are processed using a relationship network to obtain relationship information between different objects in the video sequence.

4. The generation method according to any one of claims 1 to 3, characterized in that: The relationship network includes a spatial encoder, a temporal encoder and a classifier, wherein the use of the relationship network to process the object relationship features of the important frames to obtain the relationship information between different objects in the video sequence includes: Inputting the object relationship feature of the important frame into the spatial encoder to obtain a first feature; Inputting the object relationship features of the plurality of image frames into a long-term encoder to obtain a second feature; Inputting the object relationship feature of the historical image frame of the important frame into the temporal encoder to obtain a third feature, wherein the historical image frame is an image frame that is earlier than the important frame in time sequence; Inputting the first feature, the second feature and the third feature into the time series encoder to obtain a fourth feature; The fourth feature is input into the classifier to obtain the relationship information between different objects in the important frame.

5. The generation method according to claim 4, characterized in that: The relationship network is trained by the following steps: Adjusting the structure of the relationship network to obtain a relationship network to be trained, wherein the relationship network to be trained includes a spatial encoder, a temporal encoder and a classifier connected in sequence, and the spatial encoders are at least two connected in parallel; Pre-training the to-be-trained relational network by a supervised training method to obtain a pre-trained relational network, wherein each pre-training step inputs object relational features of at least two image frames in a sample video sequence into the to-be-trained relational network; Referring to the structure of the relationship network, adjusting the structure of the pre-trained relationship network to obtain a relationship network to be fine-tuned; The relationship network to be fine-tuned is fine-tuned by a supervised training method to obtain the relationship network.

6. The generation method according to claim 5, characterized in that: The loss function used in the relational network training includes at least one of the following: time continuity loss, multi-label margin loss, and cross entropy loss.

7. A device for generating a dynamic scene graph, characterized in that: The generating device comprises: An acquisition unit is configured to acquire a video sequence, wherein the video sequence includes a plurality of image frames arranged in time sequence; A first processing unit is configured to process the multiple image frames using a dynamic scene understanding network to obtain an object relationship feature of each image frame, wherein the object relationship feature is used to represent an association relationship between different objects in the corresponding image frame; an extracting unit, configured to extract image frames with representative contents from the plurality of image frames as important frames; A second processing unit is configured to process the object relationship features of the important frame using a relationship network to obtain relationship information between different objects in the video sequence; The generating unit is configured to generate a dynamic scene graph of the video sequence according to the relationship information between different objects in the video sequence.

8. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer executable instructions, Wherein, when the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the method for generating a dynamic scene graph as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: When the instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to perform the method for generating a dynamic scene graph according to any one of claims 1 to 6.

10. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by at least one processor, the at least one processor is prompted to execute the method for generating a dynamic scene graph according to any one of claims 1 to 6.

Images