Method, apparatus, electronic device, storage medium, and program product for generating dynamic scene graph

Abstract

The present disclosure provides a method, apparatus, electronic device, storage medium, and program product for generating a dynamic scene graph. The generation method includes: obtaining a video sequence, where the video sequence includes a plurality of image frames arranged in chronological order; using a dynamic scene understanding network to process the plurality of image frames to obtain object relationship features for each image frame, where the object relationship features are used to represent the association relationships between different objects in the corresponding image frame; extracting content-representative image frames from the plurality of image frames as important frames; using a relationship network to process the object relationship features of the important frames to obtain relationship information between different objects in the video sequence; and generating a dynamic scene graph of the video sequence according to the relationship information between different objects in the video sequence. This method can improve the detection and modeling performance of rare objects and relationships, and contribute to improving the integrity and generalization ability of the generated dynamic scene graph.

Description

Technical Field

[0001] The present disclosure generally relates to the field of classification technology, and more specifically, to a method, apparatus, electronic device, storage medium, and program product for generating a dynamic scene graph. Background Art

[0002] A scene graph is a graph that represents the objects in an image or video and the relationships between them. Scene graph generation methods can be roughly divided into two categories: static scene graph generation, that is, generating a scene graph from a single image; and dynamic scene graph generation, that is, generating a scene graph from a video. For the generation of static scene graphs, popular object detectors such as Faster R-CNN and Mask R-CNN are generally used to extract objects, and then the relationships between objects are predicted based on visual and semantic features. Dynamic scene graph generation methods mainly capture temporal information through 3D convolutional models and transformers, mainly mine temporal structure information at the feature level, and model dynamic scene graph generation as a classification task.

[0003] One of the core challenges in dynamic scene graph generation is how to effectively fuse information in the spatio-temporal dimension. Generating a dynamic scene graph not only requires detecting the objects in the scene, but also requires effectively modeling and reasoning about the semantic relationships between the objects. Existing methods have limitations in the recognition and reasoning of object relationships. Especially when facing complex, dynamic, and long-distance object interactions, semantic ambiguity or misrecognition is likely to occur, and there is a lack of accurate modeling and reasoning capabilities for fine-grained relationships. In a video scene, the distribution of objects and relationships exhibits a significant long-tail characteristic, that is, a small number of high-frequency objects and relationships are frequently observed, while a large number of low-frequency objects and rare relationships are difficult to be fully learned. Existing technical solutions are difficult to effectively cope with this long-tail distribution, resulting in poor performance in the detection and modeling of rare objects and relationships, thus affecting the integrity and generalization ability of the dynamic scene graph. Summary of the Invention

[0004] The present disclosure provides a method, apparatus, electronic device, storage medium, and program product for generating a dynamic scene graph to solve at least one of the above problems.

[0005] According to a first aspect of the embodiments of the present disclosure, a method for generating a dynamic scene graph is provided. The generation method includes: obtaining a video sequence, where the video sequence includes a plurality of image frames arranged in time sequence; using a dynamic scene understanding network to process the plurality of image frames to obtain object relationship features of each image frame, where the object relationship features are used to represent the association relationships between different objects in the corresponding image frame; extracting content-representative image frames from the plurality of image frames as important frames; using a relationship network to process the object relationship features of the important frames to obtain relationship information between different objects in the video sequence; and generating a dynamic scene graph of the video sequence according to the relationship information between different objects in the video sequence.

[0006] Optionally, the dynamic scene understanding network includes a spatio-temporal Transformer network. Wherein, the step of using the dynamic scene understanding network to process the plurality of image frames to obtain object relationship features of each image frame includes: for each image frame, performing object detection processing to extract visual features of each object in the image frame, and based on the visual features, determining the original object relationship features of the image frame; obtaining a temporal encoding of the video sequence, where the temporal encoding is used to represent the time order of each image frame in the video sequence; and inputting the temporal encoding and the original object relationship features of the plurality of image frames into the spatio-temporal Transformer network to obtain object relationship features of each image frame in the video sequence, where the temporal encoding is input into the temporal decoder of the spatio-temporal Transformer network.

[0007] Optionally, the step of using the relationship network to process the object relationship features of the important frames to obtain relationship information between different objects in the video sequence includes: for any one of the important frames, in the case where there are multiple objects of the same category in the important frame, determining irrelevant objects from the multiple objects of the same category according to the reference objects of the corresponding category in the adjacent image frames of the important frame; clearing the features related to the irrelevant objects from the object relationship features of the important frame to obtain the corrected object relationship features of the important frame; and using the relationship network to process the corrected object relationship features of the important frame to obtain relationship information between different objects in the video sequence.

[0008] Optionally, the relationship network includes a spatial encoder, a temporal encoder, and a classifier. Wherein, using the relationship network to process the object relationship features of the important frames to obtain the relationship information between different objects in the video sequence includes: inputting the object relationship features of the important frames into the spatial encoder to obtain a first feature; inputting the object relationship features of the multiple image frames into the long-term encoder to obtain a second feature; inputting the object relationship features of the historical image frames of the important frames into the temporal encoder to obtain a third feature, where the historical image frames are the image frames that are earlier than the important frames in time sequence; inputting the first feature, the second feature, and the third feature into the temporal encoder to obtain a fourth feature; and inputting the fourth feature into the classifier to obtain the relationship information between different objects in the important frames.

[0009] Optionally, the relationship network is trained through the following steps: adjusting the structure of the relationship network to obtain a relationship network to be trained, where the relationship network to be trained includes a spatial encoder, a temporal encoder, and a classifier connected in sequence, and the spatial encoder is at least two in parallel; pre-training the relationship network to be trained through a supervised training method to obtain a pre-trained relationship network, where each time of pre-training inputs the object relationship features of at least two image frames in the sample video sequence into the relationship network to be trained; adjusting the structure of the pre-trained relationship network with reference to the structure of the relationship network to obtain a relationship network to be fine-tuned; and fine-tuning the relationship network to be fine-tuned through a supervised training method to obtain the relationship network.

[0010] Optionally, the loss function used for training the relationship network includes at least one of the following: temporal continuity loss, multi-label margin loss, cross-entropy loss.

[0011] According to a second aspect of the embodiments of the present disclosure, there is provided a dynamic scene graph generation device. The generation device includes: an acquisition unit configured to acquire a video sequence, where the video sequence includes a plurality of image frames arranged in time sequence; a first processing unit configured to use a dynamic scene understanding network to process the plurality of image frames to obtain object relationship features of each image frame, where the object relationship features are used to represent the association relationship between different objects in the corresponding image frame; an extraction unit configured to extract content-representative image frames from the plurality of image frames as important frames; a second processing unit configured to use a relationship network to process the object relationship features of the important frames to obtain the relationship information between different objects in the video sequence; and a generation unit configured to generate a dynamic scene graph of the video sequence according to the relationship information between different objects in the video sequence.

[0012] Optionally, the dynamic scene understanding network includes a spatio-temporal Transformer network, and the first processing unit is further configured to: for each image frame, perform object detection processing, extract visual features of each object in the image frame, and determine original object relationship features of the image frame based on the visual features; obtain a temporal encoding of the video sequence, where the temporal encoding is used to represent the time order of each image frame in the video sequence; input the temporal encoding and the original object relationship features of the multiple image frames into the spatio-temporal Transformer network to obtain object relationship features of each image frame in the video sequence, where the temporal encoding is input into the temporal decoder of the spatio-temporal Transformer network.

[0013] Optionally, the second processing unit is further configured to: for any one of the important frames, in the case that there are multiple objects of the same category in the important frame, determine irrelevant objects from the multiple objects of the same category according to a reference object of the corresponding category in the neighboring image frames of the important frame; remove features related to the irrelevant objects from the object relationship features of the important frame to obtain corrected object relationship features of the important frame; use a relationship network to process the corrected object relationship features of the important frame to obtain relationship information between different objects in the video sequence.

[0014] Optionally, the relationship network includes a spatial encoder, a temporal encoder, and a classifier, and the second processing unit is further configured to: input the object relationship features of the important frame into the spatial encoder to obtain a first feature; input the object relationship features of the multiple image frames into a long-term encoder to obtain a second feature; input the object relationship features of the historical image frames of the important frame into the temporal encoder to obtain a third feature, where the historical image frames are image frames that are earlier than the important frame in time sequence; input the first feature, the second feature, and the third feature into the temporal encoder to obtain a fourth feature; input the fourth feature into the classifier to obtain relationship information between different objects in the important frame.

[0015] Optionally, the relationship network is trained through the following steps: adjusting the structure of the relationship network to obtain a relationship network to be trained, where the relationship network to be trained includes a spatial encoder, a temporal encoder, and a classifier connected in sequence, and the spatial encoder is at least two in parallel; pre-training the relationship network to be trained through a supervised training method to obtain a pre-trained relationship network, where each time of pre-training inputs the object relationship features of at least two image frames in the sample video sequence into the relationship network to be trained; referring to the structure of the relationship network, adjusting the structure of the pre-trained relationship network to obtain a relationship network to be fine-tuned; and fine-tuning the relationship network to be fine-tuned through a supervised training method to obtain the relationship network.

[0016] Optionally, the loss function used for training the relationship network includes at least one of the following: temporal continuity loss, multi-label margin loss, and cross-entropy loss.

[0017] According to a third aspect of the embodiments of the present disclosure, there is provided an electronic device, including: at least one processor; and at least one memory storing computer-executable instructions, where when the computer-executable instructions are run by the at least one processor, the at least one processor is caused to execute the method for generating a dynamic scene graph according to the exemplary embodiments of the present disclosure.

[0018] According to a fourth aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, where instructions in the computer-readable storage medium, when run by at least one processor, cause the at least one processor to execute the method for generating a dynamic scene graph according to the exemplary embodiments of the present disclosure.

[0019] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product, including computer instructions, where when the computer instructions are run by at least one processor, the at least one processor is caused to execute the method for generating a dynamic scene graph according to the exemplary embodiments of the present disclosure.

[0020] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects:

[0021] According to the method and apparatus for generating a dynamic scene graph, electronic device, and storage medium of the present disclosure, by first using an existing dynamic scene understanding network to extract object relationship features, the object relationship features of each image frame can fully contain the association relationships between different image frames, but the amount of information is large and the focus is not clear. On this basis, by first extracting representative important frames and only further processing the important frames to extract the object relationship information therein, the extracted object and relationship information can be made more representative, thereby effectively coping with the long-tail distribution, improving the detection and modeling performance of rare objects and relationships, and helping to improve the integrity and generalization ability of the generated dynamic scene graph.

[0022] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings herein are incorporated into the specification and constitute a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0024] <00001 + 12 + 10>is a flowchart of a method for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure;

[0025] Figure 2 is a schematic framework diagram of an execution system of a method for generating a dynamic scene graph according to a specific embodiment of the present disclosure;

[0026] Figure 3 is a schematic diagram of a training architecture of a relationship network according to a specific embodiment of the present disclosure;

[0027] Figure 4 is a block diagram of a device for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure;

[0028] Figure 5 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that such used data may be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0031] It should be noted that for the tag Figure 1 in the original text, it seems there might be a mistake in the numbering as it doesn't match the sequential pattern. I translated it as <00001 + 12 + 10> according to the rule of preserving the original 7 - digit tags exactly. If there is a specific correction or additional information about this tag, the translation can be adjusted accordingly.It should be noted here that "at least one of several items" as used in this disclosure all represents three parallel cases, namely "any one of the several items", "a combination of any multiple of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. Another example, "performing at least one of Step 1 and Step 2" means the following three parallel cases: (1) performing Step 1; (2) performing Step 2; (3) performing Step 1 and Step 2.

[0032] Next, a method, apparatus, electronic device, storage medium, and program product for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure will be described in detail with reference to the accompanying drawings.

[0033] Figure 1 is a flowchart of a method for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure. The method for generating a dynamic scene graph can be executed by an electronic device with sufficient computing power.

[0034] Referring to Figure 1 , in step S101, a video sequence is obtained, where the video sequence includes a plurality of image frames arranged in time sequence.

[0035] The purpose of the method according to the exemplary embodiment of the present disclosure is to generate a dynamic scene graph for the video sequence obtained here. In other words, the video sequence obtained here is the processing object of the method according to the exemplary embodiment of the present disclosure. This step can obtain the video sequence from the memory of the electronic device executing the method, or obtain the video sequence from other electronic devices through a communication network.

[0036] In step S102, a dynamic scene understanding network is used to process the plurality of image frames to obtain the object relationship features of each image frame.

[0037] The dynamic scene understanding network can be a network in the related art for capturing the temporal information and / or spatial information of a video. The object relationship features obtained via the dynamic scene understanding network are used to represent the association relationships between different objects (objects usually refer to concrete objects, such as people, cups, tables, etc.) in the corresponding image frame, and are detailed features obtained after a comprehensive analysis of the objects in each image frame. They not only reflect the relationships between the objects within each image frame but also can refer to the objects in other image frames, so that the object relationship features of a single image frame contain information in the time dimension.

[0038] In step S103, image frames with content representativeness are extracted from the plurality of image frames as important frames.

[0039] An important frame is an image frame in a video sequence that contains a large amount of information, is representative, or is crucial for understanding the overall content. Important frames are usually obtained through motion analysis, visual content detection, temporal sampling, or model-based selection algorithms to identify and extract frames with large amounts of information or key frames. As an example, for a model-based selection algorithm, a CNN (Convolutional Neural Network) can first be used to extract feature vectors from each image frame, calculate the similarity between different image frames (e.g., using cosine similarity), form a similarity matrix, and then perform clustering analysis. Each cluster represents a group of similar image frames. The central frame or representative frame is selected from each cluster based on the mean of the feature vectors as the important frame. The important frame can be selected based on the position of the image frame in the cluster or the amount of change between frames. Redundant frames can then be removed to find representative image frames. This process can also use unsupervised metrics to evaluate the image frames as a reference for selecting important frames.

[0040] In step S104, a relationship network is used to process the object relationship features of the important frames to obtain the relationship information between different objects in the video sequence. [[ID=[]]

[0041] On the one hand, using a relationship network to further process the object relationship features of the image frames can complete information integration and extract the relationship information between different objects in the video sequence. On the other hand, the specific object relationship features of the important frames are processed. Although the object relationship features of the multiple image frames obtained in step S102 contain comprehensive and detailed information, the amount of information is large, the focus is not clear, and there is a long-tail distribution. By screening these features and only retaining the object relationship features of the important frames extracted in step S103, the amount of data to be processed can be significantly reduced, and the data quality can be improved, making the object and its relationship information extracted therefrom more representative.

[0042] In step S105, a dynamic scene graph of the video sequence is generated according to the relationship information between different objects in the video sequence.

[0043] This step uses the clear object relationship information to achieve image generation and obtain a dynamic scene graph.

[0044] The method for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure first uses an existing dynamic scene understanding network to extract object relationship features, enabling the object relationship features of each image frame to fully contain the association relationships between different image frames. However, the amount of information is large and the key points are not clear. On this basis, by first extracting representative important frames and only further processing the important frames to extract the object relationship information therein, the extracted object and relationship information can be made more representative, thus effectively dealing with the long-tail distribution and improving the detection and modeling performance of rare objects and relationships, which helps to improve the integrity and generalization ability of the generated dynamic scene graph.

[0045] Next, a further introduction is made to the method for generating a dynamic scene graph according to an exemplary embodiment of the present disclosure.

[0046] Regarding step S102, optionally, the dynamic scene understanding network includes a spatio-temporal Transformer network. The spatio-temporal Transformer network, through an encoder-decoder structure and the combination of a spatial encoder and a temporal decoder, effectively captures information in the time dimension. Correspondingly, step S102 includes: for each image frame, performing object detection processing to extract the visual features of each object in the image frame, and based on the visual features, determining the original object relationship features of the image frame; obtaining the temporal encoding of the video sequence, where the temporal encoding is used to represent the time order of each image frame in the video sequence; inputting the temporal encoding and the original object relationship features of multiple image frames into the spatio-temporal Transformer network to obtain the object relationship features of each image frame in the video sequence, where the temporal encoding is input into the temporal decoder of the spatio-temporal Transformer network. By first determining the original object relationship features of the image frame, an original relationship representation can be established for different objects in the image frame. By obtaining the temporal encoding, the description of the time relationship between different image frames can be compensated, allowing the model to consider the time order of multiple image frames when processing the object relationship features, thereby enhancing the modeling of time-dependent relationships. As an example, when extracting the visual features of each object in the image frame, an object detection network (such as but not limited to Faster R-CNN) can be used.

[0047] Further optionally, the operation of determining the original object relationship features of the image frame based on the visual features includes: for each image frame, determining multiple object pairs from all the objects in the image frame, determining the relationship features of the object pairs according to the visual features of the two objects in the object pairs, and taking the relationship features of the multiple object pairs as the original object relationship features of the image frame where t represents the moment corresponding to the image frame, Represents the relationship feature of the k-th object pair, and K(t) represents the number of object pairs in the image frame. When determining the object pairs, for example, each object can be paired with other objects to achieve a complete construction of object pairs. In other words, if the number of objects recognized in the image frame is N(t), then

[0048] As an example, when the spatio-temporal Transformer network processes the temporal encoding and the original object relationship features of multiple image frames, it first takes the original object relationship features of a single image frame Input into the spatial encoder. The encoder with the same multi-layer structure and connected sequentially in the spatial encoder is used to perform iterative calculations on the original object relationship features of each image frame. Then, the calculation results of the spatial encoder for each image frame are aggregated to obtain the intermediate object relationship features of multiple image frames, which are input into the temporal decoder together with the temporal encoding. The temporal decoder can use a sliding window to batch process multiple image frames to enable message passing between adjacent frames to avoid interference with distant frames. Due to the sliding window, the object relationship features in the same image frame have different representations in different batches. For example, for an image frame, the object relationship feature that appears earliest in the window can be selected as the output result of step S102.

[0049] Regarding step S104, optionally, step S104 includes: for any important frame, in the case where there are multiple objects of the same category in the important frame, determining the irrelevant objects from the multiple objects of the same category according to the reference objects of the corresponding category in the adjacent image frames of the important frame; clearing the features related to the irrelevant objects from the object relationship features of the important frame to obtain the corrected object relationship features of the important frame; using the relationship network to process the corrected object relationship features of the important frame to obtain the relationship information between different objects in the video sequence. By determining the objects irrelevant to the reference object as irrelevant objects based on the reference objects of the same category and clearing some features in the object relationship features of the important frame, specifically clearing the features related to the irrelevant objects, it can further reduce the amount of data processed and reduce the interference of irrelevant objects, improve the data quality, make the corrected object relationship features determined thereby more representative, further enhance the detection and modeling performance of rare objects and relationships, and contribute to enhancing the integrity and generalization ability of the finally generated dynamic scene graph.

[0050] It should be understood that the reference object is a representative object in the image frame and can thus be used as a reference for cleaning up irrelevant objects. Specifically, in a video, objects of the same category usually maintain a certain degree of consistency in consecutive frames. If the bounding box of the reference object is detected in a certain frame, it can generally be assumed that objects of the same category also exist in its previous and next frames. Therefore, through this motion consistency, the model can make inferences and selections with the help of the information in the previous and next frames, determine the objects related to the reference object in the current image frame, and regard other objects of the same category in the current image frame as irrelevant objects. At the same time, the selection of objects related to the reference object in the previous and next frames can be based on the context relationship. Even if there is no clear reference object in the neighboring image frames of the current image frame, the model can make speculations based on the detected visual features and motion trends. As an example, in the training phase, the reference object can be an object with a true annotation, or an object speculated based on the object with a true annotation and the context relationship. For this purpose, a sample data set with high annotation quality and good time continuity can be used during training to improve the reliability of the model in obtaining true annotations between different frames. In the inference phase, the reference object can be an object manually annotated, or an object determined by other analysis methods, such as an object obtained after semantic analysis of an important frame. The present disclosure places no restrictions on this.

[0051] It should be noted that step S104 uses the object relationship features of the important frame, that is, feature cleaning is performed based on the important frame, which can be called the frame enhancement strategy. For the object relationship features of the important frame, further screening of the features of objects with relatively high relevance based on the reference object can be called the irrelevant object cleaning strategy. These two strategies both perform refined screening on the features of the image frame based on prior rules and can be collectively called the spatio-temporal prior inference mechanism. When the prior art processes the spatio-temporal dynamic relationship in a video, it usually only focuses on the independent features between static frames and fails to fully model the dynamic changes of objects and their relationships in the time dimension. By introducing the spatio-temporal prior inference mechanism, the present disclosure can effectively capture the changes in object relationships over time in the video and establish a more accurate spatio-temporal model. This enables the subsequent relationship network to not only process the object relationships in the current image frame but also combine the information in the historical image frames to predict the changes in object relationships in future image frames, thereby achieving spatio-temporal consistency modeling over a long time span. At the same time, prior knowledge can be referred to when inferring object relationships, enhancing the rationality and generality of the inference.

[0052] Regarding the relational network used in step S104, optionally, the relational network includes a spatial encoder (which may have the same structure as or a different structure from the spatial encoder in the aforementioned spatio-temporal Transformer network, and the present disclosure does not limit this), a temporal encoder, and a classifier. At this time, step S104 includes: inputting the object relationship features of the important frame into the spatial encoder to obtain a first feature; inputting the object relationship features of multiple image frames into a long-term encoder (which may be an encoder obtained by separate training in advance or a temporal encoder, and the present disclosure does not limit this) to obtain a second feature; inputting the object relationship features of the historical image frames of the important frame (specifically, the image frames that are earlier than and adjacent to the important frame in time, and at least two historical image frames may be used, or all historical image frames of the important frame may be used) into the temporal encoder to obtain a third feature; inputting the first feature, the second feature, and the third feature into the temporal encoder to obtain a fourth feature; and inputting the fourth feature into the classifier to obtain the relationship information between different objects in the important frame. The relational network is used to process each important frame one by one. By jointly inputting the first feature (representing the spatial information of the current important frame) obtained by extracting the spatial information of the current important frame, the second feature (representing the global temporal information) obtained by extracting the temporal information of all image frames, and the third feature (representing the historical information of the current important frame) obtained by extracting the temporal information of at least two historical image frames into the temporal encoder, it can take into account the pertinence and comprehensiveness of the input information and improve the accuracy and integrity of the finally obtained object relationship information.

[0053] Further optionally, the relational network is trained through the following steps: adjusting the structure of the relational network to obtain a relational network to be trained, where the relational network to be trained includes a spatial encoder, a temporal encoder, and a classifier connected in sequence, and the spatial encoder is at least two in parallel; pre-training the relational network to be trained through a supervised training method to obtain a pre-trained relational network, where each time of pre-training inputs the object relationship features of at least two image frames in the sample video sequence into the relational network to be trained; adjusting the structure of the pre-trained relational network with reference to the structure of the relational network to obtain a relational network to be fine-tuned; and fine-tuning the relational network to be fine-tuned through a supervised training method.

[0054] Generally speaking, the training of the relational network includes two stages: pre-training and fine-tuning, that is, first adjusting the network structure to obtain a relational network to be trained and pre-training it, and then adjusting the structure and fine-tuning the parameters of the obtained pre-trained relational network to obtain the relational network.

[0055] In the pre-training stage, different from processing image frames one by one (specifically important frames) in the inference stage, at this time, it is equivalent to processing the object relationship features of at least two historical image frames each time, extracting the third feature from them and inputting it into the temporal encoder, but not inputting the first feature and the second feature. It is possible to perform object relationship extraction only for at least two input image frames in the pre-training stage, improving the training efficiency in the pre-training stage. It should be understood that the number of spatial encoders can be the same as the number of input image frames at this time, so that each spatial encoder is used to process the object relationship features of one image frame, ensuring that the processing object of one spatial encoder is always the object relationship features of one image frame, which is consistent with the inference stage and the fine-tuning stage, and each spatial encoder is exactly the same, that is, the same update needs to be performed on the parameters of each spatial encoder during pre-training.

[0056] In the fine-tuning stage, the network structure is readjusted so that the relationship network to be fine-tuned can input image frames one by one according to the operations in the inference stage, and input the first feature, the second feature, and the third feature into the temporal encoder, maintaining consistency with the inference stage.

[0057] Optionally, the loss function used for training the relationship network includes at least one of the following: temporal continuity loss, multi-label margin loss, and cross-entropy loss. The temporal continuity loss can ensure the consistency of the analysis of the object relationships in image frames at different times. The multi-label margin loss takes into account that the relationships of objects can be described by multiple predicates. For example, the person-object relationship can be described by multiple predicates as "person-holds-bottle" and "person-grabs-bottle". Correspondingly, multiple classifiers can be used to infer different types of relationships. At this time, using the multi-label margin loss can achieve coordinated training of different classifiers. The cross-entropy loss, as a commonly used loss function for classification models, can improve the learning ability of the relationship model for object relationships. The combined use of these three loss functions can improve the training effect of the relationship network.

[0058] As an example, the temporal continuity loss L d can be calculated by the distance metric (such as Euclidean distance) between two objects in an image frame, and the formula is as follows:

[0059]

[0060] where, N f is the number of image frames currently being processed, represents the object distance in the i-th image frame among the currently processed N f image frames, represents the corresponding object distance in the relevant frame of the i-th image frame, and compare it with By making comparisons, the continuity over time can be evaluated. For the current i-th frame, for example, the annotations of the (i-1)-th and (i+1)-th frames can be used as references.

[0061] Multi-label margin loss L p has a formula such as:

[0062]

[0063] For an object pair (specifically a subject-object pair) r, p + is the set of labeled positive samples, which includes multiple predicates and several correct predicates are labeled, q - is the set of negative samples, which includes multiple incorrect predicates. is the confidence score of the p-th predicate, and these scores reflect the credibility of the model for each predicate in the current context. Similarly, is the confidence score of the q-th predicate. The above multi-label margin loss L p specifically represents constructing multiple positive and negative sample pairs based on p + and q - calculating for each sample pair, and then summing the max values of all positive and negative sample pairs as the multi-label margin loss L p . The relation network predicts object relations, specifically by calculating the confidence scores of multiple predicates for a subject-object pair r, and then taking the predicates with scores exceeding the threshold as the prediction results, which can ensure the effectiveness of the prediction and avoid misjudgments with low confidence. This structural design aims to enhance the model's ability to capture fine-grained relations in spatio-temporal scenarios and ensure the accuracy of the prediction.

[0064] As an example, the formula of the total loss function (spatio-temporal prior continuous loss function) used for training the relation network is:

[0065] L total = L p + L d + L o .

[0066] Among them, L o is the standard cross-entropy loss.

[0067] As an example, the same loss function can be used in the pre-training stage and the fine-tuning stage to ensure the consistency of the training objectives. Of course, different loss functions can also be used as needed, and the present disclosure does not limit this.

[0068] Next, a method for generating a dynamic scene graph in a specific embodiment of the present disclosure will be introduced.

[0069] Referring to Figure 2, the system implementing this specific embodiment mainly includes a video sequence processing module, a spatio-temporal Transformer architecture, a spatio-temporal prior reasoning module (which can also be called a frame enhancement and object cleaning module), and a relational network module.

[0070] First, the video sequence processing module is used to extract image frames from the video sequence to obtain input data.

[0071] Then, the spatio-temporal Transformer architecture is used to process the spatio-temporal information in the video sequence. Using the multi-head self-attention mechanism, it realizes the modeling of objects and relationships in the image frames of the video sequence.

[0072] In addition, the spatio-temporal prior reasoning module introduces a spatio-temporal prior reasoning mechanism to capture the dynamic changes of object relationships in the video. By modeling in the time series, the model can predict the evolution of objects and relationships between different image frames, thereby enhancing the understanding of the temporal dependence of video content.

[0073] Finally, the relational network module improves its prediction ability for object relationships in future image frames by pre-training and fine-tuning the relational network. In the pre-training stage, the relational network can learn spatio-temporal features on large-scale unlabeled video data, which enables it to better adapt to downstream tasks, thereby improving the overall prediction accuracy and generalization ability.

[0074] Specifically, the operations of the spatio-temporal Transformer architecture for processing the spatio-temporal information in the video sequence mainly include relationship representation and the processing of the spatio-temporal Transformer network.

[0075] Relationship representation:

[0076] Faster R-CNN is used to extract the visual features of each object in image frame t and extract the bounding box of each object as well as the object category distribution of the target candidate boxes representing the category of each extracted object, the relationship representation of the relationship between the i-th and j-th target candidate boxes contains visual appearance, spatial information, and semantic embeddings, and can be expressed as:

[0077]

[0078] where <,> represents the concat connection operation, represents the flatten operation, represents the element-wise addition, f s 、f o are respectively implemented by a fully connected layer, mapping a 2048-dimensional vector to a 512-dimensional vector, f uIt is implemented by a fully connected layer that maps a 12,544 - dimensional vector to a 512 - dimensional vector. The feature map corresponding to the union box generated by RoIAlign combines the features of two object candidate boxes, enabling the subsequent decoder to better understand their interaction. Through such a design, the model's understanding of the inter - object relationship can be improved, thereby enhancing the overall performance of object detection. f box A function that converts the bounding boxes of the subject and object into a form that can be used for feature fusion or further calculation. This conversion usually involves mapping the information of the bounding boxes (such as coordinates, dimensions) to a specific feature space for easy combination with other features. Semantic embedding vector It is determined by the object categories of the subject and object and is obtained by directly processing the object category texts of the subject and object. Among them, the 200 - dimensional semantic embedding vector can capture the information of the object category to a certain extent. If the task requires richer feature representation, the dimension of the semantic embedding vector can also be increased. Relationship representation Capable of exchanging spatial and temporal information in the spatio - temporal Transformer network.

[0079] Processing of the spatio - temporal Transformer network:

[0080] The spatio - temporal Transformer network maintains the original encoder - decoder architecture. The difference is that the encoder and decoder are given more specific tasks. The input to the spatial encoder is a single The query Q, key K, and value V share the same input. The output of the n - th layer encoder is represented as:

[0081]

[0082] The encoder consists of N identical Att enc. layers, which are stacked in sequence. The input of the n - th layer is equal to the output of the (n - 1) - th layer. Among them, the spatial information hidden in the relationship representation plays a crucial role in the self - attention mechanism. The final output of the encoder stack is sent to the temporal decoder.

[0083] Frame encoding is introduced as temporal encoding in the temporal decoder. Without convolution and recursion, by customizing frame encoding as sequence - order knowledge, the position of time can be injected into the relationship representation X t so as to explicitly convey time information through frame encoding without convolution and recursion, helping the model better capture temporal dynamics. In addition, since the number of embedding vectors depending on the window size η in the temporal decoder is fixed and relatively short, frame encoding is constructed using learned embedding parameters: E f =[e1,…,eη ], where e1,…,e η ∈R 1936 , is the same as The learned vectors of the same length can be used to train the frame encoding while training the model to achieve its learning.

[0084] Temporal decoders capture temporal dependencies between image frames, but this not only significantly increases computational effort and memory consumption, but also easily overwhelms useful information with a large number of irrelevant representations. By batching image frames using a sliding window, messages are passed between adjacent frames, avoiding interference with distant frames. Because frame encodings provide sequential order knowledge, the frame encoding used in each batch remains unchanged.

[0085] Self-attention layer of temporal decoder and spatial encoder Att enc. () is the same, that is, the masked multi-head self-attention layer is removed. The sliding window of size η represents the sequence [X1,…,X T ], the i-th generated input batch is represented as:

[0086] Z i =[X i ,…,X i+η-1 ],i∈{1,…,T-η+1}

[0087] Where the window size η≤T, and T is the length of the video. The decoder consists of N layers of identical self-attention layers Att stacked together. dec. (), the structure is similar to the encoder. Consider the first layer:

[0088] Q=K=Z i +E f ,

[0089] V=Z i ,

[0090]

[0091] The same encoding is added to the relation representations in the same frame as the query and key, and the final prediction uses the output from the last decoder layer. Due to the sliding window, relations in the same frame have different representations in different batches. Selecting the earliest representation in the window can more effectively capture temporal dependencies and reduce redundant information, thereby improving performance in video understanding or time series prediction tasks. This strategy helps ensure that the model can dynamically adapt to changing contexts, improving the accuracy of its predictions.

[0092] It should be noted that Figure 2The spatio-temporal Transformer network has a noisy video inference part, which means that it can consider the impact of noise during the inference stage. This part is carried out during network training and does not directly affect the structure of the encoder or decoder.

[0093] The operations of the spatio-temporal prior inference module mainly include frame enhancement and object cleaning.

[0094] Regarding frame enhancement, a temporal prior frame enhancement strategy is specifically designed to automatically obtain more reliable samples as important frames in the video. Specifically, important frames are selected through an unsupervised method, and it is ensured that these frames contain object information related to the task, thereby improving the performance and accuracy of the model in temporal tasks. This strategy enables the model to effectively capture valuable temporal information without manual annotation, enabling the model to capture fine-grained temporal continuity. Frame enhancement is performed based on the results of detected unannotated frames without additional manual annotation.

[0095] At the same time, object cleaning is required, that is, deleting irrelevant objects. If multiple objects of the same category are detected in a certain frame, the irrelevant objects need to be removed, and the objects related to the true annotated objects in the previous and next frames are retained. Specifically, the bounding boxes of the same category objects in the true annotations of the previous frame and the next frame of the current frame are used as references, and the objects most relevant to them are selected and retained, while other objects are cleaned as irrelevant objects. For the bounding box where (x i , y i ) is the center point of the bounding box, and (w i , h i ) represents the width and height of the bounding box. i ∈ {1, 2, …, N V}, where N V is the number of objects of the same category detected in the important frames. The object bounding box in the current frame and the same category annotated object B p = [x p , y p , w p , h p in the previous frame, the correlation between them is calculated by the following formula:

[0096]

[0097] The smaller this value, the stronger the correlation between objects. If the annotation of the previous frame does not contain the category of the i-th detected object, then is 0, obtaining the sequence Similarly, the correlation of the next frame is Using the calculation formula to represent the correlation between the i-th detected object and the previous and next frames, where i ∈ {1, 2, …, N v}. The u-th object with the smallest value in this sequence is selected as the final result, representing the object most relevant to the previous and next frames, which is retained, while other objects are removed as irrelevant objects. Specifically, the relationship representation X t of this important frame is removed for the elements related to the irrelevant objects.

[0098] The relational network used in the relational network module is used to implement object relationship prediction (since it is used to generate a dynamic scene graph, it can also be called dynamic relationship prediction). It needs to be pre-trained and fine-tuned first. Referring to Figure 3 The pre-training and fine-tuning strategies are as follows:

[0099] Multiple linear classifiers are used to predict different types of object relationships, rather than just using one classifier. r t,ij represents the distribution of relationship categories between object O t,i and O t,j . The prediction formula in the pre-training stage is where t represents the time index of the current image frame, referring to the position of the image frame being processed in the entire video sequence. l represents the level or number of layers of the feature representation, usually the features of object O t,i and O t,j obtained after being processed by multiple layers of the network, which can help the model extract information related to relationship prediction at different levels. y t,ij represents the predicted object category. Since there may be multiple correct relationships between two objects in practice, such as "person touches food" and "person eats food", a multi-label margin loss is adopted in the pre-training, and its formula is as follows:

[0100]

[0101] where Y + represents the set of predicates containing the true labels, Y - represents the set of unlabeled negative predicates, and is the predicted confidence score of the p-th predicate, is the predicted confidence score of the q-th predicate.

[0102] During the fine-tuning process, the spatial encoder mentioned above is reused to capture the spatial information of the current frame. A global temporal encoder is then used to capture temporal correlations based on the output of the long-term encoder (the input of the long-term encoder is the object relationship features of multiple image frames of the video sequence output by the spatial-temporal Transformer network). This process may involve aggregating information at different time points in the sequence to better understand the temporal relationship between objects and actions. The global temporal encoder shares parameters with the progressive temporal encoder in the pre-training phase (which can also be called a short-term encoder relative to the long-term encoder). Both are temporal encoders in the relational network introduced above, but they receive different input information and can focus on different time scales, thereby jointly improving the performance of the model in spatiotemporal tasks. Therefore, different names are used here to distinguish their functions. The formula is as follows:

[0103]

[0104] The subscript g indicates that this feature is generated by the global temporal encoder and is used to capture temporal correlations over a long time range. When using a pre-trained relational network to process the object relation features of the t-2th image frame and the t-1th image frame, the lth layer feature output by the pre-trained progressive temporal encoder corresponds to Figure 3 The global temporal encoder in the middle is fed with input from above. t,ij Indicates time-related coding, which can be output by the long-term encoder, corresponding to Figure 3 The global temporal encoder in Z comes from the input below. f It is a frame encoding, mainly used for global temporal encoders, focusing on the specific features and temporal context of the current frame, such as the feature representation generated from the spatial features of the current frame, which can be output by the spatial encoder, corresponding to Figure 3 The global temporal encoder in the figure takes input from the left, and thus has the same characteristics as the frame encoder E f The final output of the global temporal encoder is expressed as Similar to pre-training, multiple linear classifiers are also used in fine-tuning:

[0105]

[0106] in, yes The last element in the fine-tuning process uses the same loss function as the pre-training. In the inference phase, only the classifiers in the fine-tuning phase are used. fin , while the classifiers in the pre-training stage are pre It will no longer be used to obtain the predicted object relationships and then generate a dynamic scene graph based on them.

[0107] The prior art for dynamic scene graph generation includes, for example, weakly-supervised methods that input a target video into a trained weakly-supervised dynamic scene graph generation model to obtain the dynamic scene graph corresponding to the target video. There are also methods based on temporal transition regularization, which create a transformation matrix conditional on the categories and contexts of images to capture the statistical correlations based on each category and the fine-grained correlations based on each context image, and explicitly incorporate these correlations into the current VidSGG algorithm to provide stronger and more fine-grained regularization. Additionally, there are methods based on tracking algorithms that use object element tracking methods to obtain the element information in each frame, then fit the scene relationships between the object elements in each frame in connection with the temporal context information, and generate the corresponding scene graph based on the scene relationships.

[0108] Specific embodiments of the present disclosure mainly aim to generate a dynamic scene graph from a video scene, extract the key objects and their mutual relationships in the video sequence, so that the generated dynamic scene graph can accurately represent the video content, improve the efficiency of video understanding and analysis, and further provide more comprehensive data support for video-related applications.

[0109] Compared with the weakly-supervised method, the weakly-supervised method relies on a pre-trained weakly-supervised model. Although such methods do not require a large amount of labeled data, they fail to effectively incorporate domain knowledge or prior information during the inference process, resulting in the model possibly lacking sufficient semantic information support when facing complex scenes. In contrast, the method of the present specific embodiment addresses the problem of inaccurate inference caused by lack of knowledge in existing methods by introducing prior knowledge into the model, significantly enhancing the model's inference ability and understanding of object relationships in complex scenes.

[0110] Compared with the method based on temporal transition regularization, it captures the statistical correlations between categories and contexts through an explicit transition matrix and regularizes the temporal transitions. Although this method captures some temporal correlations at the fine-grained level, it relies on a fixed statistical transition matrix, lacking flexibility, especially with poor adaptability when dealing with dynamic changes. The method of the present specific embodiment introduces a temporal prior inference mechanism that predicts the dynamic changes of object relationships by analyzing the temporal patterns in the video sequence, and further improves the prediction ability of future frame relationships through a pre-trained model. This approach not only considers the dependency between the current frame and historical frames but also can predict the relationship evolution of future frames, having a stronger temporal modeling ability.

[0111] Compared with the method based on the tracking algorithm, this method obtains the scene relationship by tracking object elements. It relies on tracking objects frame by frame, resulting in the temporal information being only locally captured, the modeling of spatio-temporal information being relatively limited, prone to spatio-temporal context breaks, and unable to fully utilize the dynamic changes of objects in the video. The method of this specific embodiment introduces a spatio-temporal Transformer network, which can perform global modeling simultaneously in the temporal and spatial dimensions of the entire video sequence through the multi-head self-attention mechanism, can capture richer spatio-temporal information between image frames, ensure that the changes of objects and their relationships at different times are accurately modeled, and thus improve the spatio-temporal consistency of dynamic scene graph generation.

[0112] Figure 4 is a block diagram of a dynamic scene graph generation device according to an exemplary embodiment of the present disclosure. Referring to Figure 4 As shown in the figure, the generation device 400 includes an acquisition unit 401, a first processing unit 402, an extraction unit 403, a second processing unit 404, and a generation unit 405.

[0113] The acquisition unit 401 can acquire a video sequence, where the video sequence includes a plurality of image frames arranged in time sequence.

[0114] The first processing unit 402 can use a dynamic scene understanding network to process a plurality of image frames to obtain the object relationship features of each image frame, where the object relationship features are used to represent the association relationships between different objects in the corresponding image frame.

[0115] The extraction unit 403 can extract image frames with content representativeness from a plurality of image frames as important frames.

[0116] The second processing unit 404 can use a relationship network to process the object relationship features of the important frames to obtain the relationship information between different objects in the video sequence.

[0117] The generation unit 405 can generate a dynamic scene graph of the video sequence according to the relationship information between different objects in the video sequence.

[0118] Optionally, the dynamic scene understanding network includes a spatio-temporal Transformer network, and the first processing unit 402 can also: for each image frame, perform object detection processing, extract the visual features of each object in the image frame, and based on the visual features, determine the original object relationship features of the image frame; obtain the temporal encoding of the video sequence, where the temporal encoding is used to represent the time order of each image frame in the video sequence; input the temporal encoding and the original object relationship features of the plurality of image frames into the spatio-temporal Transformer network to obtain the object relationship features of each image frame in the video sequence, where the temporal encoding is input into the temporal decoder of the spatio-temporal Transformer network.

[0119] Optionally, the second processing unit 404 may further: for any important frame, in the case that there are multiple objects of the same category in the important frame, determine irrelevant objects from the multiple objects of the same category according to the reference objects of the corresponding category in the adjacent image frames of the important frame; remove the features related to the irrelevant objects from the object relationship features of the important frame to obtain the corrected object relationship features of the important frame; use a relationship network to process the corrected object relationship features of the important frame to obtain the relationship information between different objects in the video sequence.

[0120] Optionally, the relationship network includes a spatial encoder, a temporal encoder, and a classifier. The second processing unit 404 may further: input the object relationship features of the important frame into the spatial encoder to obtain a first feature; input the object relationship features of multiple image frames into the long-term encoder to obtain a second feature; input the object relationship features of the historical image frames of the important frame into the temporal encoder to obtain a third feature, where the historical image frames are the image frames that are earlier than the important frame in time sequence; input the first feature, the second feature, and the third feature into the temporal encoder to obtain a fourth feature; input the fourth feature into the classifier to obtain the relationship information between different objects in the important frame.

[0121] Optionally, the relationship network is trained through the following steps: adjust the structure of the relationship network to obtain a relationship network to be trained, where the relationship network to be trained includes a spatial encoder, a temporal encoder, and a classifier connected in sequence, and the spatial encoder is at least two in parallel; perform pre-training on the relationship network to be trained through a supervised training method to obtain a pre-trained relationship network, where each time of pre-training inputs the object relationship features of at least two image frames in the sample video sequence into the relationship network to be trained; refer to the structure of the relationship network and adjust the structure of the pre-trained relationship network to obtain a relationship network to be fine-tuned; perform fine-tuning on the relationship network to be fine-tuned through a supervised training method to obtain the relationship network.

[0122] Optionally, the loss function used for training the relationship network includes at least one of the following: temporal continuity loss, multi-label margin loss, cross-entropy loss.

[0123] According to an embodiment of the present disclosure, an electronic device may be provided. Figure 5 It is a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 500 includes at least one memory 501 and at least one processor 502. A set of computer-executable instructions 5011 and an operating system 5012 are stored in the at least one memory. When the set of computer-executable instructions 5011 is executed by the at least one processor 502, a method for generating a dynamic scene graph according to an embodiment of the present disclosure is executed.

[0124] As an example, the electronic device 500 can be a PC computer, a tablet device, a personal digital assistant, a smart phone, or other devices capable of executing the above instruction set. Here, the electronic device 500 does not have to be a single electronic device, and can also be a collection of devices or circuits that can execute the above instructions (or instruction sets) individually or jointly. The electronic device 500 can also be a part of an integrated control system or system manager, or can be configured as a portable electronic device that interfaces with a local or remote (e.g., via wireless transmission).

[0125] In the electronic device 500, the processor 502 can include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor 502 can also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, and the like.

[0126] The processor 502 can run instructions or code stored in the memory, where the memory 501 can also store data. The instructions and data can also be sent and received via the network interface device over a network, where the network interface device can employ any known transmission protocol.

[0127] The memory 501 can be integrated with the processor 502, for example, by arranging RAM or flash memory within an integrated circuit microprocessor, etc. In addition, the memory 501 can include a separate device, such as an external disk drive, a storage array, or other storage devices that can be used by any database system. The memory 501 and the processor 502 can be operatively coupled, or can communicate with each other, for example, through an I / O port, a network connection, etc., such that the processor 502 can read files stored in the memory 501.

[0128] In addition, the electronic device 500 can also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.). All components of the electronic device can be connected to each other via a bus and / or a network.

[0129] According to an embodiment of the present disclosure, a computer-readable storage medium may also be provided, wherein when instructions in the computer-readable storage medium are run by at least one processor, the at least one processor is caused to execute the method for generating a dynamic scene graph according to the embodiment of the present disclosure. Examples of such computer-readable storage media include: read-only memory (ROM), programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid state drive (SSD), cartridge memory (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer such that the processor or computer can execute the computer program. The computer program in the above computer-readable storage medium may run in an environment deployed in computer devices such as clients, hosts, proxy devices, servers, etc. In addition, in one example, the computer program and any associated data, data files, and data structures are distributed on a networked computer system such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0130] According to an embodiment of the present disclosure, there is provided a computer program product including computer instructions which, when executed by a processor, implement the method for generating a dynamic scene graph according to the embodiment of the present disclosure.

[0131] Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the appended claims.

[0132] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A method for generating a dynamic scene graph, characterized in that, The generation method includes: Obtain a video sequence, where the video sequence includes a plurality of image frames arranged in chronological order; Use a dynamic scene understanding network to process the plurality of image frames to obtain object relationship features of each image frame, where the object relationship features are used to represent the association relationships between different objects in the corresponding image frame; Extract content-representative image frames from the plurality of image frames as important frames; Use a relationship network to process the object relationship features of the important frames to obtain relationship information between different objects in the video sequence; Generate a dynamic scene graph of the video sequence according to the relationship information between different objects in the video sequence; Wherein, the relationship network includes a spatial encoder, a temporal encoder, and a classifier, and the using the relationship network to process the object relationship features of the important frames to obtain the relationship information between different objects in the video sequence includes: Input the object relationship features of the important frames into the spatial encoder to obtain a first feature; Input the object relationship features of the plurality of image frames into a long-term encoder to obtain a second feature; Input the object relationship features of the historical image frames of the important frames into the temporal encoder to obtain a third feature, where the historical image frames are image frames that are earlier than the important frames in time sequence; Input the first feature, the second feature, and the third feature into the temporal encoder to obtain a fourth feature; Input the fourth feature into the classifier to obtain the relationship information between different objects in the important frames.

2. The generation method according to claim 1, wherein The dynamic scene understanding network includes a spatio-temporal Transformer network, and the using the dynamic scene understanding network to process the plurality of image frames to obtain object relationship features of each image frame includes: For each image frame, perform object detection processing, extract visual features of each object in the image frame, and determine the original object relationship features of the image frame based on the visual features; Obtain the temporal encoding of the video sequence, where the temporal encoding is used to represent the time order of each image frame in the video sequence; Input the temporal encoding and the original object relationship features of the plurality of image frames into the spatio-temporal Transformer network to obtain the object relationship features of each image frame in the video sequence, where the temporal encoding is input into the temporal decoder of the spatio-temporal Transformer network.

3. The generation method according to claim 1, wherein The using the relationship network to process the object relationship features of the important frames to obtain the relationship information between different objects in the video sequence includes: For any one of the important frames, in the case where there are multiple objects of the same category in the important frame, determine irrelevant objects from the multiple objects of the same category according to the reference objects of the corresponding category in the neighboring image frames of the important frame; Clear the features related to the irrelevant objects from the object relationship features of the important frame to obtain the corrected object relationship features of the important frame. Use a relational network to process the corrected object relationship features of the important frames, and obtain the relationship information between different objects in the video sequence.

4. The generation method according to any one of claims 1 to 3, characterized in that The relational network is trained through the following steps: Adjust the structure of the relational network to obtain a relational network to be trained, where the relational network to be trained includes a spatial encoder, a temporal encoder, and a classifier connected in sequence, and the spatial encoder is at least two in parallel; Pre-train the relational network to be trained through a supervised training method to obtain a pre-trained relational network, where each pre-training inputs the object relationship features of at least two image frames in the sample video sequence into the relational network to be trained; Refer to the structure of the relational network and adjust the structure of the pre-trained relational network to obtain a relational network to be fine-tuned; Fine-tune the relational network to be fine-tuned through a supervised training method to obtain the relational network.

5. The generation method according to claim 4, wherein The loss function used in the relational network training includes at least one of the following: temporal continuity loss, multi-label margin loss, cross-entropy loss.

6. A device for generating a dynamic scene graph, characterized in that, The generation device includes: An acquisition unit configured to acquire a video sequence, where the video sequence includes a plurality of image frames arranged in time sequence; A first processing unit configured to use a dynamic scene understanding network to process the plurality of image frames to obtain the object relationship features of each image frame, where the object relationship features are used to represent the association relationship between different objects in the corresponding image frame; An extraction unit configured to extract the image frames with content representativeness from the plurality of image frames as important frames; A second processing unit configured to use a relational network to process the object relationship features of the important frames to obtain the relationship information between different objects in the video sequence; A generation unit configured to generate a dynamic scene graph of the video sequence according to the relationship information between different objects in the video sequence; where the relational network includes a spatial encoder, a temporal encoder, and a classifier, and the second processing unit is further configured to: Input the object relationship features of the important frames into the spatial encoder to obtain a first feature; Input the object relationship features of the plurality of image frames into a long-term encoder to obtain a second feature; Input the object relationship features of the historical image frames of the important frames into the temporal encoder to obtain a third feature, where the historical image frames are image frames earlier than the important frames in time sequence; Input the first feature, the second feature, and the third feature into the temporal encoder to obtain a fourth feature; Input the fourth feature into the classifier to obtain the relationship information between different objects in the important frames.

7. An electronic device, characterized in that, Includes: At least one processor; At least one memory storing computer-executable instructions, wherein when the computer-executable instructions are run by the at least one processor, the at least one processor is caused to execute the dynamic scene graph generation method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are run by at least one processor, the at least one processor is caused to execute the method for generating a dynamic scene graph according to any one of claims 1 to 5.

9. A computer program product comprising computer instructions, characterized in that, When the computer instructions are run by at least one processor, the at least one processor is caused to execute the method for generating a dynamic scene graph according to any one of claims 1 to 5.

Images