Weakly supervised video scene graph generation method and device, and electronic equipment

Abstract

The application provides a weakly supervised video scene graph generation method and device and electronic equipment, wherein the method comprises: inputting a target video into a trained weakly supervised video scene graph generation model to obtain a video scene graph corresponding to the target video; wherein the weakly supervised video scene graph generation model is trained based on a sample video set, a set of unpositioned scene graphs corresponding to the sample video set, and a text concept data set. The workload of manual labeling can be reduced, thereby effectively reducing the cost of video data labeling.

Description

Technical Field

[0001] This invention relates to the field of visual understanding technology, and in particular to a method, apparatus and electronic device for generating weakly supervised video scene graphs. Background Technology

[0002] With the development of deep learning technology, current techniques for image data detection and segmentation are relatively mature. Among them, video scene graph generation models can express the relationships and interactions between various entities by taking a video as input and outputting visual relationships in the form of "subject-relationship-object," helping people understand visual content from a higher semantic level.

[0003] However, most current video scene graph generation algorithms are fully supervised, requiring a large amount of expensive manually labeled data for model training. Summary of the Invention

[0004] This invention provides a weakly supervised video scene graph generation method, apparatus, and electronic device to address the shortcomings of existing scene graph generation algorithms that require a large amount of expensive manually labeled data for model training, thereby effectively reducing the cost of video data labeling.

[0005] This invention provides a method for generating weakly supervised video scene graphs, comprising:

[0006] The target video is input into a trained weakly supervised video scene graph generation model to obtain the video scene graph corresponding to the target video.

[0007] The weakly supervised video scene graph generation model is trained based on a sample video set, an unlocalized scene graph set corresponding to the sample video set, and a text concept dataset.

[0008] According to a weakly supervised video scene graph generation method provided by the present invention, the training method of the weakly supervised video scene graph generation model includes:

[0009] Select sample videos from the sample video set;

[0010] Based on the sample video and the corresponding unlocalized scene map, the category of each entity in the sample video is predicted to obtain the predicted category of each entity in the sample video.

[0011] The first classifier of the weakly supervised video scene graph generation model is trained based on the category labels of each entity in the sample video and the predicted category.

[0012] Based on the sample video, the unlocalized scene graph, and the text concept dataset, the classification probability of multiple relation categories corresponding to each subject-object pair in the sample video is predicted to obtain the classification prediction probability of each subject-object pair in the sample video.

[0013] The second classifier of the weakly supervised video scene graph generation model is trained based on the real relation labels corresponding to each subject-object pair in the sample video and the classification prediction probability.

[0014] According to a weakly supervised video scene graph generation method provided by the present invention, the step of predicting the classification probability of multiple relation categories corresponding to each subject-object pair in the sample video based on the sample video, the unlocalized scene graph, and the text concept dataset includes:

[0015] Determine the unbiased features of each text concept in the text concept dataset;

[0016] The subject category set and the object category set of the sample video are determined based on the category labels of each entity in the sample video.

[0017] Based on the text features of each text concept in the text concept dataset, the unbiased features of each text concept in the text concept dataset, and the context-aware cue template, the text entity representations of the subject category set and the object category set are determined respectively.

[0018] Based on the text features of each text concept in the text concept dataset, the unbiased features of each text concept in the text concept dataset, and the trajectory features of each entity in the sample video, the visual entity representations of the subject category set and the object category set are determined respectively.

[0019] The text entity representations of the subject category set, the text entity representations of the object category set, the visual entity representations of the subject category set, and the visual entity representations of the object category set are input into a second classifier to predict the classification probabilities of multiple relation categories corresponding to each subject-object pair in the sample video.

[0020] According to a weakly supervised video scene graph generation method provided by the present invention, the step of determining the text entity representations of the subject category set and the object category set respectively specifically includes:

[0021] Determine the context-aware cue templates for the subject and the context-aware cue templates for the object respectively;

[0022] Based on the text features of the subject category set, the unbiased features of the subject category set, the context-aware cue template of the subject, and the bidirectional long short-term memory network, the text entity representation of the subject category set is determined.

[0023] Based on the text features of the object category set, the unbiased features of the object category set, the context-aware cue template of the object, and the bidirectional long short-term memory network, the text entity representation of the object category set is determined.

[0024] According to a weakly supervised video scene graph generation method provided by the present invention, the step of determining the visual entity representations of the subject category set and the object category set respectively specifically includes:

[0025] A first fully connected concept graph is determined; the nodes in the first fully connected concept graph represent text concepts in the text concept dataset.

[0026] The entities corresponding to the subject category set and the entities corresponding to the object category set are respectively embedded as embedding nodes into the first fully connected concept graph to obtain the second fully connected concept graph corresponding to the subject category set and the third fully connected concept graph corresponding to the object category set;

[0027] Based on the second fully connected concept map and the third fully connected concept map, the visual entity features of the subject category set and the visual entity features of the object category set are determined respectively.

[0028] The features of the nodes in the first fully connected conceptual graph are determined based on the text features and unbiased features corresponding to the nodes in the first fully connected conceptual graph; the features of the embedded nodes are determined based on the trajectory features, text features, and unbiased features corresponding to the embedded nodes.

[0029] According to a weakly supervised video scene graph generation method provided by the present invention, determining the unbiased features of each text concept in the text concept dataset specifically includes:

[0030] Determine the co-occurrence frequency between any two text concepts in the text concept dataset, and the similarity between the text features of any two text concepts;

[0031] For any text concept, multiple text concepts are selected as the co-occurrence concept set of the arbitrary text concept in ascending order of their co-occurrence frequency with the arbitrary text concept, and multiple text concepts are selected as the semantic concept set of the arbitrary text concept in ascending order of their similarity to the text features of the arbitrary text concept.

[0032] Based on the set of co-occurring concepts of the arbitrary text concept, the set of semantic concepts of the arbitrary text concept, and the set of semantic concepts of each text concept in the set of co-occurring concepts of the arbitrary text concept, the unbiased features of the arbitrary text concept are determined.

[0033] According to a weakly supervised video scene graph generation method provided by the present invention, the step of predicting the category of each entity in the sample video based on the sample video and the corresponding unlocalized scene graph specifically includes:

[0034] Based on the trajectory features of each entity in the sample video, the relationship features of each entity in the unlocalized scene map, and the Gated Recurrent Unit (GRU) network based on the attention mechanism, the relationship-aware visual features of each entity in the sample video are obtained.

[0035] The visual features of the relationships between the entities in the sample video are input into the first classifier to predict the category of each entity in the sample video.

[0036] The category labels of each entity in the sample video are determined according to the following method:

[0037] Based on the text features of each entity in the unlocated scene graph, the relation features of each entity in the unlocated scene graph, and the GRU network based on the attention mechanism, the relation-aware text features of each entity in the unlocated scene graph are obtained.

[0038] For any entity in the sample video, calculate the similarity between the relation-aware visual features of the arbitrary entity and the relation-aware text features of each entity in the unlocalized scene graph, and select the text concept corresponding to the relation-aware text feature with the highest similarity to the relation-aware visual features of the arbitrary entity as the category label of the arbitrary entity.

[0039] The present invention also provides a weakly supervised video scene graph generation device, comprising:

[0040] The generation module is used to input the target video into a trained weakly supervised video scene graph generation model to obtain the video scene graph corresponding to the target video.

[0041] The weakly supervised video scene graph generation model is trained based on a sample video set, an unlocalized scene graph set corresponding to the sample video set, and a text concept dataset.

[0042] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the weakly supervised video scene graph generation method as described above.

[0043] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the weakly supervised video scene graph generation method as described above.

[0044] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the weakly supervised video scene graph generation method as described above.

[0045] The weakly supervised video scene graph generation method, apparatus, and electronic device provided by this invention train a weakly supervised video scene graph generation model based on a sample video set, an unlocalized scene graph set corresponding to the sample video set, and a text concept dataset. Then, the target video is input into the trained weakly supervised video scene graph generation model to obtain the video scene graph corresponding to the target video. Compared with fully supervised algorithms, this reduces the workload of manual annotation, thereby effectively reducing the cost of video data annotation. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 A flowchart illustrating the weakly supervised video scene graph generation method provided by the present invention;

[0048] Figure 2 A schematic diagram of the unbiased concept diagram provided for this invention;

[0049] Figure 3 A schematic diagram of the weakly supervised video scene graph generation device provided by the present invention;

[0050] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0052] Figure 1 The flowchart of the weakly supervised video scene graph generation method provided by the present invention is shown below. Figure 1 As shown, the method includes the following steps:

[0053] Step 100: Input the target video into the trained weakly supervised video scene graph generation model to obtain the video scene graph corresponding to the target video.

[0054] The weakly supervised video scene graph generation model is trained based on a sample video set, an unlocalized scene graph set corresponding to the sample video set, and a text concept dataset.

[0055] Specifically, the execution subject of the method provided by the present invention can be a processing device that can receive input from other devices and has a certain computing capability. The following uses a computer device as an example to illustrate the method provided by the present invention.

[0056] The target video is the video for which a video scene graph needs to be generated. The video scene graph corresponding to the target video is a visual relationship in the form of "subject-relationship-object" for the target video. It includes the location of each visual entity in the target video, the text concept corresponding to each visual entity, and the text concept corresponding to the relationship between each visual entity.

[0057] Most current video scene graph generation algorithms are fully supervised, meaning they simultaneously label the location of each visual entity in the sample video, the textual concept corresponding to each visual entity, and the relationships between each visual entity. This requires a large amount of expensive manually labeled data for model training.

[0058] Therefore, this invention provides a weakly supervised video scene graph generation method to reduce the cost of video data annotation. Weak supervision means that only the unlocalized scene graph is used as supervision information during the training phase. The unlocalized scene graph only contains the category information of entities in the video and the relationship information between entities, but does not include annotations of the location of visual entities. This weakly supervised training method can effectively reduce the cost of video data annotation.

[0059] First, a weakly supervised video scene graph generation model is trained on a computer device based on a sample video set, a corresponding unlocalized scene graph set, and a text concept dataset. This model generates multiple sample videos from the sample videos and unlocalized scenes from the unlocalized scene graph set. Figure 1 In a one-to-one correspondence, the text concept dataset can include multiple entity category text concepts (i.e. text concepts as subjects and objects) and multiple relation category text concepts (i.e. text concepts as predicates).

[0060] After obtaining a trained weakly supervised video scene graph generation model, inputting the target video into the trained weakly supervised video scene graph generation model can produce the video scene graph corresponding to the target video.

[0061] The weakly supervised video scene graph generation method provided by this invention trains a weakly supervised video scene graph generation model based on a sample video set, an unlocalized scene graph set corresponding to the sample video set, and a text concept dataset. Then, the target video is input into the trained weakly supervised video scene graph generation model to obtain the video scene graph corresponding to the target video. Compared with fully supervised algorithms, this method reduces the workload of manual annotation, thereby effectively reducing the cost of video data annotation.

[0062] Optionally, training methods for weakly supervised video scene graph generation models include:

[0063] Select sample videos from the sample video set;

[0064] Based on the sample video and the corresponding unlocalized scene map, the category of each entity in the sample video is predicted to obtain the predicted category of each entity in the sample video.

[0065] The first classifier of the weakly supervised video scene graph generation model is trained based on the category labels and predicted categories of each entity in the sample video.

[0066] Based on sample videos, unlocalized scene graphs, and text concept datasets, the classification probabilities of multiple relation categories corresponding to each subject-object pair in the sample videos are predicted, thus obtaining the classification prediction probabilities of each subject-object pair in the sample videos.

[0067] The second classifier of the weakly supervised video scene graph generation model is trained based on the real relation labels and classification prediction probabilities corresponding to each subject-object pair in the sample video.

[0068] Specifically, the weakly supervised video scene graph generation model can include two classifiers. The first classifier, which is trained, is used to predict the category of each entity in the target video, and the second classifier, which is trained, is used to predict the subject-object pair relationship formed by each entity in the target video.

[0069] It is understandable that the training process for the two classifiers in the weakly supervised video scene graph generation model involves selecting sample videos from the sample video set multiple times to train the two classifiers. Only after a preset number of training iterations or the achievement of preset conditions can a well-trained weakly supervised video scene graph generation model be obtained.

[0070] The unlocalized scene map corresponding to the sample video contains the category information of the entities in the sample video. Therefore, based on the sample video and the unlocalized scene map corresponding to the sample video, the first classifier can be used to predict the category of each entity in the sample video, and the predicted category of each entity in the sample video can be obtained.

[0071] Then, based on the category labels and predicted categories of each entity in the sample video, a first classifier can be trained using a loss function.

[0072] In some implementations, the classification loss of the first classifier can be directly used as the loss function of the first classifier. (Classification loss of the first classifier) The calculation formula is as follows:

[0073]

[0074] in It is the predicted category of the i-th visual entity. It is the category label of the i-th visual entity.

[0075] Optionally, based on the sample video and the corresponding unlocalized scene map, the category of each entity in the sample video is predicted, specifically including:

[0076] Based on the trajectory features of each entity in the sample video, the relationship features of each entity in the unlocalized scene map, and the Gated Recurrent Unit (GRU) network based on the attention mechanism, the relationship-aware visual features of each entity in the sample video are obtained.

[0077] The visual features that perceive the relationships between entities in the sample video are input into the first classifier to predict the category of each entity in the sample video.

[0078] The category labels for each entity in the sample video are determined as follows:

[0079] Based on the text features of each entity in the unlocalized scene graph, the relation features of each entity in the unlocalized scene graph, and the GRU network based on the attention mechanism, the relation-aware text features of each entity in the unlocalized scene graph are obtained.

[0080] For any entity in the sample video, calculate the similarity between the relation-aware visual features of the arbitrary entity and the relation-aware text features of each entity in the unlocalized scene graph, and select the text concept corresponding to the relation-aware text feature with the highest similarity to the relation-aware visual features of the arbitrary entity as the category label of the arbitrary entity.

[0081] Specifically, this invention provides an attention-based Relation-aware Interaction module (RAIM), which consists of a Gate Recurrent Unit (GRU) network based on an attention mechanism (Attention, Attn), and its formula is as follows:

[0082] RAIM(x,y)=GRU(x,Attn(x,y,y))

[0083] By inputting the trajectory features of each entity in the sample video and the relationship features of each entity in the unlocalized scene map into RAIM, the relationship-aware visual features of each entity in the sample video can be obtained, and the formula is as follows:

[0084] H = RAIM(F, P) G )

[0085] In the formula, F = {f1, f2, ..., f N} represents the trajectory features of each entity in the sample video (a total of N entities), H represents the visual features perceiving the relationships between each entity in the sample video, and P represents the trajectory features of each entity in the sample video. G This represents the relationship features of various entities in the unlocalized scene graph.

[0086] In some implementations, entity detection and extraction can be performed using third-party object detection models. For example, entities can be detected using a Fast Region-based Convolutional Neural Network (Fast-RCNN) model, and continuous bounding boxes can be generated for each entity using a Deep Simple Online and Real-time Tracking (Deep SORT) algorithm. Region of Interest (RoI) features for each entity are then obtained through RoI alignment. Finally, the bounding boxes and RoI features of each entity in the sample video are passed through a visual encoder to obtain the trajectory features of each entity in the sample video.

[0087] For example, using the Fast-RCNN model to detect N entities in a video, the Deep SORT algorithm is used to generate continuous bounding boxes for each entity. For the i-th entity, the bounding box of that entity is generated, denoted as . l i This indicates the duration of the entity's presence. Subsequently, the RoI feature of the entity is obtained using RoI Align, denoted as... Here d roi=2048 represents the dimension of the RoI feature. Subsequently, a visual encoder consisting of one-dimensional convolution and max pooling was designed to obtain the trajectory features of the entity, denoted as . Where d v =512 is the dimension of the trajectory feature.

[0088] After obtaining the relational visual features of each entity in the sample video, these features can be input into a first classifier to predict the category of each entity in the sample video. The first classifier can be a multilayer perceptron (MLP) based classifier. The formula for obtaining the predicted category is as follows:

[0089]

[0090] In the formula, Let H[i] be the predicted category of the i-th visual entity, and H[i] be the relational perception feature of the i-th visual entity.

[0091] You can also input the text features and relational features of each entity in the unlocalized scene graph into RAIM to obtain the relation-aware text features of each entity in the unlocalized scene graph. The formula is as follows:

[0092] Q = RAIM(C G ,P G )

[0093] In the formula, C G For the text features of each entity in the unlocalized scene graph, P G This represents the relationship features of various entities in the unlocalized scene graph.

[0094] For any entity in the sample video, as mentioned earlier, we can obtain the relation-aware visual features of that entity and the relation-aware text features of each entity in the unlocalized scene graph. Therefore, we can calculate the similarity between the relation-aware features of that entity and the relation-aware text features of each entity in the unlocalized scene graph, and select the text concept corresponding to the relation-aware text feature with the highest similarity as the category label of that entity.

[0095] In some implementations, the text features and relational features of each entity in the unlocalized scene graph can be obtained by embedding a Global Vectors for Word Representation embedding (GloVe embedding). The dimensions of the text features and relational features of each entity in the unlocalized scene graph can both be 300.

[0096] In some implementations, to better predict entity categories, the classification loss of the first classifier is combined with the triplet loss as the loss function of the first classifier. The triplet loss... The calculation formula is as follows:

[0097]

[0098] In the formula, dis(·) represents the calculation of Euclidean distance, and q i ∈Q and h i ∈H represents a pair of positive samples, both from the i-th visual entity; while q i and h j Let h represent a pair of negative samples. j From other visual entities and Δ represents a preset constant.

[0099] By obtaining the relation-aware visual features of each entity in the sample video and the relation-aware text features of each entity in the unlocalized scene map using RAIM, and then obtaining the category label and predicted category of each entity in the sample video, the entity category predicted by the trained first classifier can be made more accurate.

[0100] The unlocalized scene graph corresponding to the sample video contains relational information of entities in the sample video. The text concept dataset includes multiple entity category text concepts and multiple relation category text concepts. Therefore, based on the sample video, the unlocalized scene graph corresponding to the sample video, and the text concept dataset, a second classifier can be used to predict the classification probability of each subject-object pair in the sample video for the multiple relation categories, thus obtaining the classification prediction probability of each subject-object pair in the sample video.

[0101] Then, based on the classification prediction probability of each subject-object pair in the sample video and the true relationship label (i.e., the relationship information contained in the unlocalized scene graph), a second classifier can be trained using a loss function.

[0102] Optionally, based on sample videos, unlocalized scene graphs, and text concept datasets, the classification probabilities of multiple relation categories corresponding to each subject-object pair in the sample videos are predicted, including:

[0103] Determine the unbiased features of each text concept in the text concept dataset;

[0104] The subject category set and object category set of the sample video are determined based on the category labels of each entity in the sample video.

[0105] Based on the text features of each text concept in the text concept dataset, the unbiased features of each text concept in the text concept dataset, and the context-aware cue template, the text entity representations of the subject category set and the object category set are determined respectively.

[0106] Based on the text features of each text concept in the text concept dataset, the unbiased features of each text concept in the text concept dataset, and the trajectory features of each entity in the sample video, the visual entity representations of the subject category set and the object category set are determined respectively.

[0107] The text entity representations of the subject category set, the text entity representations of the object category set, the visual entity representations of the subject category set, and the visual entity representations of the object category set are input into the second classifier to predict the classification probability of multiple relation categories corresponding to each subject-object pair in the sample video.

[0108] Specifically, due to the weakly supervised training setting, the bias in the data distribution will be further amplified. Therefore, in order to better predict the relationship between visual entities, unbiased knowledge can be constructed based on the text concept dataset. After determining the unbiased features of each text concept in the text concept dataset, the relationship between visual entities can be predicted based on the unbiased features.

[0109] Optionally, determine the unbiased features of each text concept in the text concept dataset, specifically including:

[0110] Determine the co-occurrence frequency between any two text concepts in the text concept dataset, and the similarity between the text features of any two text concepts;

[0111] For any text concept, multiple text concepts are selected as the co-occurrence concept set of the arbitrary text concept by sorting them in ascending order of co-occurrence frequency with the arbitrary text concept, and multiple text concepts are selected as the semantic concept set of the arbitrary text concept by sorting them in ascending order of similarity with the text features of the arbitrary text concept.

[0112] Based on the set of co-occurring concepts of any text concept, the set of semantic concepts of any text concept, and the set of semantic concepts of each text concept in the set of co-occurring concepts of any text concept, determine the unbiased features of any text concept.

[0113] Specifically, all text concepts in the text concept dataset can be regarded as text concepts for constructing unbiased knowledge. The text features of each text concept can be obtained, and then the co-occurrence frequency between any two text concepts and the similarity between the text features of any two text concepts can be determined respectively.

[0114] For any given text concept, other text concepts can be sorted in ascending order of their co-occurrence frequency with the given text concept, and a preset number of text concepts can be selected as the co-occurring concepts of the given text concept to obtain the set of co-occurring concepts of the given text concept.

[0115] Similarly, a predetermined number of text concepts can be selected as semantic concepts of the text concept by sorting them in ascending order according to the similarity between the text features of other text concepts and the text features of the text concept, thus obtaining the semantic concept set of the text concept.

[0116] For example, the number of entity category text concepts and relation category text concepts in the text concept dataset are N, respectively. C and N P We can consider them all as textual concepts that construct unbiased knowledge, denoted as... N A =N C +N P Subsequently, the co-occurrence frequency among all text concepts was statistically analyzed to construct a co-occurrence matrix. Furthermore, obtain the textual features of all textual concepts, calculate the similarity between them, and construct a semantic similarity matrix. For the k-th text concept c k We can obtain information about its co-occurrence, A. co [k]. Sort them in ascending order and take the first K1 as their co-occurring concepts. Similarly, for the k-th text concept c k It is possible to obtain semantic information about it, A. sem [k]. Sort the k elements in ascending order and take the first K2 as their semantic concepts to obtain their semantic concepts.

[0117] After obtaining the co-occurring concept set and semantic concept set for each text concept, an unbiased concept graph can be obtained based on the co-occurring concept set, the semantic concept set, and the semantic concept sets of each text concept in the co-occurring concept set. Then, the unbiased concept graph is input through a graph message passing network, such as a graph convolutional nueral network (GCN), to determine the unbiased features of the text concept.

[0118] Figure 2 A schematic diagram of the unbiased conceptual diagram provided by the present invention, such as Figure 2 As shown, for any text concept c k c k With c k The concept of co-occurrence c co and ck semantic concept c sim Connection, c k The concept of co-occurrence c co With co-occurrence concept c co Each of their respective semantic concepts c cs Connect to form an unbiased concept map

[0119] After inputting the unbiased concept graph into the graph message passing network, the root node (i.e., the text concept c) is obtained. k Unbiased characteristics

[0120] Then, the subject category set and object category set of the sample video can be determined based on the category labels of each entity in the sample video. For example, the category label of each entity in the sample video can be used as the subject category, or the category label of each entity in the sample video can be used as the object category. Alternatively, the category labels of some entities in the sample video can be selected as the subject category, and the category labels of some entities in the sample video can be used as the object category.

[0121] Understandably, the text concept dataset includes text concepts of multiple entity categories. Therefore, after determining the subject category set and object category set of the sample videos, the text features of each category in the subject category set and the text features of each category in the object category set can be determined based on the text features of each text concept in the text concept dataset. Furthermore, the unbiased features of each category in the subject category set and the unbiased features of each category in the object category set can be determined based on the unbiased features of each text concept in the text concept dataset.

[0122] Then, by combining the text features and unbiased features of each category in the subject category set, and using a context-aware cue template, the text entity representation of the subject category set can be determined. Similarly, by combining the text features and unbiased features of each category in the object category set, and using a context-aware cue template, the text entity representation of the object category set can be determined.

[0123] In some implementations, the context-aware prompt template formula is as follows:

[0124] f W (CLASS) = [w1, w2, ..., w L ,CLASS]

[0125] Here, w l (l∈1,2,..,L) are the learnable word vectors, [CLASS] represents the text features, and L is the total number of learnable word vectors.

[0126] Optionally, the text entity representations of the subject category set and the object category set are determined separately, specifically including:

[0127] Determine the context-aware cue templates for the subject and the context-aware cue templates for the object respectively;

[0128] Based on the text features of the subject category set, the unbiased features of the subject category set, the context-aware cue template of the subject, and the bidirectional long short-term memory network, the text entity representation of the subject category set is determined.

[0129] Based on the text features of the object category set, the unbiased features of the object category set, the context-aware cue template of the object, and the bidirectional long short-term memory network, the text entity representation of the object category set is determined.

[0130] Specifically, considering the different contexts of the subject and object in a triplet, different context-aware prompt templates can be designed for the subject and object respectively, denoted as... and

[0131] The text features and unbiased features of the subject category set can be combined and fed into a bidirectional long short-term memory (Bi-LSTM) network after passing through the context-aware cue template of the subject. The Bi-LSTM can be used as a text encoder to obtain the context-aware unbiased text features of each subject category based on cue learning, which can then be used as the text entity representation of the subject category.

[0132] Similarly, the text features and unbiased features of the object category set can be combined with the context-aware cue template of the object, and then input into a Bi-LSTM to obtain the context-aware unbiased text features of each object category based on cue learning, which can be used as the text entity representation of the object category.

[0133] For example, the subject category set is The object category set is Based on the unbiased features of the text concept dataset, the unbiased features of the subject category set and the object category set can be determined separately, respectively. and Then, a bidirectional long short-term memory network (Bi-LSTM) is used as the text encoder to generate context-aware unbiased text features S and O for each subject and object category, based on cue learning, as shown in the following formula:

[0134]

[0135]

[0136] In the formula, represents the text features obtained by GloVe embedding for the subject category and object category, respectively, and [;] represents the concatenation operation.

[0137] For visual representation learning, in order to learn the hidden semantic relationships between visual entities and between visual entities and text concepts, it is necessary to link the features of text concepts in the text concept dataset with the features of visual entities. Therefore, based on the text features of each text concept in the text concept dataset, the unbiased features of each text concept in the text concept dataset, and the trajectory features of each entity in the sample video, the visual entity representations of the subject category set and the object category set can be determined respectively.

[0138] Optionally, the visual entity representations of the subject category set and the object category set are determined separately, specifically including:

[0139] Determine the first fully connected concept graph; the nodes in the first fully connected concept graph represent text concepts in the text concept dataset;

[0140] The entities corresponding to the subject category set and the entities corresponding to the object category set are respectively embedded as embedding nodes into the first fully connected concept graph to obtain the second fully connected concept graph corresponding to the subject category set and the third fully connected concept graph corresponding to the object category set.

[0141] Based on the second and third fully connected concept maps, the visual entity features of the subject category set and the visual entity features of the object category set are determined respectively.

[0142] The features of nodes in the first fully connected conceptual graph are determined based on the text features and unbiased features corresponding to the nodes in the first fully connected conceptual graph; the features of embedded nodes are determined based on the trajectory features, text features, and unbiased features corresponding to the embedded nodes.

[0143] Specifically, we can first assume that all text concepts are semantically related. Therefore, we can construct a first fully connected concept graph based on all text concepts in the text concept dataset. Each node in the first fully connected concept graph represents a text concept in the text concept dataset, and the features on the node can be the text features and unbiased features of that text concept obtained through a fully connected layer. The adjacency matrix of the first fully connected concept graph can be initialized using a semantic similarity matrix, that is, the edge weight between any two nodes corresponds to the similarity between the text features of the text concepts corresponding to those two nodes.

[0144] For example, the adjacency matrix of the first fully connected conceptual graph is denoted as... Each node represents a textual concept, where A is an example. base Using semantic similarity matrix A sem Perform initialization. For the k-th node, the text concept it represents is c. k Then the feature of the node is c′ k =φ u ([c k ; Π(c k )]), φ u Represents a fully connected layer, c k c k Textual features, Π(c k ) represents c k Its unbiased characteristics.

[0145] Then, the entities corresponding to the subject category set can be embedded into the first fully connected concept graph to obtain the second fully connected concept graph. The features of the embedded nodes can be the text features of the subject category, the unbiased features of the subject category, and the trajectory features corresponding to the entity, obtained through the fully connected layer.

[0146] Similarly, entities corresponding to the object category set can be embedded into the first fully connected concept graph to obtain the third fully connected concept graph. The features of the embedded nodes can be the text features of the object category, the unbiased features of the object category, and the trajectory features corresponding to the entity, obtained through the fully connected layer.

[0147] The edge weights between any embedded node and nodes in the first fully connected concept graph can be initialized based on the similarity between the text features of the text concepts corresponding to these two nodes; while the edge weights between embedded nodes can be initialized based on the cosine similarity between the features of these two embedded nodes.

[0148] For example, the i-th embedded node represents the category as follows: The characteristics of embedded nodes are φ v f represents a fully connected layer i It is the trajectory feature of the entity. yes Textual features, yes The unbiased features. Construct a learnable weight matrix. For the i-th embedded node, based on its category label Obtain the index idx of this category in the text concept dataset C. i Then the adjacency matrix of the embedded node with other concepts is A. pro [idx i ] Then, Apro [idx i spliced ​​to A base This results in an extended graph, whose adjacency matrix is ​​denoted as... N is the number of visual entities in the video (i.e., the number of subject category sets or the number of object sets). In A extend In this context, the weights of edges between visual entities are initialized using the cosine similarity between node features.

[0149] After obtaining the second and third fully connected concept graphs, the second and third fully connected concept graphs can be passed through graph message passing networks, such as graph attention networks (GAT), to obtain visual entity representations of the subject category set and the object category set, respectively.

[0150] For example, the knowledge-based visual entity representation obtained by using GAT from a fully connected concept graph is... N is the number of visual entities in the video (i.e., the number of subject category sets or the number of object sets), d v Let F' be the dimension of the trajectory features. The visual entity representations of the subject category set and the object category set can be denoted as F′, respectively. sub and F′ obj .

[0151] After determining the text entity representations of the subject category set, the text entity representations of the object category set, the visual entity representations of the subject category set, and the visual entity representations of the object category set, they can be passed to a second classifier to predict the relationship between each subject-object pair. When the second classifier is an MLP-based classifier, the formula for obtaining the classification prediction probability of each subject-object pair is as follows:

[0152] R pred =MLP([S;F′) sub ;O;F′ obj ])

[0153] N represents the classification prediction probability of the relation of the j-th subject-object pair. R The number of subject category sets and object category sets, N P This represents the number of relational text concepts in the text concept dataset.

[0154] Then, a second classifier can be trained based on the true relation labels and classification prediction probabilities corresponding to each subject-object pair in the sample video.

[0155] In some implementations, the classification loss of the second classifier is used as the loss function of the second classifier, and the formula for the classification loss of the second classifier is as follows:

[0156]

[0157] In the formula, y j The true relation label for the j-th subject-object pair. N represents the classification prediction probability of the relation of the j-th subject-object pair. R The number of subject category sets and object category sets, N P This represents the number of relational text concepts in the text concept dataset.

[0158] It is understandable that when the first and second classifiers are integrated into the weakly supervised video scene graph generation model provided by this invention for training, the loss function can be optimized. The optimized loss function is as follows:

[0159]

[0160] The weakly supervised video scene graph generation apparatus provided by the present invention is described below. The weakly supervised video scene graph generation apparatus described below and the weakly supervised video scene graph generation method described above can be referred to in correspondence with each other.

[0161] Figure 3 This is a schematic diagram of the structure of the weakly supervised video scene graph generation device provided by the present invention, as shown below. Figure 3 As shown, the device includes:

[0162] The generation module 300 is used to input the target video into the trained weakly supervised video scene graph generation model to obtain the video scene graph corresponding to the target video.

[0163] The weakly supervised video scene graph generation model is trained based on a sample video set, an unlocalized scene graph set corresponding to the sample video set, and a text concept dataset.

[0164] Optionally, training methods for weakly supervised video scene graph generation models include:

[0165] Select sample videos from the sample video set;

[0166] Based on the sample video and the corresponding unlocalized scene map, the category of each entity in the sample video is predicted to obtain the predicted category of each entity in the sample video.

[0167] The first classifier of the weakly supervised video scene graph generation model is trained based on the category labels and predicted categories of each entity in the sample video.

[0168] Based on sample videos, unlocalized scene graphs, and text concept datasets, the classification probabilities of multiple relation categories corresponding to each subject-object pair in the sample videos are predicted, thus obtaining the classification prediction probabilities of each subject-object pair in the sample videos.

[0169] The second classifier of the weakly supervised video scene graph generation model is trained based on the real relation labels and classification prediction probabilities corresponding to each subject-object pair in the sample video.

[0170] Optionally, based on sample videos, unlocalized scene graphs, and text concept datasets, the classification probabilities of multiple relation categories corresponding to each subject-object pair in the sample videos are predicted, including:

[0171] Determine the unbiased features of each text concept in the text concept dataset;

[0172] The subject category set and object category set of the sample video are determined based on the category labels of each entity in the sample video.

[0173] Based on the text features of each text concept in the text concept dataset, the unbiased features of each text concept in the text concept dataset, and the context-aware cue template, the text entity representations of the subject category set and the object category set are determined respectively.

[0174] Based on the text features of each text concept in the text concept dataset, the unbiased features of each text concept in the text concept dataset, and the trajectory features of each entity in the sample video, the visual entity representations of the subject category set and the object category set are determined respectively.

[0175] The text entity representations of the subject category set, the text entity representations of the object category set, the visual entity representations of the subject category set, and the visual entity representations of the object category set are input into the second classifier to predict the classification probability of multiple relation categories corresponding to each subject-object pair in the sample video.

[0176] Optionally, the text entity representations of the subject category set and the object category set are determined separately, specifically including:

[0177] Determine the context-aware cue templates for the subject and the context-aware cue templates for the object respectively;

[0178] Based on the text features of the subject category set, the unbiased features of the subject category set, the context-aware cue template of the subject, and the bidirectional long short-term memory network, the text entity representation of the subject category set is determined.

[0179] Based on the text features of the object category set, the unbiased features of the object category set, the context-aware cue template of the object, and the bidirectional long short-term memory network, the text entity representation of the object category set is determined.

[0180] Optionally, the visual entity representations of the subject category set and the object category set are determined separately, specifically including:

[0181] Determine the first fully connected concept graph; the nodes in the first fully connected concept graph represent text concepts in the text concept dataset;

[0182] The entities corresponding to the subject category set and the entities corresponding to the object category set are respectively embedded as embedding nodes into the first fully connected concept graph to obtain the second fully connected concept graph corresponding to the subject category set and the third fully connected concept graph corresponding to the object category set.

[0183] Based on the second and third fully connected concept maps, the visual entity features of the subject category set and the visual entity features of the object category set are determined respectively.

[0184] The features of nodes in the first fully connected conceptual graph are determined based on the text features and unbiased features corresponding to the nodes in the first fully connected conceptual graph; the features of embedded nodes are determined based on the trajectory features, text features, and unbiased features corresponding to the embedded nodes.

[0185] Optionally, determine the unbiased features of each text concept in the text concept dataset, specifically including:

[0186] Determine the co-occurrence frequency between any two text concepts in the text concept dataset, and the similarity between the text features of any two text concepts;

[0187] For any text concept, multiple text concepts are selected as the co-occurrence concept set of the arbitrary text concept by sorting them in ascending order of co-occurrence frequency with the arbitrary text concept, and multiple text concepts are selected as the semantic concept set of the arbitrary text concept by sorting them in ascending order of similarity with the text features of the arbitrary text concept.

[0188] Based on the set of co-occurring concepts of any text concept, the set of semantic concepts of any text concept, and the set of semantic concepts of each text concept in the set of co-occurring concepts of any text concept, determine the unbiased features of any text concept.

[0189] Optionally, based on the sample video and the corresponding unlocalized scene map, the category of each entity in the sample video is predicted, specifically including:

[0190] Based on the trajectory features of each entity in the sample video, the relationship features of each entity in the unlocalized scene map, and the Gated Recurrent Unit (GRU) network based on the attention mechanism, the relationship-aware visual features of each entity in the sample video are obtained.

[0191] The visual features that perceive the relationships between entities in the sample video are input into the first classifier to predict the category of each entity in the sample video.

[0192] The category labels for each entity in the sample video are determined as follows:

[0193] Based on the text features of each entity in the unlocalized scene graph, the relation features of each entity in the unlocalized scene graph, and the GRU network based on the attention mechanism, the relation-aware text features of each entity in the unlocalized scene graph are obtained.

[0194] For any entity in the sample video, calculate the similarity between the relation-aware visual features of the arbitrary entity and the relation-aware text features of each entity in the unlocalized scene graph, and select the text concept corresponding to the relation-aware text feature with the highest similarity to the relation-aware visual features of the arbitrary entity as the category label of the arbitrary entity.

[0195] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430, and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 can call logical instructions in the memory 430 to execute a weakly supervised video scene graph generation method, which includes:

[0196] The target video is input into a trained weakly supervised video scene graph generation model to obtain the video scene graph corresponding to the target video.

[0197] The weakly supervised video scene graph generation model is trained based on a sample video set, an unlocalized scene graph set corresponding to the sample video set, and a text concept dataset.

[0198] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0199] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the weakly supervised video scene graph generation method provided by the above methods, the method comprising:

[0200] The target video is input into a trained weakly supervised video scene graph generation model to obtain the video scene graph corresponding to the target video.

[0201] The weakly supervised video scene graph generation model is trained based on a sample video set, an unlocalized scene graph set corresponding to the sample video set, and a text concept dataset.

[0202] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the weakly supervised video scene graph generation method provided by the methods described above, the method comprising:

[0203] The target video is input into a trained weakly supervised video scene graph generation model to obtain the video scene graph corresponding to the target video.

[0204] The weakly supervised video scene graph generation model is trained based on a sample video set, an unlocalized scene graph set corresponding to the sample video set, and a text concept dataset.

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating weakly supervised video scene graphs, characterized in that, include: The target video is input into a trained weakly supervised video scene graph generation model to obtain the video scene graph corresponding to the target video. The weakly supervised video scene graph generation model is trained based on a sample video set, an unlocalized scene graph set corresponding to the sample video set, and a text concept dataset. The training method for the weakly supervised video scene graph generation model includes: Select sample videos from the sample video set; Based on the sample video and the corresponding unlocalized scene map, the category of each entity in the sample video is predicted to obtain the predicted category of each entity in the sample video. The first classifier of the weakly supervised video scene graph generation model is trained based on the category labels of each entity in the sample video and the predicted category. Based on the sample video, the unlocalized scene graph, and the text concept dataset, the classification probability of multiple relation categories corresponding to each subject-object pair in the sample video is predicted to obtain the classification prediction probability of each subject-object pair in the sample video. The second classifier of the weakly supervised video scene graph generation model is trained based on the real relation labels corresponding to each subject-object pair in the sample video and the classification prediction probability. The step of predicting the classification probabilities of multiple relation categories corresponding to each subject-object pair in the sample video, based on the sample video, the unlocalized scene graph, and the text concept dataset, includes: Determine the unbiased features of each text concept in the text concept dataset; The subject category set and the object category set of the sample video are determined based on the category labels of each entity in the sample video. Based on the text features of each text concept in the text concept dataset, the unbiased features of each text concept in the text concept dataset, and the context-aware cue template, the text entity representations of the subject category set and the object category set are determined respectively. Based on the text features of each text concept in the text concept dataset, the unbiased features of each text concept in the text concept dataset, and the trajectory features of each entity in the sample video, the visual entity representations of the subject category set and the object category set are determined respectively. The text entity representations of the subject category set, the text entity representations of the object category set, the visual entity representations of the subject category set, and the visual entity representations of the object category set are input into the second classifier to predict the classification probability of multiple relation categories corresponding to each subject-object pair in the sample video. The step of predicting the category of each entity in the sample video based on the sample video and the corresponding unlocalized scene map specifically includes: Based on the trajectory features of each entity in the sample video, the relationship features of each entity in the unlocalized scene map, and the Gated Recurrent Unit (GRU) network based on the attention mechanism, the relationship-aware visual features of each entity in the sample video are obtained. The visual features of the relationships between the entities in the sample video are input into the first classifier to predict the category of each entity in the sample video. The category labels of each entity in the sample video are determined according to the following method: Based on the text features of each entity in the unlocated scene graph, the relation features of each entity in the unlocated scene graph, and the GRU network based on the attention mechanism, the relation-aware text features of each entity in the unlocated scene graph are obtained. For any entity in the sample video, calculate the similarity between the relation-aware visual features of the arbitrary entity and the relation-aware text features of each entity in the unlocalized scene graph, and select the text concept corresponding to the relation-aware text feature with the highest similarity to the relation-aware visual features of the arbitrary entity as the category label of the arbitrary entity.

2. The method for generating weakly supervised video scene graphs according to claim 1, characterized in that, The step of determining the text entity representations of the subject category set and the object category set respectively specifically includes: Determine the context-aware cue templates for the subject and the context-aware cue templates for the object respectively; Based on the text features of the subject category set, the unbiased features of the subject category set, the context-aware cue template of the subject, and the bidirectional long short-term memory network, the text entity representation of the subject category set is determined. Based on the text features of the object category set, the unbiased features of the object category set, the context-aware cue template of the object, and the bidirectional long short-term memory network, the text entity representation of the object category set is determined.

3. The method for generating weakly supervised video scene graphs according to claim 2, characterized in that, The step of determining the visual entity representations of the subject category set and the object category set respectively specifically includes: A first fully connected concept graph is determined; the nodes in the first fully connected concept graph represent text concepts in the text concept dataset. The entities corresponding to the subject category set and the entities corresponding to the object category set are respectively embedded as embedding nodes into the first fully connected concept graph to obtain the second fully connected concept graph corresponding to the subject category set and the third fully connected concept graph corresponding to the object category set; Based on the second fully connected concept map and the third fully connected concept map, the visual entity features of the subject category set and the visual entity features of the object category set are determined respectively. The features of the nodes in the first fully connected conceptual graph are determined based on the text features and unbiased features corresponding to the nodes in the first fully connected conceptual graph; the features of the embedded nodes are determined based on the trajectory features, text features, and unbiased features corresponding to the embedded nodes.

4. The method for generating weakly supervised video scene graphs according to any one of claims 1 to 3, characterized in that, Determining the unbiased features of each text concept in the text concept dataset specifically includes: Determine the co-occurrence frequency between any two text concepts in the text concept dataset, and the similarity between the text features of any two text concepts; For any text concept, multiple text concepts are selected as the co-occurrence concept set of the arbitrary text concept in ascending order of their co-occurrence frequency with the arbitrary text concept, and multiple text concepts are selected as the semantic concept set of the arbitrary text concept in ascending order of their similarity to the text features of the arbitrary text concept. Based on the set of co-occurring concepts of the arbitrary text concept, the set of semantic concepts of the arbitrary text concept, and the set of semantic concepts of each text concept in the set of co-occurring concepts of the arbitrary text concept, the unbiased features of the arbitrary text concept are determined.

5. A weakly supervised video scene graph generation device, characterized in that, include: The generation module is used to input the target video into a trained weakly supervised video scene graph generation model to obtain the video scene graph corresponding to the target video. The weakly supervised video scene graph generation model is trained based on a sample video set, an unlocalized scene graph set corresponding to the sample video set, and a text concept dataset. The training method for the weakly supervised video scene graph generation model includes: Select sample videos from the sample video set; Based on the sample video and the corresponding unlocalized scene map, the category of each entity in the sample video is predicted to obtain the predicted category of each entity in the sample video. The first classifier of the weakly supervised video scene graph generation model is trained based on the category labels of each entity in the sample video and the predicted category. Based on the sample video, the unlocalized scene graph, and the text concept dataset, the classification probability of multiple relation categories corresponding to each subject-object pair in the sample video is predicted to obtain the classification prediction probability of each subject-object pair in the sample video. The second classifier of the weakly supervised video scene graph generation model is trained based on the real relation labels corresponding to each subject-object pair in the sample video and the classification prediction probability. The step of predicting the classification probabilities of multiple relation categories corresponding to each subject-object pair in the sample video, based on the sample video, the unlocalized scene graph, and the text concept dataset, includes: Determine the unbiased features of each text concept in the text concept dataset; The subject category set and the object category set of the sample video are determined based on the category labels of each entity in the sample video. Based on the text features of each text concept in the text concept dataset, the unbiased features of each text concept in the text concept dataset, and the context-aware cue template, the text entity representations of the subject category set and the object category set are determined respectively. Based on the text features of each text concept in the text concept dataset, the unbiased features of each text concept in the text concept dataset, and the trajectory features of each entity in the sample video, the visual entity representations of the subject category set and the object category set are determined respectively. The text entity representations of the subject category set, the text entity representations of the object category set, the visual entity representations of the subject category set, and the visual entity representations of the object category set are input into the second classifier to predict the classification probability of multiple relation categories corresponding to each subject-object pair in the sample video. The step of predicting the category of each entity in the sample video based on the sample video and the corresponding unlocalized scene map specifically includes: Based on the trajectory features of each entity in the sample video, the relationship features of each entity in the unlocalized scene map, and the Gated Recurrent Unit (GRU) network based on the attention mechanism, the relationship-aware visual features of each entity in the sample video are obtained. The visual features of the relationships between the entities in the sample video are input into the first classifier to predict the category of each entity in the sample video. The category labels of each entity in the sample video are determined according to the following method: Based on the text features of each entity in the unlocated scene graph, the relation features of each entity in the unlocated scene graph, and the GRU network based on the attention mechanism, the relation-aware text features of each entity in the unlocated scene graph are obtained. For any entity in the sample video, calculate the similarity between the relation-aware visual features of the arbitrary entity and the relation-aware text features of each entity in the unlocalized scene graph, and select the text concept corresponding to the relation-aware text feature with the highest similarity to the relation-aware visual features of the arbitrary entity as the category label of the arbitrary entity.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the weakly supervised video scene graph generation method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the weakly supervised video scene graph generation method as described in any one of claims 1 to 4.

Images