Open vocabulary video scene graph generation method based on joint region perception semantic alignment

Abstract

The invention provides a method for generating an open vocabulary video scene graph by combining region perception semantic alignment, which comprises the following steps of: considering visual information and text information of videos, carrying out video semantic learning on the videos in a training set, and carrying out object track classification of open vocabularies; learning open vocabulary knowledge in a distillation mode; object visual information, object motion information and a visual joint area of the object are constructed, and comprehensive context semantic understanding is carried out; combining the semantic perception representation obtained in the previous step with a learnable relationship token to generate visual relationship representation and perform relationship prediction; and all the parts are integrated into a unified framework, and overall model training is carried out. According to the method, the alignment between the object track joint region and the relation predicate is clearly explored by combining the region perception semantic alignment network, so that more comprehensive relation prediction is carried out.

Description

Technical Field

[0001] The invention belongs to the field of video scene understanding and content analysis, and in particular relates to an open vocabulary video scene graph generation method with joint region-aware semantic alignment. Background Art

[0002] The video scene graph generation (VidSGG) task aims to detect and localize visual relations between different entity trajectories in a given video, and construct these relations as relation triplets in the form of <subject-relation-object>. It plays a key role in various visual understanding tasks, enhancing the understanding of videos by providing structured knowledge.

[0003] Existing video scene graph generation methods are still limited to recognizing objects and predicting visual relations in closed-set scenes, which makes them unable to be directly applied to real-world scenes. Because real-world scenes often contain more object and relation categories than the model training dataset. Therefore, it is necessary to explore open-vocabulary video scene graph generation methods. Benefiting from the encyclopedic knowledge of popular visual language models (VLMs), such as CLIP, existing open-vocabulary scene graph generation methods can easily recognize new objects and explore new relations through object-level alignment between visual objects and textual category labels. However, these open-vocabulary scene graph methods overly rely on the aligned visual and textual object features provided by pre-trained VLMs for model learning, while ignoring the alignment between visual relation regions and relation predicate concepts. Visual relation regions are represented as joint regions of two objects and are usually considered as an auxiliary supplement for relation prediction. In fact, most existing methods simply fuse this joint region representation with object features without further exploration. In addition, as shown in previous studies, VLMs (such as CLIP) still face challenges in performing combinatorial scene understanding, such as relations between objects. This leads to incomplete and unreliable alignment between object pair (subject-object) representations and relation predicate representations, and results in ambiguous relation predictions.

[0004] Therefore, in order to build an open-vocabulary video scene graph generation algorithm, the following two aspects need to be considered: (1) In order to avoid dependence on object representations extracted by VLMs, it is necessary to explore the association between visual joint regions and the relation predicates that need to be predicted; (2) In order to make the model have open-vocabulary capabilities, a reasonable distillation method is needed to inject rich knowledge into the model. Summary of the invention

[0005] To solve the above technical problems, the present invention provides a joint region-aware semantically aligned open vocabulary video scene graph generation method, which is an open vocabulary video scene graph generation algorithm that predicts visual relations by explicitly modeling the association between visual joint regions and relation predicates. In addition, a suitable distillation method also needs to be considered to achieve the prediction effect of open vocabulary.

[0006] In order to achieve the above object, the present invention adopts the following technical solution: An open vocabulary video scene graph generation algorithm with joint region-aware semantic alignment includes the following steps: Step S1: Considering the visual information and text information of the video, video semantic learning is performed on the videos in the training set, and object trajectory classification of open vocabulary is performed; Step S2: learning open vocabulary knowledge through distillation; Step S3: construct object visual information, object motion information and object visual joint area to perform comprehensive contextual semantic understanding; Step S4: using the semantic perception representation obtained in the previous step combined with the learnable relationship token to generate a visual relationship representation and perform relationship prediction; Step S5: Integrate steps S1 to S4 into a unified framework and perform overall training of the model of the unified framework.

[0007] The beneficial effects of the present invention are: We jointly use a region-aware semantic alignment network to explicitly explore the alignment between object trajectory joint regions and relation predicates for more comprehensive relation prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flow chart of the open vocabulary video scene graph generation method based on joint region-aware semantic alignment proposed by the present invention. DETAILED DESCRIPTION

[0009] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0010] The open vocabulary video scene graph generation method with joint region-aware semantic alignment proposed in the present invention constructs visual joint regions, and generates visual relationship representations through context awareness and joint region alignment to predict relationships, namely, including: 1) constructing a visual refiner, transferring knowledge into the framework of this algorithm through a distillation method to obtain a visually perceptual representation; 2) using a semantic-aware context encoder to achieve comprehensive semantic-aware interaction; 3) using a joint relationship alignment decoder to integrate semantic-aware representations and generate distinguishing relationship tags.

[0011] like Figure 1 As shown, the open vocabulary video scene graph generation method of the present invention with joint region-aware semantic alignment includes the following steps: Step S1: Considering the visual information and text information of the video, video semantic learning is performed on the videos in the training set, and object trajectory classification of open vocabulary is performed; Step S2: learning open vocabulary knowledge through distillation; Step S3: construct object visual information, object motion information and object visual joint area to perform comprehensive contextual semantic understanding; Step S4: using the semantic perception representation obtained in the previous step combined with the learnable relationship token to generate a visual relationship representation and perform relationship prediction; Step S5: Integrate steps S1 to S4 into a unified framework and perform overall training of the model of the unified framework.

[0012] Specifically, the step S1 includes: Step S11: In order to make the model have the ability to generalize to unknown classes, the present invention performs open vocabulary model training. Specifically, for the objects and relationship categories contained in the data set, the present invention divides them into basic categories and new categories respectively: for objects, Represents the basic class, Represents a new class. For relations, Represents the basic class, Represents a new class. Note that only labeled data involving basic classes are used during training.

[0013] Step S12: For each video in the training data set, a third-party pre-trained model is used to detect and extract visual entities and obtain entities, and their trajectories are .in For the Here, we use the VinVL model based on Fast-RCNN to generate the entity detection box, and use the Seq-NMS algorithm to generate the entity trajectory. entity, generate the trajectory of the entity, denoted as ,in, is the detection box of the object. Then, the ViT (Visual Transformer) feature is used to obtain the features of the entity , denoted as ,here Represents the dimension of ViT features.

[0014] Step S13: To explore the alignment between visual joint regions and predicate concepts, extract the joint regions of <subject-object> pairs in each video clip. Yes, among them The visual joint region is the union of the bounding boxes of each pair of visual entities represented by <subject-object>. The present invention also uses pre-trained ViT features to extract visual joint region features. , denoted as In addition, for each pair of <subject-object>, the present invention also obtains its motion information, which is recorded as , and use a mapping function to map it to the visual feature space of the entity: ,in . It is the object motion information obtained through a third-party algorithm. Refers to a multi-layer perceptron, It is the feature of motion information after passing through a multi-layer perceptron. No specific meaning, used to indicate that this feature is associated with a pair of <subject-object> Related.

[0015] Step S14: Pre-train a bridge encoder to perform open vocabulary trajectory classification training. Specifically, a bridge encoder based on the Transformer structure is constructed, denoted as BridgeEnc. Then, the visual features of each trajectory are obtained. And the category text features WordEmb refers to a word embedding map, which obtains the word embedding features of each category by inputting the category text.

[0016] Then, the object trajectory classification task is used to pre-train it. , first obtain its classification probability : ; in, and is the text representation of the category, c represents the category name, The text features representing the category; Indicates the category name, The text features representing the category; represents a hyperparameter.

[0017] The loss function of the final trajectory classification task is recorded as , exp represents the exponential function.

[0018] Specifically, the step S2 includes: A visual refiner is constructed to transfer the knowledge in the pre-trained encoder to the framework of the present invention through a distillation strategy, prompting it to pay more attention to visual-related semantic information.

[0019] First, we use manual annotation to obtain a small amount of accurate object trajectory annotations. , denoted as In order to obtain a visually perceptual representation of each object trajectory, a set of learnable representations is used here And modules based on cross-attention mechanisms, is a one-dimensional learnable representation Elements of is the number of all learnable vectors, as follows: ; ; Among them, among them, It is obtained by the following formula: The self-attention representation of Depend on Expanded, . Represents the attention module used.

[0020] The final distillation loss is for: ; in, . Refers to the 2-norm operation.

[0021] Specifically, step S3 includes: First, similar to step S2, The process of obtaining the trajectory characteristics of visual perception and joint regional characteristics . Since the pre-trained bridge encoder has the ability to connect visual and textual modalities, and this ability has been distilled into the visual refiner. Therefore, the previously obtained object trajectory features and joint region features also contain textual semantic information. Here, a semantically aware context encoder (SACEncoder) is used to obtain a representation of comprehensive semantic interaction. Specifically, according to the previously obtained <subject-object> pairs, where the subject object trajectory features and object object trajectory features are and The subject, object, visual joint region representation, and object motion information are then concatenated into a sequence to simulate the structure of relational triples. In order to reduce the influence of the order of the components in the sequence, the SACEncoder based on the attention mechanism is used to combine them with a learnable representation. Interact to obtain context-aware representations: ; in, Refers to the features obtained by simulating the triple structure above, is a contextual semantically aware representation, is the object motion information obtained in step S13.

[0022] ; in, is a multi-layer perceptron (MLP), Represents a normalization operation.

[0023] Specifically, step S4 includes the following steps: Step S41: After aggregating the contextual semantics, a joint region-aware relation representation is generated for the final relation classification. Therefore, a joint region relation alignment decoder (URADecoder) based on a multi-head attention mechanism is designed: ; in, Through learnable representation The features obtained by integrating the contextual semantics belong to the intermediate process; ; in, is a multi-head attention module. Represents a multi-layer perceptron (MLP). With the help of this decoder, a joint region-aware relational representation can be generated Used for subsequent relationship prediction.

[0024] Step S42: Unlike traditional classification tasks, in open vocabulary tasks, the number of categories that need to be predicted in the training and testing phases is uncertain, so this type of task relies on pre-extracted category representations. In order to obtain better relationship representations, a prompt learning method is introduced here. Specifically, some learnable word vectors are used as prompt tags for each relationship category, and the relationship category representation is obtained through a pre-trained bridge encoder: ; Among them, WordEmb refers to a word embedding mapping, which obtains the word embedding features of each category by inputting the category text; Refers to the prompt representation constructed for the relation category c; represents the number of word vectors that can be learned in each prompt representation; ; in, It is the learnable word vector that is set. Is the relationship type Category representation.

[0025] Step S43: In order to fully consider the initial information and the semantically perceived joint region information, the representations obtained in the previous steps are connected and input into the relationship classifier for relationship prediction: ; Among them, R is the representation of the relationship between objects obtained; is from The encoded joint region representation obtained in is a multi-layer perceptron (MLP). For each <subject-object> pair (here denoted as ), the relationship between them is represented as Similar to step S14, here we calculate for each relationship category The predicted probability : ; in, For any <subject-object> pair , features that represent the visual relationship between them; For any relation category c, its text representation Similarly, for relationship categories text representation of .

[0026] Finally, the same classification loss function as in the trajectory classification task is used, and the loss of relation classification is recorded as .

[0027] Specifically, step S5 includes: In order to supplement the joint region-level relationship prediction process, the traditional video scene graph generation loss is used for auxiliary training. The specific process is: directly concatenate the trajectory features of the two in the <subject-object> pair as their relationship features, and use the prompt learning strategy to generate a relationship representation similar to that in step S42 to perform the final relationship category prediction. The loss here is recorded as . Therefore, the final training loss for: .

[0028] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating open vocabulary video scene graphs with region-aware semantic alignment, characterized in that: The steps include: Step S1: Considering the visual information and text information of the video, video semantic learning is performed on the videos in the training set, and object trajectory classification of open vocabulary is performed; Step S2: learning open vocabulary knowledge through distillation; Step S3: construct object visual information, object motion information and object visual joint area to perform comprehensive contextual semantic understanding; Step S4: using the semantic perception representation obtained in the previous step combined with the learnable relationship token to generate a visual relationship representation and perform relationship prediction; Step S5: Integrate steps S1 to S4 into a unified framework and perform overall training of the model of the unified framework.

2. The open vocabulary video scene graph generation method with joint region-aware semantic alignment according to claim 1, characterized in that: The step S1 comprises: Step S11: Perform open vocabulary model training, including: classifying the objects and relationship categories contained in the data set into basic categories and new categories: For objects, Represents the basic class, represents a new class; for relations, Represents the basic class, Represent new classes; only labeled data involving basic classes are used during training; Step S12: For each video in the training data set, a third-party pre-trained model is used to detect and extract visual entities and obtain entities, and their trajectories are ,in For the The trajectory of the entity is obtained by using the VinVL model based on Fast-RCNN to generate the detection box of the entity and the Seq-NMS algorithm to generate the trajectory of the entity. Then, the ViT feature, i.e. the visual Transformer feature, is used to obtain the feature of the entity. , denoted as ,here Represents the dimension of ViT features; Step S13: Extract the joint region of the <subject-object> pair in each video segment and construct Yes, among them ; The visual joint region is the union of the bounding boxes of each pair of visual entities represented by <subject-object>, and the pre-trained ViT features are used to extract the visual joint region features , denoted as ; Get each pair of <subject-object> (denoted as ) is recorded as , and use a mapping function to map it to the visual feature space of the entity: ,in , It is the object motion information obtained through a third-party algorithm. Refers to a multi-layer perceptron, It is the feature of motion information after passing through a multi-layer perceptron. No specific meaning, used to indicate that this feature is associated with a pair of <subject-object> Related; Step S14: pre-train a bridge encoder to perform open vocabulary trajectory classification training, including: constructing a bridge encoder based on the Transformer structure, denoted as BridgeEnc; then, obtaining the visual features of each trajectory And the category text features ; Among them, WordEmb refers to a word embedding mapping, which obtains the word embedding features of each category by inputting the category text; Then, the object trajectory classification task is used to pre-train it; , first obtain its classification probability : ； in, and is the text representation of the category, c represents the category name, The text features representing the category; Indicates the category name, The text features representing the category; represents a hyperparameter; The loss function of the final trajectory classification task is recorded as , exp represents the exponential function.

3. The open vocabulary video scene graph generation method with joint region-aware semantic alignment according to claim 1, characterized in that: The step S2 comprises: Construct a visual refiner that transfers the knowledge in the pre-trained encoder through a distillation strategy, making it pay more attention to visual-related semantic information; First, we use manual annotation to obtain a small amount of accurate object trajectory annotations and obtain their features through the ViT model , denoted as ; M represents the number of entity trajectories obtained by manual annotation, is a subscript, It is features of manually annotated trajectories; obtain a visually perceptual representation of each object trajectory using a set of learnable representations And modules based on cross-attention mechanisms, is a one-dimensional learnable representation Elements of is the number of all learnable vectors, as follows: ； in, It is obtained by the following formula: Self-attention representation of ； in, Depend on Expanded, ; Represents the attention module used and the final distillation loss for: ； in, ; in, Refers to the 2-norm operation.

4. The open vocabulary video scene graph generation method with joint region-aware semantic alignment according to claim 1, characterized in that: The step S3 comprises: First, obtain the trajectory features of visual perception and joint regional characteristics ; A semantically aware context encoder SACEncoder is used to obtain a comprehensive representation of semantic interactions, including: <subject-object> pairs, where the subject object trajectory features and object object trajectory features are and ; The subject, object, visual joint region representation, and object motion information are then connected into a sequence to simulate the structure of the relation triple; the subject, object, visual joint region representation, and object motion information are combined with a learnable representation using the context encoder SACEncoder based on the attention mechanism Interact to obtain context-aware representations: ； in, Refers to the features obtained by simulating the triple structure above, is a contextual semantically aware representation, is the object motion information obtained in step S13; ； in, is a multi-layer perceptron, Represents a normalization operation.

5. The open vocabulary video scene graph generation method with joint region-aware semantic alignment according to claim 1, characterized in that: The step S4 comprises the following steps: Step S41: After aggregating the contextual semantics, a joint region-aware relation representation is generated for the final relation classification, and a joint region relation alignment decoder URADecoder based on a multi-head attention mechanism is designed: ； in, Through learnable representation The features obtained by integrating the contextual semantics belong to the intermediate process; ； in, is a multi-head attention module. Represents a multi-layer perceptron; with the help of the decoder URADecoder, a joint region-aware relational representation is generated For subsequent relationship prediction; Step S42: Introducing a hint learning method, including: constructing hint representations for each category using learnable word vectors, and obtaining relation category representations through a pre-trained bridge encoder: ； Among them, WordEmb refers to a word embedding mapping, which obtains the word embedding features of each category by inputting the category text; Refers to the prompt representation constructed for the relation category c; represents the number of word vectors that can be learned in each prompt representation; ； in, is the learnable word vector of the setting; Is the relationship type categorical representation of; Step S43: Represent the previously acquired contextual semantic perception , Joint Region-aware Relational Representation , combined regional characteristics , and object motion information Connect and input into the relation classifier for relation prediction: ； Among them, R is the representation of the relationship between objects obtained; is from The encoded joint region representation obtained in is a multi-layer perceptron (MLP); for any <subject-object> pair , the relationship between them is represented as ; Calculate for each relationship category The predicted probability : ； in, For any <subject-object> pair , features that represent the visual relationship between them; For any relation category c, its text representation Similarly, for relationship categories Textual representation of Finally, the same classification loss function as in the trajectory classification task is used, and the loss of relation classification is recorded as .

6. The open vocabulary video scene graph generation method with joint region-aware semantic alignment according to claim 1, characterized in that: The step S5 comprises: Directly concatenate the trajectory features of the two in the <subject-object> pair as their relational features, and use the prompt learning strategy to generate relational representations for the final relational category prediction; the loss is recorded as ; Therefore, the final training loss for: ； in, is the loss function for the trajectory classification task, is the final distillation loss, The loss for relation classification.

Images