Text knowledge enhanced scene graph generation method

Abstract

The invention relates to a text knowledge enhanced scene graph generation method, and belongs to the field of computational vision. The method comprises the following steps: extracting text features of text information; extracting visual features and entity labels of the scene image, and combining the extracted visual features into paired entity pair features; introducing entity pair feature guidance to generate visual guidance text feature representation, so that the extracted text features are associated with visual entities; obtaining refined entity enhancement features and entity-to-visual predicate features based on the visual features and the entity tags; performing cross-modal alignment on the visual guidance text features and the entity pair visual predicate features to obtain graphical multi-modal feature representation; and inputting the multi-modal feature representation and the entity enhancement feature into a multi-layer perceptron for prediction to obtain prediction output of the entity pair and the entity pair relationship. According to the invention, the accuracy of entity object prediction and entity relationship prediction in the scene graph generation process can be improved.

Description

Technical Field

[0001] The invention belongs to the field of computer vision and relates to a scene graph generation method enhanced with text knowledge. Background Art

[0002] At present, computer technology, communication technology and network technology are changing with each passing day, and human society is gradually transitioning from the personal computer era to the ubiquitous computing era. Therefore, computer reasoning about entity relationships in scenes has received more and more attention and research. Among them, scene graphs have been a hot research topic in this field in recent years. The task of scene graph generation is to generate a comprehensive graph structure in the image, which can show the entities in the scene and the relationship between entity pairs. The graph consists of triples in the form of <subject, predicate, object>, where the subject and object are classified into predefined entity classes, and the predicate is the visual relationship between the subject and the object. With the continuous development of computer vision technology, people are no longer satisfied with simply detecting and identifying objects in the scene for scene understanding, but accurately obtaining the entity relationships in the scene.

[0003] The scene graph generation methods that use single visual information have limitations, because using only information from a single visual modality will result in incomplete information, making it difficult to capture cross-modal contextual information, resulting in the lack of entity relationship information, which in turn limits the comprehensive understanding of complex scenes. In addition, in the scene graph generation task, even if the model can successfully obtain entity relationship information in the scene image and perform recognition, the lack of rich contextual information in the single visual information will lead to ambiguity and uncertainty in the recognition results. For example, in a two-dimensional image, interactive information such as "holding" and "raising" can only be displayed as other spatial position information such as "in front" or "behind". This is mainly due to the inherent constraints of the single-view image itself, which cannot provide complete three-dimensional structural information of the entities in the scene. It is difficult to know the depth relationship and spatial order information between different entities by relying solely on image information for entity relationship recognition. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a scene graph generation method enhanced with text knowledge, which combines additional scene text information with scene visual information to generate entity interaction relationship prediction results that are more in line with actual scenes to enhance the scene graph generation effect.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for generating a scene graph enhanced with text knowledge, the method generates a scene graph based on a given scene image and text information corresponding to the scene image, the scene graph includes a visual relationship triple, i.e. <subject, predicate, object>, and the key lies in the prediction of the relationship (i.e. predicate) of each entity pair (i.e. <subject, object>). Specifically, the method includes:

[0007] S1, extracting text features of the text information; extracting visual features and entity labels of the scene image, and combining the extracted visual features into paired entity pair features;

[0008] S2, introduce entity pair feature guidance to generate visually guided text feature representation, so that the extracted text features are associated with visual entities;

[0009] S3, obtaining refined entity enhancement features and entity pair visual predicate features based on the visual features and entity labels;

[0010] S4. Cross-modally align visually guided text features with entity pair visual predicate features to obtain a fused feature representation, i.e., a graphical multimodal feature representation, which is used to generate a scene graph containing rich contextual information, enabling the model to effectively capture and process the complex interactions between visual and textual data.

[0011] S5. Input the multimodal feature representation and entity enhancement features into a multi-layer perceptron for prediction, and obtain the prediction output of entity pairs and entity pair relationships.

[0012] Furthermore, in step S1, a text representation with relational triple semantics may be extracted through an information extraction model, and visual features of an input image may be extracted through a visual entity detector.

[0013] Furthermore, step S2 is specifically as follows: using the attention mechanism to extract the visual feature r v and text feature r xp Map to the same dimension, calculate the similarity between the text feature and the visual feature to generate an attention weight map, wherein entities with high relevance to each entity pair are assigned higher weights, and entities with low relevance to each entity pair are assigned lower weights; the attention weight map is combined with the text feature r xp Multiply to get the visual entity feature r′ v , and r′ is converted into v The information is then integrated with the text feature r xp Perform linear fusion and splicing to obtain the visually guided text feature r′ xp .

[0014] Furthermore, step S3 is specifically as follows: transforming the visual feature r v , the entity label embedding C and the bounding box embedding B are concatenated to form an overall feature vector, and the initial entity feature is obtained through linear transformation Initial Solid Feature Perform multi-head self-attention operations to enhance entity features and obtain entity enhanced features γ;

[0015] The visual feature r v , the entity enhancement feature γ and the entity label embedding C are concatenated and the initial feature r for predicate prediction is obtained through linear transformation p , for r p Perform multi-head self-attention operations to obtain enhanced predicate features; divide the enhanced predicate features into entity pairs to construct entity pair visual predicate features r′ with rich contextual information o .

[0016] Furthermore, step S4 is specifically as follows: firstly, the same-modal interaction is performed through paired self-attention modules, and the entity pair visual predicate features and visually guided text features are respectively input into the self-attention modules to obtain the processed visual features and text features in, represents the output of the N-layer self-attention module, Represents the output of the n-th layer self-attention module;

[0017] Then, through paired guided attention modules, cross-modal interaction is performed. and Input into the guided attention module and obtain the text feature output f of the N-layer guided attention module t (N) ;

[0018] use and f t (N) Calculate the attention weights to get the aggregated text features Then and The concatenation is performed to obtain a graphical multimodal feature representation.

[0019] Furthermore, step S5 is specifically as follows: inputting the multimodal feature representation and the entity enhancement feature into a multilayer perceptron for prediction, and optimizing the parameters by minimizing the cross entropy loss, so as to obtain accurate entity pair and entity pair relationship prediction outputs.

[0020] The beneficial effects of the present invention are as follows: the present invention utilizes text knowledge representation and text context representation to supplement the missing visual context in the scene, effectively aligns visual information and text information through cross-modal information alignment, integrates information from different modalities, improves the accuracy of entity object prediction and entity relationship prediction in the scene graph generation process, reduces the lack of entity relationship information due to a single perceptual modality information and the ambiguity and uncertainty in the relationship recognition results, and enables the machine to have a deeper and richer understanding of complex environments and subtle interactions.

[0021] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0023] Figure 1 It is a schematic diagram of the process of the method of the present invention;

[0024] Figure 2 The structure framework diagram of the method described in the present invention. DETAILED DESCRIPTION

[0025] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0026] In order to solve the problem that the entity relationship information in the scene is lost and the relationship prediction is ambiguous due to the limitations of single visual information, the present invention proposes a scene graph generation method enhanced by text knowledge, which utilizes the additional text information in the scene to supplement the missing visual context with text knowledge representation and text context representation, effectively aligns the visual information and text information through cross-modal information alignment, integrates information from different modalities, better captures the details and complex information of the scene, generates more accurate entity relationship predictions, and enhances the computer's situational awareness capability.

[0027] See also Figure 1 and Figure 2 , which is a text knowledge enhanced scene graph generation method proposed in one embodiment of the present invention, which includes visual feature extraction, text feature extraction, entity relationship enhancement and cross-modal relationship representation alignment.

[0028] The specific contents of this embodiment are as follows:

[0029] Given an input scene image I and the corresponding text T, the scene graph is finally generated It contains the visual relationship triplet, namely g i =(subject i ,prediction i ,object i ), the key is that each entity pair (subject i ,object i ) i prediction. N represents the number of generated triplets.

[0030] 1. For the input scene image I, the visual features of the input image are extracted through the visual feature extraction module.

[0031] This example uses a pre-trained visual entity detector F v To extract the visual features of the input image, the Faster R-CNN model is used, and the ResNeXt-101-FPN network is used to extract the visual features of the scene image I. This network combines the characteristics of the ResNeXt network and the FPN network. When processing complex visual tasks that require fine features and multi-scale recognition, it can take into account both efficient computing performance and powerful feature learning capabilities. First, the image is converted into a set of candidate regions C = {c i |i=1,2,...,N v},N v Represents the number of extracted visual candidate regions. Then object-level visual features are extracted from the candidate regions. and the corresponding labels

[0032]

[0033] Combine the extracted visual entity features into pairs of features r y , which is used to subsequently generate the relationship between visual entities:

[0034]

[0035] 2. For the input text T, extract text features through the text feature extraction module.

[0036] The additional text information in the scene can supplement the context information of the visual scene graph. This embodiment adopts a BERT-based information extraction model to obtain text representation with relational triple semantics. The information extraction model can be expressed as F t , which accepts the tag sequence and the segmentation sequence as input and outputs text features. Text Information Extraction Model F tThe text sequence T consisting of pre-extracted named entity pairs is accepted as input. In order to adapt to the BERT encoding requirements, a [CLS] tag is added at the beginning of the sequence and a [SEP] tag is added at the end. These tags help the model identify the beginning and end of the sequence. At the same time, additional tags [E Start ] and [E End ], these markers are used to highlight the start and end positions of the entities, which helps the model to more accurately extract and associate information between entities. For each pair of entities, the sequence T′ after adding the markers is expressed as:

[0037]

[0038] Where m s and m o Respectively represent the starting position of the subject entity and the object entity; n s and n o Respectively represent the length of the subject entity and the object entity; t0 and The two tags are the [CLS] tag and the [SEP] tag.

[0039] The sequence is then normalized to meet the BERT encoding processing standards and pruned to a maximum length of M. t , and use [PAD] to fill the insufficient length. Construct the segmentation sequence T s = (1,1,...,1,0,...,0) separates valid markers from fill markers, where 1 represents a valid marker and 0 represents a fill marker. Finally, the model F t Convert text to a matrix Among them, M t is the sequence length, d x is the dimension of the text representation.

[0040] The above method improves the model's recognition and extraction of entity information in the text in a structured way, thereby improving the effect of the text information extraction task. The overall encoding process can be formulated as:

[0041]

[0042] In order to generate text features that can be used later, the encoding representations of all possible entity pairs are combined, using N e Represents the number of possible entity pairs in the text sequence T:

[0043]

[0044] Since the text knowledge extracted from the text sequence T is not necessarily closely related to the visual entity, the visual entity pair feature r extracted by the visual feature extraction module is introduced.y Guide the module to generate visually guided textual knowledge. Specifically, the attention mechanism is used to map the extracted visual and textual features to the same dimension, and then the similarity between the textual features and the visual entity pair features is calculated to generate an attention weight map. Entities with high relevance to each entity pair are assigned higher weights, and entities with low relevance are assigned lower weights. Multiply the weight map with the textual features to obtain the visual entity features. d h Represents a representation that maps visual features and text features to the same dimension, and r′ is converted by summing v and integrate it with the text entity pair feature r xp Perform linear fusion splicing to obtain the text knowledge representation r′ guided by vision xp .

[0045] 3. Generate more accurate and detailed entity relationship predictions through the entity relationship enhancement module.

[0046] This module extracts visual features and entity tags As input, we finally get the enhanced entity feature γ and the entity pair visual predicate feature r′ o , to generate more accurate and detailed entity predictions, while being able to leverage both visual features and entity context information for predicate prediction.

[0047] Specifically, the visual feature r v , the entity label embedding C and the bounding box embedding B are concatenated to form an overall feature vector, and the initial entity feature is obtained through linear transformation Initial Solid Feature Perform multi-head self-attention operations to enhance entity features and obtain the final entity enhanced feature γ:

[0048]

[0049] Among them, W o , b o represents learnable parameters.

[0050] The enhanced entity features γ will be used in the subsequent decoding process to generate more accurate and detailed entity predictions. This process ensures that the model can better understand and represent entities and their relationships in the image.

[0051] Similarly, the visual feature r v , the entity enhancement feature γ and the entity label embedding C are concatenated and the initial feature r for predicate prediction is obtained through linear transformation p , for the initial feature r pPerform multi-head self-attention operations to obtain enhanced predicate features. Divide the enhanced predicate features into entity pairs to construct entity pair visual predicate features r′ with rich contextual information o , used to capture multimodal interaction information in the subsequent decoder.

[0052] 4. Alignment of cross-modal relationship representations

[0053] For the alignment of textual features with visual features, we model intra-modal and inter-modal interactions through stacked attention-based modules. o and the visually guided text feature representation r′ xp As input, a multi-head attention module consisting of a pair of self-attention modules (SAM) and a pair of guided attention modules (GAM) captures the dense interaction of the input features, where both SAM and GAM have N layers. The SAM module focuses on the same-modal interaction (i.e., the relationship within the visual information or within the text information), and the GAM module handles the cross-modal interaction between visual features and text features (i.e., the relationship between visual and text information). The model finally outputs a fused feature representation, which is used to generate a scene graph, that is, a graphical multimodal representation that contains rich contextual information, enabling the model to effectively capture and process the complex interactions between visual and text features. The calculation form is as follows:

[0054]

[0055] The SAM module is used to process the same-modal interaction between visual features and text features. represents the visual feature input of the n-1th layer, f t (n-1) Represents the text feature input of the n-1th layer. SAM (n) take over and f t (n-1) As input, and output and Represent the visual feature output and text feature output of the nth layer respectively.

[0056] The GAM module is used to handle cross-modal interactions between visual features and text features. (n) Received by SAM (n) Processed text features And the final visual feature output ( is the output of the Nth layer SAM module) as input and outputs f t (n) , represents the text feature output of the nth layer.

[0057] In order to generate a comprehensive multimodal representation, the final output obtained by the module is and f t (N) Calculate the attention weights to aggregate the final text representation:

[0058]

[0059] Among them, W v , W t represents the parameters in training and learning, f t (N) Represents the text feature output of the Nth layer after being processed by the self-attention module (SAM) and the guided attention module (GAM) in the N-layer multi-head attention module.

[0060] Concatenating each visual feature output with the aggregated text feature is a multimodal feature representation:

[0061]

[0062] The generated multimodal feature representation θ and the enhanced entity feature γ are fed into the multi-layer perceptron for prediction. The model optimizes the parameters by minimizing the cross entropy loss, thereby improving the prediction accuracy of entity labels and predicate categories:

[0063]

[0064] Loss = CE(η o ,R o )+CE(η p ,R p ) (16)

[0065] Among them, W p and b p Represents the predicted output η of the calculation model for the entity pair relationship p Parameters used when p is the weight parameter, b p is the bias term; φ o represents the set of object labels, φ p represents the set of predicate categories, CE(·) represents the cross entropy loss, η o Represents the model’s predicted output for the entity pair, R o represents the true label of the entity pair, η p Represents the model’s prediction output for entity pair relationships, R p Represents the true label of the entity pair relationship.

[0066] Based on the method described in the above embodiment, the method can be applied to the robot's visual understanding, which is critical for deepening the robot's visual understanding in the process of human-computer interaction. Specifically, the method can allow the robot to carefully capture and analyze the complex relationships between people and people, and people and objects in the interactive scene through real-time image and voice input. Through this high-level understanding, the robot can accurately identify and interpret the scene structure and dynamic interactions between characters revealed by its visual input, thereby significantly improving the naturalness and immersion of the interaction. It can not only enhance the operational accuracy of the robot, but also optimize the user experience, enabling the robot to perform tasks and respond to user needs more effectively in a complex human-computer interaction environment.

[0067] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A text knowledge enhanced scene graph generation method, the method generates a scene graph based on a given scene image and text information corresponding to the scene image, characterized in that: The process of generating a scene graph includes: Extracting text features of the text information; extracting visual features and entity labels of the scene image, and combining the extracted visual features into paired entity pair features; Introducing entity pair feature guidance to generate visually guided text feature representation, so that the extracted text features are associated with visual entities; Obtaining refined entity enhancement features and entity pair visual predicate features based on the visual features and entity labels; Cross-modally align visually guided text features with entity pair visual predicate features to obtain a graphical multimodal feature representation; The multimodal feature representation and entity enhancement features are input into a multi-layer perceptron for prediction, and the prediction output of entity pairs and entity pair relationships is obtained.

2. The scene graph generation method according to claim 1, characterized in that: The method of introducing visual pair feature guidance to generate visual guided text feature representation includes: using an attention mechanism to extract the visual feature r v and text feature r xp Map to the same dimension, calculate the similarity between the text feature and the visual feature to generate an attention weight map, wherein entities with high relevance to each entity pair are assigned higher weights, and entities with low relevance to each entity pair are assigned lower weights; the attention weight map is combined with the text feature r xp Multiply to get the visual entity feature r v ′, and by summing r v ′ is integrated with the text feature r xp Perform linear fusion and splicing to obtain the visually guided text feature r x ' p .

3. The scene graph generation method according to claim 1, characterized in that: The refined entity enhancement features and entity pair visual predicate features obtained based on the visual features and entity labels include: v , the entity label embedding C and the bounding box embedding B are concatenated to form an overall feature vector, and the initial entity feature is obtained through linear transformation Initial Solid Feature Perform multi-head self-attention operations to enhance entity features and obtain entity enhanced features γ; The visual feature r v , the entity enhancement feature γ and the entity label embedding C are concatenated and the initial feature r for predicate prediction is obtained through linear transformation p , for r p Perform multi-head self-attention operations to obtain enhanced predicate features; divide the enhanced predicate features into entity pairs to construct entity pair visual predicate features r with rich contextual information o ′.

4. The scene graph generation method according to claim 1, characterized in that: The cross-modal alignment of the visually guided text features with the entity pair visual predicate features comprises: firstly, performing same-modal interaction through paired self-attention modules, respectively inputting the entity pair visual predicate features and the visually guided text features into the self-attention modules, and obtaining processed visual features. and text features in, represents the output of the Nth layer of self-attention module, N is the total number of layers of self-attention module, Represents the output of the n-th layer self-attention module; Then, through paired guided attention modules, cross-modal interaction is performed. and Input into the guided attention module and obtain the text feature output f of the Nth layer guided attention module t (N) ; use and f t (N) Calculate the attention weights to get the aggregated text features Then and The concatenation is performed to obtain a graphical multimodal feature representation.

5. The scene graph generation method according to claim 1, characterized in that: The multimodal feature representation and entity enhanced features are input into a multilayer perceptron for prediction, and the parameters are optimized by minimizing the cross entropy loss to obtain accurate entity pair and entity pair relationship prediction outputs.

Images