Scene graph generation method, system and equipment based on open vocabulary and medium

Abstract

The invention relates to a scene graph generation method, system and device based on an open vocabulary and a medium. The method comprises the following steps: acquiring a to-be-processed image; inputting the image into a target detection network of a scene graph generation model, identifying each entity in the image, and generating a visual embedding corresponding to each entity; inputting each entity and the corresponding visual embedding into a relation proposal network of the scene graph generation model, combining every two input entities to form a relation proposal, and generating relation embedding according to the entity corresponding to the relation proposal and the respective visual embedding; inputting the relation embedding corresponding to each relation proposal and the pre-generated text embedding into a hierarchical predictor of a scene graph generation model, and predicting the relation category of each relation proposal; and generating a scene graph of the image according to the relationship category and the entity category of the corresponding entity in the relationship proposal. According to the method, the generalization performance of the model in an open scene and the accuracy of relation prediction are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image scene understanding, and in particular to a method, system, device and medium for generating a scene graph based on an open vocabulary. Background Art

[0002] Scene graph generation is a fundamental task in computer vision, which involves building a structured representation of the scene by identifying the relationships between entities depicted in an image, and has shown significant performance improvements in various computer vision downstream tasks. Traditional scene graph generation methods usually operate in a closed vocabulary, which poses challenges in effectively modeling open relationships because the diversity of relationship concepts exceeds the existing data annotations. The diverse and complex visual relationships in real scenes can lead to inaccurate prediction results when using trained models to predict novel visual relationships. Therefore, it is necessary to provide a scene graph generation method, system, device and medium based on an open vocabulary. Summary of the invention

[0003] In view of the shortcomings of the prior art described above, an object of the present invention is to provide a method, system, device and medium for generating a scene graph based on an open vocabulary, which improves the problem that the prior art is difficult to handle unseen relationships that may appear in an open scene, resulting in low accuracy of the generated scene graph.

[0004] To achieve the above-mentioned purpose and other related purposes, the present invention provides a scene graph generation method based on an open vocabulary, comprising: obtaining an image to be processed; inputting the image into a target detection network of a scene graph generation model, identifying each entity in the image, and generating a visual embedding corresponding to each entity; inputting each entity and the corresponding visual embedding into a relationship proposal network of the scene graph generation model, combining the input entities in pairs to form relationship proposals, and generating relationship embeddings based on the entities corresponding to the relationship proposals and their respective visual embeddings; inputting the relationship embeddings corresponding to each relationship proposal and the pre-generated text embeddings into a hierarchical predictor of the scene graph generation model, predicting the relationship category of each relationship proposal; generating a scene graph of the image based on the relationship category and the entity category of the corresponding entity in the relationship proposal.

[0005] In one embodiment of the present invention, each entity and the corresponding visual embedding are input into the relationship proposal network of the scene graph generation model, the input entities are combined in pairs to form relationship proposals, and relationship embeddings are generated according to the entities corresponding to the relationship proposals and their respective visual embeddings, including: based on the spatial position and visual embedding of the entity, each entity is combined in pairs to generate a relationship proposal; wherein the two entities in the relationship proposal are arranged in a subject-object order; the two visual embeddings corresponding to the relationship proposal are embedded and fused to obtain a fused visual embedding; the two entities corresponding to the relationship proposal are respectively encoded to obtain the corresponding entity position embedding; the fused visual embedding and the corresponding entity position embedding are embedded and fused to obtain the relationship embedding corresponding to the relationship proposal.

[0006] In one embodiment of the present invention, the text embedding includes entity-aware text embedding and its corresponding region-aware text embedding, and the generation process of the entity-aware text embedding is: based on semantic similarity, multiple pre-acquired entities are clustered to obtain clustered super-class entities; the super-class entities and multiple predicates acquired in advance are arranged according to a preset entity description format, and multiple entity-aware text prompts are generated and saved; each entity-aware text prompt is input into the text encoder of the pre-trained visual language model for encoding, and the corresponding entity-aware text embedding is generated and saved.

[0007] In one embodiment of the present invention, for each entity-aware text prompt, the corresponding region-aware text embedding is generated as follows: the superclass entity corresponding to the entity-aware text prompt is input into a large language model, and each region of the input superclass entity is decomposed to obtain the key region of the superclass entity; the key region of the subject in the superclass entity and the key region of the object in the superclass entity are combined and input into the large language model again, and a plurality of region-aware text prompts are generated and saved according to a preset region description format;

[0008] Each region-aware text is promoted and input into the text encoder for encoding, and the corresponding region-aware text embedding is generated and saved.

[0009] In one embodiment of the present invention, for each relationship proposal: the relationship embedding corresponding to the relationship proposal and the pre-generated text embedding are input into the hierarchical predictor of the scene graph generation model to obtain the relationship category of the relationship proposal, including: searching for the superclass entity to which the entity in the relationship proposal belongs, determining the corresponding entity-aware text embedding and each region-aware text embedding; calculating the similarity between the relationship embedding and the corresponding entity-aware text embedding to obtain the entity-aware predicate score; calculating the similarity between the relationship embedding and each region-aware text embedding respectively to obtain the corresponding region-aware predicate score; weightedly summing the entity-aware predicate score and each region-aware predicate score respectively, and selecting the highest predicate category as the relationship category corresponding to the relationship proposal.

[0010] In one embodiment of the present invention, for each relationship proposal: the relationship embedding corresponding to the relationship proposal and the pre-generated text embedding are input into the hierarchical predictor of the scene graph generation model to obtain the relationship category of the relationship proposal, including: searching for the superclass entity to which the entity in the relationship proposal belongs, and determining the corresponding entity-aware text embedding and each region-aware text embedding; merging the bounding boxes of the two entities in the relationship proposal to form a joint region, and cropping the joint region from the image; inputting the joint region into the visual encoder of the visual language model for encoding to generate a joint embedding; filtering the region-aware text embedding according to the joint embedding, and obtaining the relationship category of the relationship proposal according to the similarity between the entity-aware text embedding, the filtered region-aware text embedding and the relationship embedding.

[0011] In one embodiment of the present invention, the method of filtering region-aware text embeddings according to joint embeddings, and obtaining a relationship category of the relationship proposal according to the similarity between entity-aware text embeddings, the filtered region-aware text embeddings and the relationship embeddings, includes: respectively calculating the similarity scores of the joint embeddings and the corresponding region-aware text embeddings, and filtering out a preset number of region-aware text embeddings with the highest similarity scores as target region-aware text embeddings; calculating the similarity between the relationship embedding and the entity-aware text embedding to obtain an entity-aware predicate score; calculating the similarity between the relationship embedding and each target region-aware text embedding, and obtaining a region-aware predicate score according to each similarity; aggregating the entity-aware predicate score and the region-aware predicate score, and selecting the category with the highest score in the aggregated score as the relationship category of the relationship proposal.

[0012] In one embodiment of the present invention, the scene graph of the image is generated based on the relationship category and the entity category of the corresponding entity in the relationship proposal, including: weighted summing the score of the relationship category of the relationship proposal and the score of the entity category to obtain the visual triplet score corresponding to the relationship proposal; arranging the various visual triplet scores in order, and screening out a preset number of visual triplet scores, and taking the corresponding relationship proposals as target relationship proposals; generating the scene graph of the image based on each target relationship proposal and its corresponding relationship category.

[0013] In one embodiment of the present invention, the scene graph generation model is obtained by training by minimizing the target loss, and the target loss includes: the mean absolute error loss and GIOU loss based on the entity's bounding box and entity position label, to obtain the entity prediction bounding box loss; the cross entropy loss based on the entity category and entity category label of the entity, to obtain the category prediction loss; the focus loss based on the relationship category and relationship category label, to obtain the relationship prediction loss; the target loss is obtained based on the entity prediction bounding box loss, the category prediction loss and the relationship prediction loss.

[0014] In one embodiment of the present invention, the target loss also includes a visual alignment loss, and the visual alignment loss is obtained by calculating the mean absolute value error of the joint embedding and the relational embedding.

[0015] In one embodiment of the present invention, a scene graph generation system based on an open vocabulary is also provided, the system comprising: an image acquisition module for acquiring an image to be processed; an entity detection module for inputting the image into a target detection network of a scene graph generation model, identifying each entity in the image, and generating a visual embedding corresponding to each entity; a proposal generation module for inputting each entity and the corresponding visual embedding into a relationship proposal network of the scene graph generation model, combining the input entities in pairs to form relationship proposals, and generating relationship embeddings based on the entities corresponding to the relationship proposals and their respective visual embeddings; a predicate prediction module for inputting the relationship embeddings corresponding to each relationship proposal and the pregenerated text embeddings into a hierarchical predictor of the scene graph generation model, predicting the relationship category of each relationship proposal; and a scene graph generation module for generating a scene graph of the image based on the relationship category and the entity category of the corresponding entity in the relationship proposal.

[0016] In one embodiment of the present invention, an electronic device is also provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the electronic device implements any of the above-mentioned scene graph generation methods based on an open vocabulary.

[0017] In one embodiment of the present invention, a computer-readable storage medium is further provided, on which a computer program is stored. When the computer program is executed by a processor of a computer, the computer executes any of the above-mentioned methods for generating a scene graph based on an open vocabulary.

[0018] As described above, the present invention provides a method, system, device and medium for generating a scene graph based on an open vocabulary, which has the following beneficial effects: identifying entities in an image and generating their visual embeddings through an object detection network, generating relationship embeddings in combination with a relationship proposal network, predicting relationship categories using text embeddings and relationship embeddings through a hierarchical predictor, and finally generating a scene graph based on entity categories and relationship categories. Since the text embeddings are pre-generated, when new vocabulary appears, the representation space of the relationship text can be expanded, providing more detailed relationship description content, greatly reducing the difficulty of the model to recognize novel relationships. In addition, the present invention utilizes a combination of text embeddings and relationship embeddings so that the model can better understand novel relationships in images. This method based on multimodal feature alignment greatly enhances the generalization ability of the scene graph generation model in open scenes and improves the accuracy of the model in generating scene graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 A schematic diagram of a flow chart of a scene graph generation method based on an open vocabulary provided in an embodiment of the present invention;

[0020] Figure 2 Shows a schematic diagram generated for the hierarchical prompt of the present invention;

[0021] Figure 3 Shown is a schematic diagram of the prediction process of the scene graph generation model of the present invention;

[0022] Figure 4 Shown is a structural block diagram of a scene graph generation system based on an open vocabulary provided by an embodiment of the present invention;

[0023] Figure 5 Shown is a structural schematic diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0025] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0026] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0027] The present invention provides a scene graph generation method based on an open vocabulary, which can help improve the performance of the model in relation prediction in unseen scenes. Before the scene graph generation model predicts the relationship, text prompts at the entity perception level and the region perception level are generated according to the vocabulary, so that the representation space of the relation text is expanded in this way, and more detailed relation description content is provided, which greatly reduces the difficulty of the model in identifying novel relations. In addition, in the process of relation prediction, in order to maintain the practicality and adaptability of the algorithm, the text prompts at the region perception level are dynamically selected using the image, allowing the model to exclude some prompts that are unlikely to appear in the image before using the text prompts at the region perception level, thereby reducing the noise interference of the text prompts at the region perception level. Finally, the relationship prediction of the open vocabulary is realized by calculating the similarity between the relation embedding and the text embedding as the predicate prediction score. In addition, during the scene graph model training process, the model also adds a regional-level visual alignment loss to align the relation embedding extracted by the model to the representation space of the visual language model, thereby enhancing the generalization ability of the model. Compared with the existing methods, the algorithm described in the present invention strengthens the text representation through hierarchical text prompts, breaks through the limitation of fixed-form text prompts to improve the relationship recognition effect. It shows significant improvements in evaluation indicators such as recall rate (Recall) and mean recall rate (Mean Recall), and has excellent performance in benchmark tests such as visual genome and open images, which further confirms the innovation and practicality of the present invention. For example, in the field of autonomous driving, the present invention can train models in a variety of traffic scenarios, so as to effectively identify and generate vehicles, pedestrians and their relationships in complex environments. This method can handle different types of objects and relationships, effectively solves the recognition limitations of traditional methods in specific categories and scenarios, and improves the adaptability of the model in diverse urban traffic environments. In addition, this method of the present invention not only enriches the text representation ability of the scene graph generation model by introducing multi-level text representation and dynamic screening mechanisms, but also significantly improves the generalization performance of the model in open scenarios and the accuracy of relationship prediction.

[0028] See also Figure 1 ,The scene graph generation method based on an open vocabulary includes the following steps:

[0029] S1. Obtain an image to be processed.

[0030] A scene graph is a graphical representation method used to describe and analyze objects and their relationships in an image. For example, if an image shows a cat lying on a sofa, its corresponding scene graph can be "Cat on the sofa". When an image needs to be analyzed and the corresponding scene graph is determined, the image can first be scaled to a preset size and the pixels can be normalized, or the image can be preprocessed such as image format conversion to improve the accuracy of subsequent recognition.

[0031] S2. Input the image into the object detection network of the scene graph generation model, identify each entity in the image, and generate a visual embedding corresponding to each entity.

[0032] The scene graph generation model includes a cascaded target detection network and a relationship proposal network, wherein the target detection network identifies each entity appearing in the image by performing target detection on the input image, and generates corresponding category labels, bounding box coordinates, and visual embeddings for each detected entity. Visual embedding refers to the representation of the entity in a high-dimensional feature space, including its visual features such as appearance, shape, texture, etc. The output of the feature extraction stage of the target detection network (such as the last layer of the feature extraction stage) can be used as the visual embedding. An entity refers to an object identified in an image, such as a cat, dog, etc. It is understandable that the target detection network can be any model that can detect objects, including but not limited to the YOLO series networks, FasterR-CNN, etc. The image passes through the target detection network and predicts N entities, and the predicted entities are recorded as N is the number of predicted entities, each v i It can be expressed as is the entity label, b i are the coordinates of the corresponding predicted bounding box in the image.

[0033] S3. Input each entity and the corresponding visual embedding into the relationship proposal network of the scene graph generation model, combine the input entities in pairs to form relationship proposals, and generate relationship embeddings based on the entities corresponding to the relationship proposals and their respective visual embeddings.

[0034] Specifically, in one embodiment of the present invention, the step of inputting each entity and the corresponding visual embedding into the relationship proposal network of the scene graph generation model, combining the input entities in pairs to form relationship proposals, and generating relationship embeddings based on the entities corresponding to the relationship proposals and their respective visual embeddings includes:

[0035] Based on the spatial position and visual embedding of the entities, the entities are combined in pairs to generate relationship proposals. The two entities in the relationship proposal are arranged in the subject-object order.

[0036] Embed and fuse the two visual embeddings corresponding to the relation proposals to obtain a fused visual embedding.

[0037] Encode the two entities corresponding to the relation proposal separately to obtain the corresponding entity position embedding;

[0038] The fused visual embedding and the corresponding entity position embedding are embedded and fused to obtain the relation embedding corresponding to the relation proposal.

[0039] The N entities predicted by the target detection network Input into the relation proposal network, predict the relation proposal and the corresponding relation embedding. Specifically, since the relation proposal network is pre-trained, it can select entities that may have semantic relationships and combine them two by two according to the visual embedding and spatial location information of the entities to generate relation proposals. i ,v j} i≠j is a relation proposal, and the corresponding relation embedding is expressed as N r is the number of predicted relation proposals, d r is the dimension of relation embedding. Among them, each relation proposal includes two entities and is arranged in the order of subject-object. For example, if the relation proposal is (person, chair), then person is the subject and chair is the object. It should be noted that the two entities in the relation proposal can be the same entity or different entities. For each relation proposal: call the multi-layer perceptron to perform feature fusion on the visual embeddings of the two entities corresponding to the proposal to generate a fused visual embedding. Through feature fusion, visual features that show a semantic relationship between the two entities can be extracted for subsequent analysis. In addition, for each relation proposal: the bounding box coordinates of the two entities corresponding to it are also encoded respectively to generate the position embedding of the corresponding entity in order to capture the interaction features between the entities. The fused visual embedding and the position embedding of the corresponding two entities are feature fused through a multi-layer perceptron to obtain the relation embedding corresponding to the relation proposal. Among them, the relation embedding can comprehensively represent the spatial information and visual information between the two entities involved in the proposal to reflect the semantic relationship between the two entities.

[0040] S4. The relation embeddings corresponding to each relation proposal and the pre-generated text embeddings are input into the hierarchical predictor of the scene graph generation model to predict the relation category of each relation proposal.

[0041] Specifically, in one embodiment of the present invention, the text embedding includes entity-aware text embedding and its corresponding region-aware text embedding, and the generation process of the entity-aware text embedding is:

[0042] Based on semantic similarity, multiple pre-acquired entities are clustered to obtain clustered super-class entities;

[0043] Arrange the superclass entity and the pre-acquired predicates according to a preset entity description format, and generate and save multiple entity-aware text prompts;

[0044] Each entity-aware text prompt is input into the text encoder of the pre-trained visual language model for encoding, and the corresponding entity-aware text embedding is generated and saved.

[0045] Considering that as the number of subjects and objects increases, the number of relationship category triples grows cubically, resulting in an increase in resource consumption and computational complexity. Therefore, it is necessary to reduce the entity categories to reduce resource consumption and computational complexity, wherein the relationship category triples refer to the three <subject, predicate, object>. Specifically, each entity in the entity vocabulary is extracted, the entity is encoded using WordNet, and by calculating the semantic similarity of each entity, a clustering algorithm (such as K-Means clustering) is used to aggregate entities with high semantic similarity into a whole, and the aggregated whole is called a superclass entity. For example, the entities are "cat", "dog", and "rabbit", and the superclass entity after clustering is "animal". In this way, the entities in the entity vocabulary can be aggregated into a preset number M of superclass entities according to semantic similarity. It should be noted that the number M of superclass entities can be adaptively set based on the actual scene graph needs, and is not limited here. In addition, the entity vocabulary can adaptively add or delete corresponding words according to actual needs, and after adding new words, the scene graph generation model can be trained again, so that the vocabulary is not limited and can be expanded indefinitely to form an open vocabulary, so that the generated scene graph is more accurate.

[0046] As an example, Figure 2 As shown in the figure, the aggregated super-class entities include people, pets, vehicles, and chairs and furniture. All predicates in the relational vocabulary are combined with these super-class entities to generate a series of entity-aware text prompts, such as <people, sitting, vehicles>, <people, sitting, chairs and furniture>, <pets, sitting, chairs and furniture>, etc. The large language model is called, and by inputting relevant instructions and examples, each entity-aware text prompt can obtain multiple region-aware text prompts, such as "body touching furniture", "hands near furniture", etc. The region-aware text prompts and entity-aware text prompts are respectively input into the text encoder of the visual language model, and the corresponding region-aware text embedding and entity-aware text embedding can be obtained through encoding.

[0047] Extract each predicate from the pre-acquired relational vocabulary, where the predicate is used to express an action relationship, such as "sitting", "standing", "holding", etc. Since the superclass entity can be used as both a subject and an object, the predicate and the superclass entity are arranged according to a preset entity description format, such as "an image describing <subject, predicate, object>", to generate an entity-aware text prompt. For example, the superclass subject is "person", the superclass object is "furniture", and the predicate is "sitting", then the corresponding entity-aware text prompt is "an image describing <person, sitting, on furniture>".

[0048] For each entity-aware text prompt: input the entity-aware text prompt into the text encoder of the pre-trained visual language model, convert the text into word units, and encode the word units of the entity-aware text prompt through the text encoder, converting them into high-dimensional vector representations, thereby capturing the semantic information therein and obtaining a text embedding vector of a preset dimension, called entity-aware text embedding. Save each generated entity-aware text embedding, and establish a mapping relationship between the entity-aware text embedding and the corresponding text prompt for subsequent predicate prediction. Among them, the encoded entity-aware text embedding can be expressed as d is the embedding dimension, and R is the real number domain. It should be noted that the visual language model includes but is not limited to CLIP, BLIP, etc., and any visual language model including a text encoder and a visual encoder is not limited here.

[0049] In one embodiment of the present invention, for each entity-aware text prompt, the corresponding region-aware text embedding is generated as follows:

[0050] Input the super-class entity corresponding to the entity-aware text prompt into the large language model, decompose each region of the input super-class entity, and obtain the key region of the super-class entity;

[0051] Combining the key regions of the subject in the superclass entity and the key regions of the object in the superclass entity and inputting them into the large language model again, generating and saving a plurality of region-aware text prompts according to a preset region description format;

[0052] Each region-aware text is promoted and input into the text encoder for encoding, and the corresponding region-aware text embedding is generated and saved.

[0053] Entity-aware text prompts are used to describe the action relationship between two entities in an integrated manner. However, in order to make the subsequent predicates more accurate, in the present invention, based on the relationship triples in the entity-aware text prompts, a large language model is used to decompose the key parts of the entity, and combined with the relationship between these parts, a region-level visual relationship prompt is naturally generated. This visual relationship prompt is a region-aware text prompt that contains visual details of a specific region. Specifically, for each entity-aware text prompt: the corresponding subject and object are respectively input into the large language model, and by inputting relevant guide words, the powerful knowledge base and language understanding ability of the large language model are used to automatically decompose the key areas or parts of the superclass entity. Exemplarily, for "an image describing <person, sitting, on a chair>", this entity-aware text prompt, for "person", can guide the large language model to decompose "person" into key parts such as buttocks, thighs, arms, and decompose "chair" into parts such as seat and backrest. By combining the performance of these parts in the relationship of "sitting", a rich region-level relationship description is generated. Through the above operations, the subject and object corresponding to each entity-aware text prompt are decomposed into several subject key areas and several object key areas, and the subject key areas and object key areas are combined, and a region-aware text prompt is generated according to a preset region description format, such as "an image described with <local detail description>", so as to enhance the richness of text representation. Exemplarily, the subject key area is the back of a person, and the object key area is the backrest of a chair, then the region-aware text prompt is: an image described with <a person's back leaning on the backrest of a chair>. Therefore, through the above method, for one entity-aware text prompt, several region-aware text prompts can be generated. Each region-aware text prompt is input into the text encoder of the aforementioned visual language model, and the text is converted into a high-dimensional vector representation to capture the semantic information therein, and a region-aware text embedding of a preset dimension is generated for subsequent predicate classification. The region-aware text embedding is saved, and a mapping relationship between it and the corresponding entity-aware text embedding is established. Among them, the encoded region-aware text embedding is recorded as In the present invention, by setting entity-aware text prompts and corresponding region-aware text prompts, hierarchical relationship prompts for relationship prediction are generated. The hierarchical relationship prompts include global entity-aware text prompts and local region-aware text prompts. Through this multi-level text prompt, the shortcomings of fixed text prompts in representation are compensated to enhance the model's understanding of visual relationships.

[0054] In one embodiment of the present invention, for each relationship proposal: the relationship embedding corresponding to the relationship proposal and the pre-generated text embedding are input into the hierarchical predictor of the scene graph generation model to obtain the relationship category of the relationship proposal, including:

[0055] Find the superclass entity to which the entity in the relation proposal belongs, and determine the corresponding entity-aware text embedding and each region-aware text embedding;

[0056] Calculate the similarity between the relation embedding and the corresponding entity-aware text embedding to obtain the entity-aware predicate score;

[0057] Calculate the similarity between the relation embedding and each region-aware text embedding respectively to obtain the corresponding region-aware predicate score;

[0058] The entity-aware predicate scores and the region-aware predicate scores are weighted and summed respectively, and the highest predicate category is selected as the relationship category corresponding to the relationship proposal.

[0059] In order to predict the relationship category (i.e., predicate) of each relationship proposal, the relationship embedding and corresponding text embedding of the relationship proposal are required. The following operations are performed for each relationship proposal: Since the relationship proposal consists of two entities, namely the subject and the object, the visual embedding corresponding to the entity is generated by the aforementioned target detection network, and the superclass entity obtained by the aforementioned clustering is used to map the two entities in the relationship proposal to the corresponding superclass entity, and the entity-aware text embedding corresponding to the superclass entity is determined, and the corresponding region-aware text embedding is determined based on the entity-aware text embedding. After determining the relationship embedding, entity-aware text embedding, and region-aware text embedding corresponding to the relationship proposal, the similarity between the relationship embedding R of the relationship proposal and the corresponding entity-aware text embedding is calculated to obtain the entity-aware predicate score corresponding to the relationship proposal, as shown in formula (1):

[0060]

[0061] in, is the similarity calculation function, is the j-th entity-aware text embedding, is the entity-aware predicate score of the j-th entity-aware text embedding and the corresponding relation proposal, C p is the preset number of predicate category spaces. It is understandable that the similarity can be cosine similarity, KL divergence, etc., which is not limited here. Preferably, in order to more accurately calculate the similarity of the two different modes of text embedding and relation embedding, cosine similarity is selected.

[0062] It should be noted that each relation proposal corresponds to a relation embedding, an entity-aware text embedding, and multiple region-aware text embeddings. Therefore, each relation proposal corresponds to an entity-aware predicate score. For each region-aware text embedding corresponding to the relation proposal: first calculate the similarity between the relation embedding R and each region-aware text embedding, then average all similarities to obtain the region-aware predicate score of the proposal, as shown in formula (2):

[0063]

[0064] The entity-aware predicate score and the region-aware predicate score are weighted and summed to obtain the final predicate score of the proposal, as shown in formula (3):

[0065]

[0066] The final predicate score corresponding to the relation proposal The predicate categories are sorted and the predicate category with the highest score is selected as the predicted relationship category of the proposal. For example, if the predicate categories and the corresponding scores are "sitting, 0.7" and "close to, 0.3", the relationship category of the proposal is "sitting".

[0067] In another embodiment of the present invention, for each relationship proposal: the relationship embedding corresponding to the relationship proposal and the pre-generated text embedding are input into the hierarchical predictor of the scene graph generation model to obtain the relationship category of the relationship proposal, including:

[0068] Find the superclass entity to which the entity in the relation proposal belongs, and determine the corresponding entity-aware text embedding and each region-aware text embedding;

[0069] Merging the bounding boxes of two entities in the relation proposal to form a joint region, and cropping the joint region from the image;

[0070] Inputting the joint region into the visual encoder of the visual language model for encoding to generate a joint embedding;

[0071] The region-aware text embedding is filtered according to the joint embedding, and the relation category of the relation proposal is obtained based on the similarity between the entity-aware text embedding, the filtered region-aware text embedding and the relation embedding.

[0072] For each relation proposal: extract the two entities involved in the proposal and the bounding box of each entity obtained through the object detection network. According to the superclass entity obtained by semantic clustering, map each entity to the corresponding superclass entity, so as to determine the entity-aware text embedding and region-aware text embedding corresponding to the proposal. Merge the bounding box of the subject and the bounding box of the object in the proposal to form a joint bounding box containing the subject and the object, where the merging of the bounding boxes refers to calculating the minimum enclosing rectangle of the two bounding boxes and taking it as the joint bounding box. The joint region containing the subject and the object is cropped from the original image through the joint bounding box. This joint region contains the two entities and their relative positions as well as background information. The joint region is input into the visual encoder of the visual language model, the information of the joint region is encoded, and the information is converted into a high-dimensional visual feature vector to obtain the joint embedding U. The joint embedding can be obtained by formula (4):

[0073] U=VisEnc(I U ) (4)

[0074] Among them, VisEnc is the visual encoder of the visual language model, I U For the joint region, The joint embedding U can reflect the visual information in the joint region. Since the visual language model has been pre-trained on a large scale and has good open scene prediction capabilities, the extracted joint embedding U also has a certain generalization ability. The present invention uses the joint embedding to screen relationship prompts.

[0075] In one embodiment of the present invention, the filtering of region-aware text embedding according to the joint embedding, and obtaining the relationship category of the relationship proposal according to the similarity between the entity-aware text embedding, the filtered region-aware text embedding and the relationship embedding, include:

[0076] Calculate the similarity scores between the joint embedding and the corresponding region-aware text embeddings respectively, and select a preset number of region-aware text embeddings with the highest similarity scores as target region-aware text embeddings;

[0077] Calculate the similarity between relation embedding and entity-aware text embedding to get entity-aware predicate score;

[0078] Calculate the similarity between the relation embedding and each target region-aware text embedding, and obtain the region-aware predicate score based on each similarity;

[0079] The entity-aware predicate score and the region-aware predicate score are aggregated, and a category with the highest score in the aggregated score is selected as the relationship category of the relationship proposal.

[0080] Predicates are predicted by calculating the similarity between the joint embedding and the multi-level text embedding. When predicting predicates, irrelevant text cues are filtered out based on the dynamic selection guided by the visual language model; hierarchical prediction aggregation is performed on the joint entity-aware text cues and region-aware text cues to ensure accurate predicate classification. Specifically, the image-text pair matching function of the visual language model is first used to filter the region-aware text cues to reduce the interference of irrelevant content. As shown in formula (5), the similarity score between the joint embedding U and each region-aware text embedding corresponding to the proposal is calculated:

[0081]

[0082] in, is the similarity calculation function, is the similarity score of the jth region-aware text embedding in the relation proposal, is the jth region-aware text embedding in the relation proposal. Arrange the calculated similarity scores in descending order, and select the top k region-aware text embeddings with the highest similarity scores as the target region-aware text embeddings for this predicate classification. In this way, text prompts are dynamically selected based on the joint embedding, and region-aware text prompts with higher probabilities are given priority for subsequent predicate prediction. This method can filter out those relation descriptions that do not appear in the image, effectively reduce noise, and improve the accuracy and speed of subsequent predicate prediction. The relation embedding R of the proposal and the entity-aware text embedding corresponding to the proposal are calculated according to the above formula (1) to obtain the entity-aware predicate score of the proposal, which encapsulates the prediction possibility between the current relation embedding and a specific entity pair. For the selected k region-aware text embeddings, the region-aware predicate score of the proposal is calculated according to formula (6):

[0083]

[0084] in, is the region-aware predicate score of the j-th relation proposal, It is the region-aware text embedding after dynamic selection. The region-aware predicate score emphasizes the region-level, fine-grained visual relationship features, and can provide rich text information to help the model understand the visual relationship. The entity-aware predicate score and the region-aware predicate score are obtained. Multiple levels of prediction scores are obtained. The present invention designs an aggregation algorithm to aggregate multiple scores to form the final predicate prediction. Specifically, the entity-aware predicate score and the region-aware predicate score are weighted and summed to obtain the final predicate score, as shown in formula (7):

[0085]

[0086] Among them, α is an adjustable weight parameter, The final predicate score for the jth relation proposal. The category with the highest score in the aggregate score will be selected as the final predicted predicate category.

[0087] S5. Generate a scene graph of the image according to the relationship category and the entity category of the corresponding entity in the relationship proposal.

[0088] Specifically, in one embodiment of the present invention, generating the scene graph of the image according to the relationship category and the entity category of the corresponding entity in the relationship proposal includes:

[0089] The score of the relation category and the score of the entity category of the relation proposal are weighted and summed to obtain the visual triplet score corresponding to the relation proposal;

[0090] Arrange the visual triplet scores in order, select a preset number of visual triplet scores, and use the corresponding relation proposals as target relation proposals;

[0091] A scene graph of the image is generated according to each target relation proposal and its corresponding relation category.

[0092] The present invention systematically eliminates invalid self-connected edges through post-processing, and excludes triples with the same subject and object entities to improve the subsequent recognition rate and accuracy. Then for each relationship proposal: the highest relationship category score of the proposal is obtained according to the aforementioned hierarchical prediction period, and the entity category scores of the subject and object in the proposal obtained by the target detection network are weighted and summed to obtain the visual triple score of the proposal. After the visual triple scores of all proposals in the image are calculated, all relationship proposals are arranged in descending order according to the visual triple scores. The higher the score, the more confident the model is in predicting the relationship proposal. From the sorted relationship proposals, the relationship proposals corresponding to the first N visual triple scores are screened out. These screened proposals are used as the target relationship proposals for the final output, providing comprehensive information on the subject entity probability, object entity probability and predicate probability. According to the screened target relationship proposals and their corresponding relationship categories, the subject and object are used as nodes, and the predicates between the two are used as edges to generate a scene graph of the input image. This method allows the present invention to derive the probability of each predicate, thereby determining the predicate category. And this multi-level prediction mechanism enhances the scene graph model's ability to predict relations while learning region-level text representations, thereby improving its open vocabulary capabilities and facilitating knowledge transfer to new relational concepts.

[0093] like Figure 3As shown, it shows the specific process of obtaining a scene graph from an image: input an image to the target detection network (not shown) of the scene graph generation model, first detect the entities that may exist in it, determine the entity category and the corresponding bounding box. The output of the target detection network is input to the scene graph generation model relationship proposal network to obtain a relationship proposal (such as {person, bicycle}) and a corresponding relationship embedding R. A pair of entities in the relationship proposal can merge the corresponding bounding boxes to form a joint region, and the joint region is cropped from the image. By calling the image encoder of the visual language model, the joint region is encoded to obtain a joint embedding U. The similarity between the joint embedding U and the region-aware text embedding corresponding to the proposal is calculated, so as to dynamically select several region-aware text embeddings. The similarity between the relationship embedding R and the corresponding entity-aware text embedding and the selected several region-aware text embeddings is calculated respectively, and the two similarities are aggregated, and the highest predicate category after aggregation is selected as the predicate of the proposal (such as "riding"). Therefore, the relationship triplet obtained in the graph is <person, riding, bicycle>, and the corresponding scene graph can be obtained.

[0094] In one embodiment of the present invention, the scene graph generation model is obtained by training by minimizing a target loss, and the target loss includes:

[0095] Based on the mean absolute error loss and GIOU loss of the entity’s bounding box and entity location label, the entity prediction bounding box loss is obtained;

[0096] Based on the cross entropy loss of the entity category and entity category label of the entity, the category prediction loss is obtained;

[0097] Based on the focal loss of relation category and relation category label, the relation prediction loss is obtained;

[0098] The target loss is obtained based on the entity prediction bounding box loss, the category prediction loss and the relationship prediction loss.

[0099] When training the scene graph generation model, the present invention uses multi-task loss to update the model parameters at each iteration. Specifically, the mean absolute error loss and GIOU loss are used for entity bounding box regression to reduce the gap between the predicted bounding box and the true label. The entity prediction bounding box loss is shown in Figure (8):

[0100] L bbox =‖bb gt ‖1+GIOU(bb gt ) (8)

[0101] Among them, L bbox Predict bounding box losses for entities, b and b gtare the predicted bounding box and the entity location label (i.e., the real bounding box), respectively. In addition, the present invention also uses the cross entropy loss L ent To ensure the accuracy of entity prediction, as shown in formula (9):

[0102] L ent =CE(c,c gt ) (9)

[0103] Among them, c gt is the true entity category (i.e., entity category label), c is the predicted entity category, and L ent is the category prediction loss, CE is the cross entropy loss function. For predicate prediction, the present invention uses focal loss to calculate the relationship prediction loss L pre , as shown in formula (10):

[0104] L pre =FL(r,r gt ) (10)

[0105] Among them, L pre is the relationship prediction loss, r gt is the true basic predicate category (i.e., the relationship category label), r is the predicted predicate category (i.e., the relationship category), and FL is the focal loss function. The target loss function is obtained by weighted summing the above three losses.

[0106] Furthermore, in another embodiment of the present invention, the objective loss function also includes a visual alignment loss, and the visual alignment loss is obtained by calculating the mean absolute error of the joint embedding and the relational embedding. In order to ensure that the relational embedding extracted by the scene graph generation model has generalization ability, the present invention is designed to align the relational embedding with the visual embedding generated by the visual language model. The visual language model is pre-trained on a large scale on an image-text pair dataset, and can well align the visual embedding with the text embedding into a unified space. By designing a visual alignment loss, it is intended to pull the distance between the relational embedding and the visual embedding of the visual language model, where the visual language model plays the role of a teacher and the scene graph generation model plays the role of a student. Since the visual language model is built on the overall image, the scene graph generation model requires regional level understanding. Therefore, the present invention adopts the aforementioned joint embedding U as teacher supervision and the relational embedding R as student embedding, and calculates the visual alignment loss by the mean absolute error loss, as shown in formula (11):

[0107] L d = ‖RU‖1 (11)

[0108] Among them, L dis the visual alignment loss, U is the joint embedding, and R is the relational embedding. The target loss is obtained by weighted summing the above four losses, as shown in formula (12):

[0109] L total =L bbox +λ1L ent +λ2L pre +λ3L d (12)

[0110] Among them, L total is the target loss, and λ1, λ2, and λ3 are the weights of the weighted sum. For the nth iteration: back propagate and update the parameters in the nth scene graph generation model through the gradient descent method. Therefore, in the training process of the scene graph generation model, the present invention calculates the training loss of triples through object detection and predicate prediction results. At the same time, the visual alignment loss is calculated based on the relational embedding extracted by the model and the joint embedding extracted by the visual language model to ensure that the relational embedding has good generalization performance for open scenes. By combining multiple loss functions, the model parameters are updated in a gradient descent manner.

[0111] In summary, for each training image, the present invention first uses the network to predict the bounding box and entity category, and then outputs the relationship proposal and the relationship embedding corresponding to the proposal through the relationship proposal network. In the predicate prediction stage, for the pre-set relationship vocabulary, the entities are first summarized into super-class entity categories through semantic clustering, the category space of the relationship triples is reduced, and the text prompts at the entity perception level are generated. Then, a large language model is used to generate a rich regional level relationship description for each visual triple to form a regional perception level text prompt. By encoding the above text prompts, the text embedding required for predicate classification is formed. When the predicate prediction is formally performed, the relationship embedding extracted by the network will calculate the similarity score with the text embedding at the entity perception level and the regional perception level respectively. After the calculation is completed, the final predicted relationship category confidence is obtained by weighted combination of scores to determine the predicted relationship category. Combined with multi-level text prompts, the text representation during relationship classification is significantly enhanced, better corresponding to the context in visual information, and the generalization ability of the scene graph model in open scenes is enhanced.

[0112] See also Figure 4, the scene graph generation system 100 based on the open vocabulary includes: an image acquisition module 110, an entity detection module 120, a proposal generation module 130, a predicate prediction module 140 and a scene graph generation module 150. The above-mentioned image acquisition module 110 is used to acquire the image to be processed. The entity detection module 120 is used to input the image into the object detection network of the scene graph generation model, identify each entity in the image, and generate the visual embedding corresponding to each entity. The proposal generation module 130 is used to input each entity and the corresponding visual embedding into the relationship proposal network of the scene graph generation model, combine the input entities in pairs to form a relationship proposal, and generate a relationship embedding according to the entities corresponding to the relationship proposal and their respective visual embeddings. The predicate prediction module 140 is used to input the relationship embedding corresponding to each relationship proposal and the pre-generated text embedding into the hierarchical predictor of the scene graph generation model to predict the relationship category of each relationship proposal. The scene graph generation module 150 is used to generate the scene graph of the image according to the relationship category and the entity category of the corresponding entity in the relationship proposal.

[0113] For the specific limitations of the scene graph generation system based on an open vocabulary, please refer to the limitations of the scene graph generation method based on an open vocabulary above, which will not be repeated here. Each module in the above-mentioned scene graph generation system based on an open vocabulary can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in a hardware format, or can be stored in a memory in the computer device in a software format, so that the processor can call the operations corresponding to the above modules.

[0114] It should be noted that, in order to highlight the innovative part of the present invention, the present embodiment does not introduce modules that are not closely related to solving the technical problem proposed by the present invention, but this does not mean that there are no other modules in the present embodiment.

[0115] See also Figure 5 The electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a scene graph generation program based on an open vocabulary.

[0116] Among them, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 12 can be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 can also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 12 can also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as code generated by a scene graph based on an open vocabulary, etc., but can also be used to temporarily store data that has been output or is to be output.

[0117] In some embodiments, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (Control Unit) of the electronic device 1, and uses various interfaces and lines to connect various components of the entire electronic device 1, and executes or executes programs or modules stored in the memory 12 (such as a scene graph generation program based on an open vocabulary, etc.), and calls data stored in the memory 12 to execute various functions of the electronic device 1 and process data.

[0118] The processor 13 executes the operating system and various installed applications of the electronic device 1. The processor 13 executes the applications to implement the steps in the above-mentioned scene graph generation method based on an open vocabulary.

[0119] Exemplarily, the computer program may be divided into one or more modules, which are stored in the memory 12 and executed by the processor 13 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program in the electronic device 1. For example, the computer program may be divided into an image acquisition module 110, an entity detection module 120, a proposal generation module 130, a predicate prediction module 140, and a scene graph generation module 150.

[0120] The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium, and the computer-readable storage medium can be non-volatile or volatile. The above-mentioned software function module is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a computer device, or a network device, etc.) or a processor to perform part of the functions of the scene graph generation method based on an open vocabulary described in various embodiments of the present application.

[0121] In summary, the present invention discloses a method, system, device and medium for generating a scene graph based on an open vocabulary. For a new scene vocabulary, the method of the present invention can be first used to generate entity-aware text prompts and region-aware text prompts. The text prompts are encoded using the text encoder of the visual language model to generate text embeddings. The image is input into the object detection network of the scene graph generation model to perform entity detection and predict each entity box and its corresponding visual embedding. Then, the entity box and the visual embedding are input into the relationship proposal network of the scene graph generation model to generate a relationship embedding. The process includes generating a relationship proposal and generating relationship features by combining the entity box and the visual embedding involved in the relationship proposal. According to the generated relationship embedding and text embedding, a hierarchical predictor is used to predict the relationship category score of each relationship proposal, and finally the relationship prediction result is output based on the relationship category prediction score and the entity category prediction score. In the present invention, relationship prompts at different levels combine subject-object information with regional relationship information, thereby providing a richer text representation, so that it can better match the visual representation in the open vocabulary task. The present invention also provides a dynamic hierarchical prompt selection mechanism, which includes merging the bounding boxes of the two entities in the relationship proposal, cropping the image, and inputting the cropped image into the visual encoder of the visual language model to obtain a joint visual embedding, and finally calculating the matching score based on the joint visual embedding and the regional hierarchical text embedding, and selecting the text embedding with the highest score to participate in the relationship category prediction. The present invention aims to overcome the limitations of fixed text prompts and improve the classification performance of the model through more detailed text representation, while removing unnecessary noise through dynamic screening. Therefore, the present invention effectively overcomes the various shortcomings in the prior art and has a high industrial utilization value.

[0122] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A scene graph generation method based on an open vocabulary, characterized in that: The generation method comprises: Obtain the image to be processed; Inputting the image into an object detection network of a scene graph generation model, identifying entities in the image, and generating visual embeddings corresponding to the entities; Input each entity and the corresponding visual embedding into the relation proposal network of the scene graph generation model, combine the input entities into pairs to form relation proposals, and generate relation embeddings based on the entities corresponding to the relation proposals and their respective visual embeddings; The relation embeddings and pre-generated text embeddings corresponding to each relation proposal are input into the hierarchical predictor of the scene graph generation model to predict the relation category of each relation proposal; A scene graph of the image is generated according to the relationship categories and the entity categories of the corresponding entities in the relationship proposals.

2. The method for generating a scene graph based on an open vocabulary according to claim 1, characterized in that: The step of inputting each entity and the corresponding visual embedding into the relationship proposal network of the scene graph generation model, combining the input entities in pairs to form relationship proposals, and generating relationship embeddings based on the entities corresponding to the relationship proposals and their respective visual embeddings, includes: Based on the spatial position and visual embedding of the entities, the entities are combined in pairs to generate relationship proposals. The two entities in the relationship proposal are arranged in the subject-object order. Embed and fuse the two visual embeddings corresponding to the relation proposals to obtain a fused visual embedding. Encode the two entities corresponding to the relation proposal separately to obtain the corresponding entity position embedding; The fused visual embedding and the corresponding entity position embedding are embedded and fused to obtain the relation embedding corresponding to the relation proposal.

3. The method for generating a scene graph based on an open vocabulary according to claim 1, characterized in that: The text embedding includes entity-aware text embedding and its corresponding region-aware text embedding, and the generation process of the entity-aware text embedding is: Based on semantic similarity, multiple pre-acquired entities are clustered to obtain clustered super-class entities; Arrange the superclass entity and the pre-acquired predicates according to a preset entity description format, and generate and save multiple entity-aware text prompts; Each entity-aware text prompt is input into the text encoder of the pre-trained visual language model for encoding, and the corresponding entity-aware text embedding is generated and saved.

4. The method for generating a scene graph based on an open vocabulary according to claim 3, characterized in that: For each entity-aware text hint, the corresponding region-aware text embedding is generated as follows: Input the super-class entity corresponding to the entity-aware text prompt into the large language model, decompose each region of the input super-class entity, and obtain the key region of the super-class entity; Combining the key regions of the subject in the superclass entity and the key regions of the object in the superclass entity and inputting them into the large language model again, generating and saving a plurality of region-aware text prompts according to a preset region description format; Each region-aware text is promoted and input into the text encoder for encoding, and the corresponding region-aware text embedding is generated and saved.

5. The method for generating a scene graph based on an open vocabulary according to claim 4, characterized in that: For each relation proposal: the relation embedding corresponding to the relation proposal and the pre-generated text embedding are input into the hierarchical predictor of the scene graph generation model to obtain the relation category of the relation proposal, including: Find the superclass entity to which the entity in the relation proposal belongs, and determine the corresponding entity-aware text embedding and each region-aware text embedding; Calculate the similarity between the relation embedding and the corresponding entity-aware text embedding to obtain the entity-aware predicate score; Calculate the similarity between the relation embedding and each region-aware text embedding respectively to obtain the corresponding region-aware predicate score; The entity-aware predicate scores and the region-aware predicate scores are weighted and summed respectively, and the highest predicate category is selected as the relationship category corresponding to the relationship proposal.

6. The method for generating a scene graph based on an open vocabulary according to claim 4, characterized in that: For each relation proposal: the relation embedding corresponding to the relation proposal and the pre-generated text embedding are input into the hierarchical predictor of the scene graph generation model to obtain the relation category of the relation proposal, including: Find the superclass entity to which the entity in the relation proposal belongs, and determine the corresponding entity-aware text embedding and each region-aware text embedding; Merging the bounding boxes of two entities in the relation proposal to form a joint region, and cropping the joint region from the image; Inputting the joint region into the visual encoder of the visual language model for encoding to generate a joint embedding; The region-aware text embedding is filtered according to the joint embedding, and the relation category of the relation proposal is obtained based on the similarity between the entity-aware text embedding, the filtered region-aware text embedding and the relation embedding.

7. The method for generating a scene graph based on an open vocabulary according to claim 6, characterized in that: The method of filtering the region-aware text embedding according to the joint embedding, and obtaining the relationship category of the relationship proposal according to the similarity between the entity-aware text embedding, the filtered region-aware text embedding and the relationship embedding, includes: Calculate the similarity scores between the joint embedding and the corresponding region-aware text embeddings respectively, and select a preset number of region-aware text embeddings with the highest similarity scores as target region-aware text embeddings; Calculate the similarity between relation embedding and entity-aware text embedding to get entity-aware predicate score; Calculate the similarity between the relation embedding and each target region-aware text embedding, and obtain the region-aware predicate score based on each similarity; The entity-aware predicate score and the region-aware predicate score are aggregated, and a category with the highest score in the aggregated score is selected as the relationship category of the relationship proposal.

8. The method for generating a scene graph based on an open vocabulary according to claim 1, characterized in that: The step of generating a scene graph of the image according to the relationship category and the entity category of the corresponding entity in the relationship proposal includes: The score of the relation category and the score of the entity category of the relation proposal are weighted and summed to obtain the visual triplet score corresponding to the relation proposal; Arrange the visual triplet scores in order, select a preset number of visual triplet scores, and use the corresponding relation proposals as target relation proposals; A scene graph of the image is generated according to each target relationship proposal and its corresponding relationship category.

9. The method for generating a scene graph based on an open vocabulary according to claim 1, characterized in that: The scene graph generation model is trained by minimizing the target loss, and the target loss includes: Based on the mean absolute error loss and GIOU loss of the entity’s bounding box and entity location label, the entity prediction bounding box loss is obtained; Based on the cross entropy loss of the entity category and entity category label of the entity, the category prediction loss is obtained; Based on the focal loss of relation category and relation category label, the relation prediction loss is obtained; The target loss is obtained based on the entity prediction bounding box loss, the category prediction loss and the relationship prediction loss.

10. The method for generating a scene graph based on an open vocabulary according to claim 9, characterized in that: The target loss also includes a visual alignment loss, which is obtained by calculating the mean absolute error of the joint embedding and the relation embedding.

11. A scene graph generation system based on an open vocabulary, characterized in that: The system comprises: An image acquisition module, used for acquiring an image to be processed; An entity detection module, configured to input the image into an object detection network of a scene graph generation model, identify each entity in the image, and generate a visual embedding corresponding to each entity; The proposal generation module is used to input each entity and the corresponding visual embedding into the relationship proposal network of the scene graph generation model, combine the input entities into pairs to form relationship proposals, and generate relationship embeddings based on the entities corresponding to the relationship proposals and their respective visual embeddings; The predicate prediction module is used to input the relation embeddings and pre-generated text embeddings corresponding to each relation proposal into the hierarchical predictor of the scene graph generation model to predict the relation category of each relation proposal; A scene graph generation module is used to generate a scene graph of the image according to the relationship category and the entity category of the corresponding entity in the relationship proposal.

12. An electronic device, characterized in that: The electronic device comprises: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the electronic device to implement the scene graph generation method based on an open vocabulary as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the scene graph generation method based on an open vocabulary as claimed in any one of claims 1 to 10.

Images