A method for generating open vocabulary scene graphs based on fine-grained relations

Abstract

The present invention relates to the field of artificial intelligence technology and provides a method for generating an open vocabulary scene graph based on fine-grained relationships. The method comprises the following steps: performing image annotation and visual feature extraction on a processed image to obtain scene graph annotation information and visual features, respectively; decomposing prompt text to obtain subword units; performing visual feature mapping to obtain mapping features, inputting the mapping features and subword units into a large language model to obtain a structured sequence, and obtaining normalized bounding box coordinates based on a hidden layer position state; obtaining a hidden layer state, determining a target category, and obtaining a vocabulary category score based on the hidden layer state and the target category, and obtaining a subword category score based on the vocabulary category score, performing vocabulary replacement on the structured sequence to obtain a target structured sequence; performing special vocabulary segmentation and vocabulary extraction to obtain a ternary relationship group, thereby obtaining a target scene graph. The present invention can efficiently annotate the processed image.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for generating an open vocabulary scene graph based on fine-grained relationships. Background Art

[0002] With the development of technology, people's lives and production are increasingly connected to the internet. Images and text are the two primary means of information acquisition. Bridging the semantic gap between image and text modalities has become a hot topic in recent years. Open-vocabulary scene graph generation is an emerging research direction at the intersection of computer vision and natural language processing. Its goal is to detect and localize entities in scene graphs from images and construct semantic relationships between pairs of entities. It also supports the recognition and reasoning of open-vocabulary categories—entity or relationship categories not seen during training. Compared to traditional scene graph generation tasks, the core challenge of open-vocabulary scene graph generation lies in breaking through the limitations of predefined categories, enabling the model to generalize to arbitrary entity and relationship categories, thereby better adapting to complex and changing real-world scenarios. A scene graph is a graphical representation of image content, where nodes represent entities and edges represent relationships between entities. For example, in an image, "person-rider-bicycle" forms a complete triplet, clearly conveying the semantic information in the image. Open-vocabulary nature is a key characteristic of open-vocabulary scene graph generation. The model needs to understand and generate categories that are not explicitly labeled in the training data through semantic alignment or knowledge transfer. This means that the model must not only be able to identify known categories but also be able to handle unknown categories, greatly expanding its application scenarios and generalization capabilities.

[0003] Existing open vocabulary scene graph generation tasks mainly focus on generating scene graphs with open predicates, and the goal of generation is to generate scene graphs that contain new entity and relationship categories that have not been seen during the training phase. By leveraging pre-trained multimodal models, it can predict categories beyond the vocabulary and understand images more comprehensively. Although this is an important advancement, it still cannot fully adapt to the generation of open vocabulary scene graphs. Previous studies have addressed the challenges of open vocabulary separately. Some work extends entity detection to new categories, but cannot effectively solve the long tail problem of predicates; some work uses visual-language models to generalize predicates, but it requires the assumption that entities are fixed. Some work uses visual cues for relationship discovery, but has difficulties in combinatorial reasoning. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides an open vocabulary scene graph generation method based on fine-grained relationships to achieve the generation of an open vocabulary scene graph.

[0005] The present invention provides a method for generating an open vocabulary scene graph based on fine-grained relationships, comprising:

[0006] S1: Acquire an image to be processed, perform image annotation and visual feature extraction on the image to be processed, and obtain scene image annotation information and visual features respectively;

[0007] S2: Obtain prompt text, and decompose the prompt text to obtain subword units;

[0008] S3: performing visual feature mapping on the visual features to obtain mapping features, inputting the mapping features and the sub-word units into a large language model to obtain a structured sequence, supervising the structured sequence through the scene image annotation information, and obtaining normalized bounding box coordinates based on a hidden layer position state of the structured sequence;

[0009] S4: Obtaining a hidden layer state of the structured sequence, determining a target category and obtaining a vocabulary category score based on the hidden layer state and the target category, determining a subword length and obtaining a subword category score based on the vocabulary category score and the subword length, performing vocabulary replacement on the structured sequence based on the subword category score to obtain a target structured sequence;

[0010] S5: performing special vocabulary segmentation and vocabulary extraction on the target structured sequence to obtain a ternary relationship group, and obtaining a target scene graph according to the ternary relationship group and the normalized bounding box coordinates.

[0011] According to a method for generating an open vocabulary scene graph based on fine-grained relationships provided by the present invention, in step S1, the entities to be labeled in the image to be processed are determined and the relationship category information between the entities to be labeled is determined, and the scene graph annotation information is obtained based on the entities to be labeled and the relationship category information.

[0012] According to a method for generating an open vocabulary scene graph based on fine-grained relationships provided by the present invention, in step S1, the image to be processed is divided into blocks, and the divided image to be processed is input into a visual encoder to obtain the visual features.

[0013] According to the method for generating an open vocabulary scene graph based on fine-grained relationships provided by the present invention, in step S2, the prompt text is decomposed into sub-word units by a tokenization algorithm.

[0014] According to the method for generating an open vocabulary scene graph based on fine-grained relationships provided by the present invention, in step S3, visual feature mapping is performed on the visual features by a multi-layer perceptron to obtain mapping features.

[0015] According to a method for generating an open vocabulary scene graph based on fine-grained relationships provided by the present invention, in step S3, the hidden layer position state of the subwords with entity tags in the structured sequence is extracted, and the hidden layer position state is mapped by a multi-layer perceptron to obtain the normalized bounding box coordinates.

[0016] According to a method for generating an open vocabulary scene graph based on fine-grained relationships provided by the present invention, in step S4, the word class hidden layer state of each word class in the structured sequence is obtained, and the word class hidden layer states are spliced ​​to obtain the hidden layer state. After determining the target category, the decoder classification head is obtained and a label vocabulary mask is selected according to the target category. The hidden layer state is input into the decoder classification head, and the vocabulary category score is obtained through the decoder classification head and the label vocabulary mask.

[0017] According to a method for generating an open vocabulary scene graph based on fine-grained relationships provided by the present invention, in step S4, the subword length is determined. When the subword length is 1, a candidate category is determined, a candidate word list is determined based on the candidate category, a vocabulary category score is obtained by using the candidate word list and the vocabulary category score, and the vocabulary category score is used as the subword category score;

[0018] When the subword length is greater than 1, the vocabulary classification score of each word in the subword is calculated, and the vocabulary classification scores are averaged according to the subword length to obtain the subword category score.

[0019] According to a method for generating an open vocabulary scene graph based on fine-grained relationships provided by the present invention, in step S4, multiple candidate categories are determined, the subword category scores of the subwords in the structured sequence under all candidate categories are calculated, and the candidate category with the highest subword category score is used to replace the original subwords in the structured sequence to obtain the target structured sequence.

[0020] According to a method for generating an open vocabulary scene graph based on fine-grained relationships provided by the present invention, in step S5, the normalized bounding box coordinates of the subwords with entity tags in each triple relationship group before vocabulary replacement are determined, and the image to be processed is relabeled according to the normalized bounding box coordinates before vocabulary replacement and the triple relationship group to obtain the target scene graph.

[0021] The above one or more technical solutions in the embodiments of the present invention have at least one of the following technical effects:

[0022] The present invention provides a method for generating an open vocabulary scene graph based on fine-grained relationships. It guides a large language model to understand the image to be processed through visual features and prompt text, and generates an open structured sequence with vocabulary tags described in natural language. Subsequently, it performs vocabulary replacement through subword category scores to converge the overly open structured sequence, thereby obtaining a more standardized target structured sequence. The target structured sequence is used to obtain a target scene graph that is highly open and relatively standardized, thereby providing relatively rich and standardized scene graph materials for model training, machine learning and other processes.

[0023] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 This is a flow chart of a method for generating an open vocabulary scene graph based on fine-grained relationships provided by the present invention. DETAILED DESCRIPTION

[0026] To make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below. Obviously, the embodiments described are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0027] In the description of the embodiments of the present invention, it should be noted that the terms “first”, “second” and “third” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0028] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0029] The following combination Figure 1 Describe the specific embodiment of the present invention:

[0030] Figure 1 This is a flow chart of a method for generating an open vocabulary scene graph based on fine-grained relationships provided by the present invention, comprising the steps of first obtaining an image to be processed, and performing image annotation and visual feature extraction on the image to be processed to obtain scene graph annotation information and visual features; then obtaining a prompt text, and decomposing the prompt text to obtain subword units; then performing visual feature mapping on the visual features to obtain mapping features, and inputting the subword units and mapping features into a large language model to obtain a structured sequence, and obtaining normalized bounding box coordinates through the hidden layer position state of the structured sequence; then determining the target category and calculating the vocabulary category score, obtaining the subword category score through the vocabulary category score and the subword length, and performing vocabulary replacement to obtain a target structured sequence; finally performing special vocabulary segmentation and vocabulary extraction on the target structured sequence to obtain a ternary relationship group, thereby performing annotation to obtain a target scene graph.

[0031] The present invention provides a method for generating an open vocabulary scene graph based on fine-grained relationships, comprising:

[0032] S1: Acquire an image to be processed, perform image annotation and visual feature extraction on the image to be processed, and obtain scene image annotation information and visual features respectively;

[0033] Furthermore, the purpose of this stage is to perform image annotation and visual feature extraction on the image to be processed, so as to obtain scene diagram annotation information and visual features; specifically, in step S1, the entities to be annotated in the image to be processed are determined and the relationship category information between the entities to be annotated is determined, and the scene diagram annotation information is obtained based on the entities to be annotated and the relationship category information.

[0034] In step S1, the image to be processed is divided into blocks, and the divided image to be processed is input into a visual encoder to obtain the visual features.

[0035] Regarding the above steps, the specific implementation methods in this embodiment are as follows:

[0036] First, obtain the image to be annotated. Here, the image to be annotated is a real scene photo. Then, perform image annotation on the image to be annotated. In the process of image annotation, in addition to taking the objects in the image as entities to be annotated and annotating their position information and type information, it is also necessary to determine the relationship category information between the entities to be annotated. The relationship category information includes the relationship between the entities to be annotated. For example, in the annotation of the sentence "a person is driving a car" on the image to be annotated, the person and the car are both type information of the entities to be annotated, and driving is the relationship category information between the entities to be annotated. The relationship category information is reflected as a predicate in the annotation. The position information of the entity to be annotated is first annotated with a annotation box to the entity to be annotated. Then, in this embodiment, the position of the annotation box in the image to be annotated is determined according to the coordinate values ​​of the points in the upper left corner and lower right corner of the annotation box in the image to be annotated, and the position of the annotation box is used as the position information of the entity to be annotated. After image annotation is performed on the image to be processed, the scene map annotation information A can be obtained:

[0037]

[0038] in, is the location information of the i-th entity to be labeled, is the type information of the i-th entity to be labeled, is the location information of the jth entity to be labeled, is the type information of the jth entity to be labeled, n is the number of entities to be labeled, is the relationship category information between the i-th entity to be labeled and the j-th entity to be labeled.

[0039] In addition, visual features of the image to be processed need to be extracted, that is, the image to be processed is first divided into blocks, and then the divided image to be processed is input into the visual encoder. The visual encoder will obtain the visual features of each image block after division, and then reorganize the visual features of the image blocks according to the positional relationship between the image blocks to obtain the visual features of the image to be processed.

[0040] S2: Obtain prompt text, and decompose the prompt text to obtain subword units;

[0041] Furthermore, the purpose of this stage is to decompose the prompt text into sub-word units. Specifically, in step S2, the prompt text is decomposed into sub-word units by a tokenization algorithm.

[0042] Regarding the above steps, the specific implementation methods in this embodiment are as follows:

[0043] First, obtain the prompt text. In this embodiment, the prompt text is in the form of:

[0044] <sys>Generate a Entity-Relation Triplet for the image.< / sys>

[0045] Use [ENT] for entities and [REL] for relation

[0046]

[0047] in, <sys>< / sys> The specific content of the prompt text is between. It is used to tell the large language model input is the mapped visual feature. The prompt text is then broken down into subword units using the single-word tokenization algorithm in the SentencePiece library. Subword units are the smallest language units with semantic meaning.

[0048] S3: performing visual feature mapping on the visual features to obtain mapping features, inputting the mapping features and the sub-word units into a large language model to obtain a structured sequence, supervising the structured sequence through the scene image annotation information, and obtaining normalized bounding box coordinates based on a hidden layer position state of the structured sequence;

[0049] Furthermore, the purpose of this stage is to generate a structured sequence in the trained large language model by mapping features and subword units, and to obtain normalized bounding box coordinates based on the hidden layer position state of the structured sequence. Specifically, in step S3, the visual features are mapped using a multi-layer perceptron to obtain mapped features.

[0050] In step S3, the hidden layer position states of the subwords with entity tags in the structured sequence are extracted, and the hidden layer position states are mapped by a multi-layer perceptron to obtain the normalized bounding box coordinates.

[0051] Regarding the above steps, the specific implementation methods in this embodiment are as follows:

[0052] First, because images cannot be directly recognized by the large language model, a multi-layer perceptron is used to map visual features to text space to obtain mapped features. The Mish activation function is then used to enhance their nonlinear expressiveness. The subword units and mapped features obtained by decomposing the prompt text are then input into the trained large language model to obtain a structured sequence. Specifically, the large language model uses entity relationship triplets as a structured sequence to describe the content of the processed image.

[0053] Here, the entity relationship triple is a language structure that describes the relationship between two entities through the subject-verb-object structure. For example, "a person driving a car" is an entity relationship triple, which expresses the relationship between the two entities, a person and a car. In addition, the structured sequence needs to be supervised by the scene diagram annotation information, that is, the part of the structured sequence that is obviously inconsistent with the scene diagram annotation information is removed to avoid the hallucination phenomenon of the large language model and the misunderstanding of the mapping features, which leads to incorrect matching of the relationship between entities. In addition, according to the requirements of the sub-word units obtained by disassembling the prompt text, the entity tag [ENT] will be added to the front of the part representing the entity in the structured sequence, and the relationship tag [REL] will be added to the predicate component to distinguish them, and the relationship tag and the entity tag will be used as vocabulary tags. For example, the structured sequence describing "a person riding a bike" in the image to be processed is [ENT]person[REL]riding[ENT]bicycle。 The structured sequence obtained here is a fine-grained relationship.

[0054] In addition, it is also necessary to determine the position information of the entities to be labeled in the image to be processed. Here, the structured sequence needs to be decomposed into the smallest language units with semantic meaning, that is, subwords. For example, working space is a subword. At this time, it has the semantic meaning of a workbench. After it is decomposed into two independent words, it loses its semantic meaning. Extract the hidden layer position state of the subwords with entity tags in the structured sequence. Since the mapping features input into the large language model are the mapping of the visual features of the image to be processed, the position information of the entities to be labeled in the image to be processed is retained. This will remain in the hidden layer of the large language model and be used as the hidden layer position state of the i-th entity to be labeled. Extract it and map it using a two-layer multi-layer perceptron to obtain the normalized bounding box coordinates of the i-th entity to be marked :

[0055]

[0056] in, is the parameter of the first fully connected layer of the first multilayer perceptron, is the parameter of the second fully connected layer of the first multi-layer perceptron, ReLU() is to activate the contents in the brackets using the ReLU activation function, is the first fully connected layer parameter of the second multilayer perceptron, is the parameter of the second fully connected layer of the second multilayer perceptron, The brackets () are normalized to the range [0, 1] using the sigmoid function. Similarly, the scene annotation information can be used for supervision. If the normalized bounding box coordinates differ significantly from the location information in the scene annotation information, the location information in the scene annotation information can be used to correct them.

[0057] S4: Obtaining a hidden layer state of the structured sequence, determining a target category and obtaining a vocabulary category score based on the hidden layer state and the target category, determining a subword length and obtaining a subword category score based on the vocabulary category score and the subword length, performing vocabulary replacement on the structured sequence based on the subword category score to obtain a target structured sequence;

[0058] Furthermore, the purpose of this stage is to obtain the hidden layer state, thereby obtaining a sequence prediction score, calculating the subword category score, and performing vocabulary replacement on the structured sequence based on the subword category score to obtain the target structured sequence. Specifically, in step S4, the word class hidden layer state of each word class in the structured sequence is obtained, and the word class hidden layer states are spliced ​​to obtain the hidden layer state. After determining the target category, the decoder classification head is obtained and a label vocabulary mask is selected based on the target category. The hidden layer state is input into the decoder classification head, and the vocabulary category score is obtained through the decoder classification head and the label vocabulary mask.

[0059] In step S4, the subword length is determined. When the subword length is 1, a candidate category is determined. A candidate word list is determined based on the candidate category. A vocabulary category score is obtained by combining the candidate word list and the vocabulary category score. The vocabulary category score is used as the subword category score.

[0060] When the subword length is greater than 1, the vocabulary classification score of each word in the subword is calculated, and the vocabulary classification scores are averaged according to the subword length to obtain the subword category score.

[0061] In step S4, multiple candidate categories are determined, and subword category scores of the subwords in the structured sequence under all candidate categories are calculated. The candidate category with the highest subword category score is used to perform vocabulary replacement on the original subwords in the structured sequence to obtain a target structured sequence.

[0062] Regarding the above steps, the specific implementation methods in this embodiment are as follows:

[0063] First, the three parts of speech of the structured sequence, namely the hidden state of the subject, predicate and object, are obtained from the large language model. , the hidden state of the predicate and the hidden state of the object As the word class hidden layer state, concatenating them can obtain the hidden layer state H of the structured sequence:

[0064]

[0065] Among them, concat() means concatenating the content within the parentheses. Subsequently, the target category is selected. Since the structured sequences generated by the large language model may have various descriptions for objects of the same class and may also have expressions outside the predetermined classes, this is very不利 for generating target scene graphs that need to be standardized for uses such as machine learning and model training. This requires classifying the subwords in the structured sequence to a certain extent. For example, subwords such as "sedan", "truck", and "jeep" are uniformly classified as automobiles, and subwords such as "slightly in front" and "in front" are uniformly classified as the front. In order to屏蔽 the expressions outside the predetermined classes in the structured sequence, a reasonable range of classes needs to be selected as the target category according to the requirements, so as to determine the label vocabulary mask for屏蔽 the subwords unrelated to the target category according to the target category , select the decoder classification head of the text decoder and input the hidden layer state into the decoder classification head to obtain the vocabulary category score :

[0066]

[0067] Among them, softmax() means performing a softmax transformation on the content within the parentheses represents the transpose of the decoder classification head parameters represents the decoder classification head model parameters. The specific values of the decoder classification head parameters and the decoder classification head model parameters are determined according to the selected decoder classification head represents matrix dot multiplication. The vocabulary category score here includes the probabilities that each word in the structured sequence of the large language model may be expressed as each word. For example, if a word in the structured sequence is "man", the vocabulary category score shows that the probability of it being expressed as "man" is 80%, the probability of it being expressed as "old man" is 3%, etc. Since the probability of being expressed as "man" is the highest, the word expressed in the output structured sequence is "man".

[0068] Subsequently, determine the subword length, that is, how many words are included in the subwords other than the relationship markers and entity markers. The subword length can be judged according to the vocabulary markers, punctuation, etc. of the standardized sequence. When the subword length is 1, obtain the probabilities that the word corresponding to the subword is expressed as each word from the vocabulary category score; select a class from the target category, that is, the k-th class as the candidate classification, determine the words included in the candidate classification and use them as the candidate word list, extract the probabilities that the word corresponding to the subword is expressed as all the words in the candidate word list from the probabilities that the word corresponding to the subword is expressed as each word, and calculate their average value as the vocabulary classification score for the word to be classified as the k-th class :

[0069] It should be noted that the Chinese term "屏蔽" in the original text seems to be an incorrect or incomplete expression. It might be better to use a more appropriate and accurate word in the translation for a more meaningful and understandable English text. Also, the overall translation is based on the literal meaning of the original text to maintain the consistency of the translation task requirements.

[0070] in, is the number of words included in the candidate word list, The probability that the word corresponding to the subword is expressed as the u-th word in the candidate word list. For example, if the candidate category is "cat", and the candidate word list includes the three words "cat", "kitty", and "pussy", then Taking 3, we calculate the average of the sum of the probabilities of the word corresponding to the subword being expressed as "cat," "kitty," and "pussy," and we get the lexical classification score for the cat class. Since the word is now a subword, we can use the lexical classification score as the subword category score.

[0071] When the subword length When it is greater than 1, after determining the candidate classification, there is a corresponding candidate word list for each word in the subword. The above method can be used to obtain the vocabulary classification score of the vth word in the subword in the kth category. , averaging them to get the subword category score of the subword being classified as the kth category :

[0072]

[0073] According to the target category, multiple candidate categories are determined, and the subword category scores of all subwords in the structured sequence except for the relationship markers and entity markers are calculated under all candidate categories. Since the expression of the candidate category is also a subword, for each subword, the original subword in the structured sequence is replaced with the candidate category with the highest subword category score, and the target structured sequence can be obtained. The candidate categories are thus used to standardize the original structured sequence. Since the candidate categories and candidate word lists used by the subject and object are quite different from those used by the predicate, the candidate categories and candidate word lists can be determined for the subject and object and used for the subject and object, and another candidate category and candidate word list can be determined for the predicate and used for the predicate.

[0074] S5: performing special vocabulary segmentation and vocabulary extraction on the target structured sequence to obtain a ternary relationship group, and obtaining a target scene graph according to the ternary relationship group and the normalized bounding box coordinates.

[0075] Furthermore, the purpose of this stage is to segment and extract the target structured sequence to obtain ternary relationship groups, thereby labeling the image to be processed and obtaining the target scene graph. Specifically, in step S5, the normalized bounding box coordinates of the subwords with entity tags in each ternary relationship group before vocabulary replacement are determined, and the image to be processed is relabeled based on the normalized bounding box coordinates before vocabulary replacement and the ternary relationship group to obtain the target scene graph.

[0076] Regarding the above steps, the specific implementation methods in this embodiment are as follows:

[0077] After obtaining the target structured sequence, perform special vocabulary segmentation on the target structured sequence T according to the relationship tag and entity tag to obtain the subword set S:

[0078]

[0079] Here, split() indicates a special lexical segmentation of the content within the brackets, namely, splitting the target structured sequence at the relation and entity tags, and breaking it down into a subword set consisting of subject and object subwords, as well as predicate subwords, based on the relation and entity tags. Lexical extraction is then performed. For each subword in the subword set, if it has an entity tag, it is extracted for future use. If it has a relation tag, it is extracted as subword Y with the relation tag, and the preceding subword X with the entity tag and the following subword Z with the entity tag are extracted. These three are then connected in the order they appear in the subword set to obtain a triple relationship group XYZ with fine-grained relationships.

[0080] After obtaining all the triple relationship groups according to the target structured sequence, the normalized bounding box coordinates of the subwords with entity tags in each triple relationship group before vocabulary replacement can be determined, so that the position of the subwords with entity tags in the image to be processed can be determined according to the normalized bounding box coordinates. Combined with the triple relationship groups, the image to be processed can be re-labeled to obtain an open vocabulary scene graph with relatively standard vocabulary, which is the target scene graph.

[0081] The present invention also verifies the effectiveness of an open vocabulary scene graph generation method based on fine-grained relationships. The images to be processed are from the widely used public scene graph generation datasets Visual Genome and OpenImage V6. The tasks performed in the experiment include: a predicate classification subtask, that is, given a pair of entity types and position boxes, the relationship between two entities is calibrated; a scene graph classification subtask, that is, given the location of an entity, the entity type and the relationship between the entities are calibrated; a scene graph generation subtask, that is, the image to be processed is labeled. In order to illustrate the effectiveness of the present invention for open vocabulary scenes, 70% of the types of relational predicates involved in the experiment have been used for training and are called basic classes, and another 30% have never been used for training and are called new classes. Under the condition that new classes of relational predicates exist, the experimental results of the method provided by the present invention and other methods in various tasks are compared as shown in Table 1:

[0082] Table 1 Comparison of experimental results of the method provided by the present invention and other methods in various tasks under the condition that new classes of relational predicates exist

[0083]

[0084] Among them, VG stands for Visual Genome, OIV6 stands for OpenImage V6, in the method column, OpenSGen is the method provided by the present invention, and others such as CaCao, MOTIFS, OvSGTR, etc. are all existing methods, mR50 / 100 represents the average recall rate of the top 50 / top 100 triple relationship groups, R50 / 100 represents the recall rate of the top 50 / top 100 triple relationship groups, bold indicates that the data is the best data in the same column of data for the same task, new class + base class indicates that the relationship predicates included in the image to be processed include both new classes and base classes, and new class indicates that the relationship predicates included in the image to be processed only include new classes.

[0085] Then, the experimental conditions were changed. 70% of the entity types involved in the experiment were used for training and were used as basic classes, and the other 30% were never used for training and were used as new classes. The comparison of the experimental results of the method provided by the present invention and other methods in various tasks under the condition of the existence of new classes of entities is shown in Table 2:

[0086] Table 2 Comparison of experimental results of the method provided by the present invention and other methods in various tasks under the condition that new classes of entities exist

[0087]

[0088] Among them, R50 represents the recall rate of the first 50 triple relationship groups, and R100 represents the recall rate of the first 100 triple relationship groups. It can be seen from the data in Table 1 and Table 2 that the present invention can annotate the processed images with high accuracy in scenarios where there are new types of open vocabularies for both relation predicates and entities.

[0089] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for generating open vocabulary scene graphs based on fine-grained relationships, characterized in that: include: S1: Acquire an image to be processed, perform image annotation and visual feature extraction on the image to be processed, and obtain scene image annotation information and visual features respectively; S2: Obtain prompt text, and decompose the prompt text to obtain subword units; S3: performing visual feature mapping on the visual features to obtain mapping features, inputting the mapping features and the sub-word units into a large language model to obtain a structured sequence, supervising the structured sequence through the scene image annotation information, and obtaining normalized bounding box coordinates based on a hidden layer position state of the structured sequence; S4: Obtaining a hidden layer state of the structured sequence, determining a target category and obtaining a vocabulary category score based on the hidden layer state and the target category, determining a subword length and obtaining a subword category score based on the vocabulary category score and the subword length, performing vocabulary replacement on the structured sequence based on the subword category score to obtain a target structured sequence; In step S4, the subword length is determined. When the subword length is 1, a candidate category is determined. A candidate word list is determined based on the candidate category. A vocabulary category score is obtained by combining the candidate word list and the vocabulary category score. The vocabulary category score is used as the subword category score. When the subword length is greater than 1, the lexical classification score of each word in the subword is calculated, and the lexical classification scores are averaged according to the subword length to obtain the subword category score; Determine multiple candidate categories, calculate subword category scores of the subwords in the structured sequence under all candidate categories, and use the candidate category with the highest subword category score to perform lexical replacement on the original subwords in the structured sequence to obtain a target structured sequence; S5: performing special vocabulary segmentation and vocabulary extraction on the target structured sequence to obtain a ternary relationship group, and obtaining a target scene graph according to the ternary relationship group and the normalized bounding box coordinates.

2. The method for generating an open vocabulary scene graph based on fine-grained relationships according to claim 1, characterized in that: In step S1, entities to be labeled in the image to be processed are determined, and relationship category information between the entities to be labeled is determined, and the scene image annotation information is obtained according to the entities to be labeled and the relationship category information.

3. The method for generating an open vocabulary scene graph based on fine-grained relationships according to claim 1, characterized in that: In step S1, the image to be processed is divided into blocks, and the divided image to be processed is input into a visual encoder to obtain the visual features.

4. The method for generating an open vocabulary scene graph based on fine-grained relationships according to claim 1, wherein: In step S2, the prompt text is broken down into sub-word units by a tokenization algorithm.

5. The method for generating an open vocabulary scene graph based on fine-grained relationships according to claim 1, wherein: In step S3, visual feature mapping is performed on the visual features by a multi-layer perceptron to obtain mapping features.

6. The method for generating an open vocabulary scene graph based on fine-grained relationships according to claim 1, wherein: In step S3, the hidden layer position states of the subwords with entity tags in the structured sequence are extracted, and the hidden layer position states are mapped by a multi-layer perceptron to obtain the normalized bounding box coordinates.

7. The method for generating an open vocabulary scene graph based on fine-grained relationships according to claim 1, wherein: In step S4, the word class hidden layer state of each word class in the structured sequence is obtained, and the word class hidden layer states are spliced ​​to obtain the hidden layer state. After determining the target category, the decoder classification head is obtained and the label vocabulary mask is selected according to the target category. The hidden layer state is input into the decoder classification head, and the vocabulary category score is obtained through the decoder classification head and the label vocabulary mask.

8. The method for generating an open vocabulary scene graph based on fine-grained relationships according to claim 1, wherein: In step S5, the normalized bounding box coordinates of the subwords with entity tags in each triple relationship group before vocabulary replacement are determined, and the image to be processed is relabeled according to the normalized bounding box coordinates before vocabulary replacement and the triple relationship group to obtain the target scene graph.

Images