Multi-modal image retrieval method and device based on scene graph

Abstract

The embodiment of the invention provides a multi-modal image retrieval method and device based on a scene graph, and the method comprises the steps: extracting image features through a built-in visual encoder after receiving a preprocessed image through the powerful understanding capability of a multi-modal large language model, and directly outputting a structured scene graph description through the generation capability of the language model. In this way, in the retrieval process, the model can achieve fine-grained semantic matching through a progressive query retrieval mechanism constructed by query decomposition and subgraphs, and finally high-precision text-image retrieval is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a multi-modal image retrieval method and device based on a scene graph. BACKGROUND

[0002] In a digital asset management system, users often need to accurately retrieve related images through text descriptions. Existing multi-modal retrieval technology mainly relies on pre-training models such as CLIP (Contrastive Language-Image Pre-training) to map text and images into the same latent space for similarity calculation. However, the existing technology has the following technical defects:

[0003] 1. Limitations of scene graph construction methods: Traditional methods rely on basic object detection models for image analysis, and then construct relationships between objects through regularization methods. This approach has limited object detection accuracy and low relationship recognition accuracy.

[0004] 2. Single training strategy problem: Existing methods usually use an end-to-end single-stage training method, which cannot fully utilize the characteristics of different types of data, resulting in poor model performance in complex query scenarios. Traditional training methods have a significant decrease in accuracy when dealing with fine-grained queries.

[0005] 3. Coarse-grained retrieval mechanism problem: Existing technology mainly performs similarity matching at the whole image level, lacks deep understanding and decomposition ability for user queries, and cannot handle complex queries containing multiple object relationships.

[0006] To solve the above problems, a method is needed that can provide standardized image content descriptions to improve the accuracy of text-image retrieval. SUMMARY

[0007] To solve the problems in the prior art, the present application provides a multi-modal image retrieval method and device based on a scene graph to improve the accuracy of text-image retrieval.

[0008] To solve at least one of the above problems, the present application provides the following technical solutions:

[0009] In a first aspect, the present application provides a multi-modal image retrieval method based on a scene graph, applied to a multi-modal large language model, the method comprising:

[0010] Obtaining an image and pre-processing the image, extracting features from the pre-processed image to obtain an image feature vector; matching the image feature vector with a pre-trained language knowledge base to generate a full image description of the image; converting the full image description into a structured scene graph and writing it into a scene graph database, the scene graph being represented in the form of a triple; the triple including a subject object, a relationship type and an object object;

[0011] Receiving a query command of a user and decomposing the query command into a plurality of sub-queries; selecting triples related to the sub-queries from the scene graph of each image and splicing them into candidate sub-graphs; splicing the sub-queries into a query sub-graph, calculating the similarity of each candidate sub-graph with the query sub-graph, and arranging the images in descending order of the similarity scores of the candidate sub-graphs and feeding them back to the user.

[0012] Further, the scene graph-based multi-modal image retrieval method further includes joint adjustment of the text encoder and the image encoder, and the steps of the joint adjustment include:

[0013] Constructing a text corpus containing multi-granularity descriptions at the scene level, object level and attribute level, and a relationship description corpus obtained by templating and extended writing of a large language model from the triples; Figure Three

[0014] In the training batch, positive samples are extracted according to a sampling ratio of 4:1 of full image description to relationship description, and negative samples are also collected;

[0015] Inputting the positive samples and negative samples into the text encoder to obtain text embedding vectors, inputting the image into the image encoder to obtain image embedding vectors, and aligning the text embedding vectors and the image embedding vectors in the same semantic space;

[0016] The text embedding vectors and the image embedding vectors jointly participate in the calculation of the contrast learning loss to maximize the similarity of the positive samples and minimize the similarity of the negative samples;

[0017] Iteratively updating the text encoder parameters until a set text-image retrieval accuracy threshold is reached.

[0018] Further, the scene graph-based multi-modal image retrieval method further includes training of the graph retrieval capability, and the training of the graph retrieval capability includes:

[0019] Constructing query-full image embedding pairs and sub-query-sub-graph embedding pairs; the query-full image embedding pair takes the full image natural language description or the real user query as the text, and the corresponding full image embedding as the positive example; the sub-query-sub-graph embedding pair is to decompose the query into a plurality of sub-queries, and embed the connected sub-graphs corresponding to the semantic of each sub-query as the positive example;

[0020] ​The full graph matching and subgraph matching are jointly optimized by using a dual-tower or multi-tower contrastive learning loss function, and a difficult example mining mechanism is introduced to dynamically update negative samples.

[0021] After the training is completed, an approximate nearest neighbor index for full graph embedding and subgraph embedding is constructed offline.

[0022] Further, the encoding strategy of the triple includes:

[0023] The complexity of the triple is analyzed, and the triple is divided into simple relations and complex relations according to the analysis result of the complexity;

[0024] For simple relations, the triple is encoded using an overall encoding method; for complex relations, the triple is encoded using a separate encoding method;

[0025] The overall encoding method is to connect the subject object, the relation type and the object object using a special separator, and then perform overall encoding through a pre-trained text encoder;

[0026] The separate encoding method is to independently encode the subject object, the relation type and the object object, and then combine them through a feature fusion strategy.

[0027] Further, the scene graph database is a multi-layer graph neural network structure, including:

[0028] The scene graph is processed using a multi-layer graph neural network structure, which includes a graph neural network layer focusing on local feature learning of nodes, a graph neural network layer learning direct relationships between nodes, and a graph neural network layer capturing global graph structure information.

[0029] Further, the scene graph-based multi-modal image retrieval method further includes primary retrieval, and the primary retrieval includes:

[0030] The candidate images are obtained by querying the scene graph database with a query command;

[0031] After the query command is decomposed into multiple sub-queries, the triples related to the sub-queries are selected from the scene graph of each candidate image and spliced into candidate subgraphs.

[0032] Further, the step of decomposing the query command into multiple sub-queries includes:

[0033] The query command is segmented, part-of-speech tagged, dependency syntax analyzed and named entity recognized by using dependency syntax and part-of-speech tagging, and object nouns, attribute adjectives, actions or spatial relationship verbs are extracted.

[0034] Based on the query command described in the dependency tree segmentation, candidate clauses are generated for parallel, subordinate, and adverbial structures; each clause is mapped to a semantic framework containing subject, relation, object, and attribute.

[0035] Merge homogeneous clauses and remove semantically empty phrases to obtain the subquery;

[0036] The step of selecting triples related to the subquery from the scene graph of each image and concatenating them into a candidate subgraph includes:

[0037] The subquery vectors are sent in parallel to the subgraph index of the image to retrieve relevant triple information in the scene graph of the image;

[0038] Candidate triples are obtained by filtering triples according to the index hierarchy, which includes object category index, relation type index, and attribute index.

[0039] Calculate the semantic similarity between the candidate triples and the subquery vector, check the relation direction, and remove triples that do not meet the requirements;

[0040] By combining the neighborhood quality of the triples in the original scene graph, the context is reordered to obtain the relevant set of triples;

[0041] Relevant ternary pairs that share nodes in the same retrieved image, whether for the subject or object, are combined into candidate subgraphs.

[0042] Secondly, this application provides a multimodal image retrieval device based on a scene graph, comprising:

[0043] The model training module is used to acquire images and preprocess them, extract features from the preprocessed images to obtain image feature vectors, match the image feature vectors with a pre-trained language knowledge base to generate a full-image description of the image, and convert the full-image description into a structured scene graph and write it into a scene graph database. The scene graph is represented in the form of triples, and the triples include a subject object, a relation type, and an object object.

[0044] The online retrieval module is used to receive the user's query command and decompose the query command into multiple subqueries; select triples related to the subqueries from the scene graph of each image and concatenate them into candidate subgraphs; concatenate the subqueries into query subgraphs, calculate the similarity between each candidate subgraph and the query subgraph, and sort the images in descending order of the similarity scores of the candidate subgraphs before feeding them back to the user.

[0045] Thirdly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the scene graph-based multimodal image retrieval method.

[0046] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the scene graph-based multimodal image retrieval method.

[0047] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the scene graph-based multimodal image retrieval method.

[0048] As can be seen from the above technical solution, this application provides a multimodal image retrieval method and apparatus based on scene graphs. Utilizing the powerful understanding capabilities of a multimodal large language model, after receiving a preprocessed image, it extracts image features through its built-in visual encoder, and then uses the generation capabilities of the language model to directly output a structured scene graph description. This enables the model to achieve fine-grained semantic matching during the retrieval process through a progressive query retrieval mechanism of query decomposition and subgraph construction, ultimately achieving high-precision text-image retrieval. Attached Figure Description

[0049] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a flowchart illustrating the multimodal image retrieval method based on scene graphs in an embodiment of this application.

[0051] Figure 2 This is a schematic diagram illustrating the process from query input to result output in the scene graph-based multimodal image retrieval method according to the embodiments of this application;

[0052] Figures 3-5 This is a schematic diagram illustrating the conversion of an image into a scene graph in the scene graph-based multimodal image retrieval method of this application embodiment;

[0053] Figure 6 This is a schematic diagram of the image retrieval results of the scene graph-based multimodal image retrieval method in the embodiments of this application;

[0054] Figure 7 This is a structural diagram of the scene graph-based multimodal image retrieval device in the embodiments of this application;

[0055] Figure 8 This is a schematic diagram of the structure of the electronic device in the embodiments of this application.

[0056] Figure label:

[0057] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0060] In view of the problems existing in the prior art, this application provides a multimodal image retrieval method and apparatus based on scene graph. It utilizes the powerful understanding ability of multimodal large language model. After receiving the preprocessed image, it extracts image features through its built-in visual encoder. Then, it uses the generation ability of language model to directly output a structured scene graph description. This enables the model to achieve fine-grained semantic matching through a progressive query retrieval mechanism of query decomposition and subgraph construction during the retrieval process, and finally achieves high-precision text-image retrieval.

[0061] To improve the accuracy of text-image retrieval, this application provides an embodiment of a multimodal image retrieval method based on scene graphs, see [link to embodiment]. Figure 1 The scene graph-based multimodal image retrieval method specifically includes the following:

[0062] Step S101: Acquire an image and preprocess it; extract features from the preprocessed image to obtain an image feature vector; match the image feature vector with a pre-trained language knowledge base to generate a full-image description of the image; convert the full-image description into a structured scene graph and write it into a scene graph database, wherein the scene graph is represented in the form of triples; the triples include a subject object, a relation type, and an object object.

[0063] This embodiment utilizes a multimodal large language model (such as Florence-2, BLIP-2, etc.) for deep analysis of the input image. This model, through joint vision-language pre-training, possesses powerful image understanding capabilities. The specific implementation process is as follows: In the image preprocessing stage, the input image is adjusted to the standard size required by the model (usually 224×224 or 384×384 pixels) and standardized. The multimodal large language model receives the preprocessed image, extracts image features through its built-in visual encoder, and then utilizes the language model's generative capabilities to directly output a structured scene graph description.

[0064] Optionally, in this embodiment, the scene graph generation includes three levels of information:

[0065] - Object layer: Identifies all entity objects in an image, including their category, location, and key attributes.

[0066] - Relationship layer: Identifies spatial relationships (such as "above", "adjacent") and semantic relationships (such as "holding", "wearing") between objects.

[0067] - Attribute layer: Extracts detailed feature descriptions of objects (such as color, shape, material, state, etc.).

[0068] Optionally, in this embodiment, the scene graph is represented in a standardized triple form: (subject object, relationship type, object object), while retaining the attribute information of each object. That is, the scene graph includes: Nodes: objects and entities in the image; Edges: relationships between objects; Attributes: feature descriptions of objects.

[0069] For example, given an image containing "a woman holding a green skincare bottle," the generated scene graph would include:

[0070] - Object nodes: [Female: {Young, Long hair, Light-colored top}], [Skincare bottle: {Green, Transparent, Cylindrical}], [Hands: {Slender, Fair}]

[0071] - Relationship elements: (hand, holding, skincare bottle), (woman, possessing, hand)

[0072] Optionally, this embodiment provides specific steps for extracting image features:

[0073] 1. Image preprocessing:

[0074] After the image is input, it is first normalized (e.g., normalizing pixel values ​​to the [0,1] range) and adjusted to the input size required by the model (e.g., 224x224 or 384x384 pixels) to ensure that the input meets the requirements of the visual encoder. Image enhancement (e.g., rotation, cropping, flipping) may also be included in this process to improve the robustness of the model.

[0075] 2. Input to the visual encoder of the multimodal large language model:

[0076] The preprocessed image is fed into the visual encoder part of the multimodal large language model. Common visual encoders are based on convolutional neural networks (CNNs) or transformer architectures, such as ViT (Visual Transformer) or ResNet. The visual encoder maps the image to a high-dimensional feature space, where vectors represent low-level features of the image (such as color, texture, shape, etc.) and high-level semantic information (such as object category, relative position, etc.).

[0077] 3. Feature vector extraction and fusion:

[0078] The feature vectors output by a visual encoder not only contain information about each object in an image, but also capture the relationships between them, such as the relative position, size, and shape of the objects. These feature vectors are typically processed through a fully connected layer or pooling operation to further integrate the visual information from each layer, resulting in a more condensed image feature vector.

[0079] 4. Transfer of feature information to the language model:

[0080] The extracted image feature vectors are passed as input to the language generation part of the large language model. These features contain semantic and structural information about the image, and the language model generates a natural language description based on these features.

[0081] II. Steps to directly output a structured scene graph description using the generative capabilities of a language model.

[0082] 1. Mapping from image features to language generation:

[0083] After receiving image features, the language model matches these features with an internally pre-trained language knowledge base. By understanding the relationship between image features and language expressions, the language model generates appropriate natural language descriptions. In this process, the model utilizes its built-in generative capabilities (such as GPT, BERT, and other structures) to generate natural language descriptions consistent with the image content.

[0084] 2. Generate a structured scene graph description:

[0085] The generated natural language description will be further transformed into a structured scene graph. The scene graph is represented in the form of triples, typically including (subject object, relation, object object) and their attributes. For example, the model might generate a triple like "hand holding a beer bottle," where "hand" is the subject, "holding" is the relation, and "beer bottle" is the object. In this step, the generated structured description not only includes the objects themselves but also indicates the spatial and semantic relationships between them (e.g., "above," "next to," etc.).

[0086] 3. Attribute extraction and hierarchical representation:

[0087] In addition to object categories and relationships, scene graphs also need to extract detailed attributes for each object, such as color, material, and shape. For example, the attributes of a "beer bottle" might include "transparent," "green," and "glass material." Scene graphs organize this information in a hierarchical manner; for example, they first identify the object hierarchy, then the relationship hierarchy, and finally supplement the attribute hierarchy for each object.

[0088] 4. Output the final structured scene diagram:

[0089] Finally, the model integrates all the generated triples and attribute information to form a complete structured scene graph. For example, the scene graph might look like this:

[0090] Objects: [Hand: {Young, light-skinned}], [Beer bottle: {Transparent, green}]

[0091] Relationship: (hand, holding, beer bottle)

[0092] Attributes: [Beer bottle: {glass, cylindrical}].

[0093] Optionally, this embodiment provides two triplet encoding schemes and determines the optimal strategy through comparative experiments:

[0094] Option 1: Overall Coding Scheme

[0095] The "subject-relation-object" is encoded as a complete semantic unit. The advantage of this approach is that it maintains the semantic integrity of the triple and avoids semantic loss caused by information segmentation. In practice, a special delimiter is used to connect the three parts, such as "hand[SEP] holding[SEP] beer can", and then the entire structure is encoded using a pre-trained text encoder.

[0096] Option 2: Separate Encoding Scheme

[0097] The subject, relation, and object are encoded independently, and then combined using a feature fusion strategy. This approach allows the model to learn representations of different types of semantic information separately, providing greater flexibility.

[0098] The results of the comparative experiment show that:

[0099] The overall coding scheme performs better on simple queries, with an accuracy rate of 91.4%.

[0100] - The split encoding scheme performs better on complex multi-relation queries, achieving an accuracy of 88.7%.

[0101] - The hybrid strategy combining the two approaches yielded the best results, achieving an overall accuracy of 93.2%.

[0102] Optionally, this triple encoding strategy can be refined as follows:

[0103] 1. Complexity assessment and sample splitting: For each triple, calculate the topological depth, node degree, dependency depth, and number of modifiers, and determine whether it is a simple or complex relationship based on the comprehensive score.

[0104] 2. Two-path encoding network construction: Overall encoding path: directly input the encoder with the "subject-relationship-object" sequence; Separate encoding path: encode the subject vector, relation vector, and object vector respectively, and fuse them into a triple vector through learnable attention.

[0105] 3. Gated weight training: A lightweight gated network is designed to output two path weights after inputting features of varying complexity. The overall loss includes triple-subgraph contrast loss and relation classification loss, which together drive the gated network to automatically select the optimal encoding strategy on samples of different complexities.

[0106] 4. Fusion strategy solidification: After completing the grid search on the validation set, fix the gating threshold; export the final triplet embedding generation module for direct use in subsequent graph network and subgraph matching.

[0107] Optionally, this embodiment uses a graph transforwer to convert the scene graph into a graph convolutional network (GCN) architecture. Compared to the traditional single-layer processing method, the multi-layer architecture can capture richer graph structure information.

[0108] Network architecture design:

[0109] A 3-layer GCN structure is adopted, with each layer containing an attention mechanism and residual connections:

[0110] - Layer 1: Focus on learning local node features

[0111] - Layer 2: Learning the direct relationships between nodes

[0112] - Layer 3: Capture global graph structure information

[0113] Feature fusion strategy:

[0114] Two implementation paths are provided:

[0115] - Path 1: (node_emb + edge_emb) → Layer 3 GCN → concat → MLP → triplet_emb

[0116] - Path 2: (node_emb, edge_emb) → concat → MLP → Layer 3 GCN → triplet_emb

[0117] Experimental comparisons show that path 1 performs better in maintaining semantic coherence, with an mAP of 0.847 for the overall retrieval task, while path 2 has an mAP of 0.823.

[0118] Implementation of attention mechanism:

[0119] A self-attention mechanism (GAT) is integrated into each layer of the GCN, enabling the model to adaptively focus on important parts of the graph structure. The attention weights are calculated considering node feature similarity, edge importance, and global graph structure information.

[0120] Optionally, this stage can be further refined into a three-layer continuous graph message passing process:

[0121] 1. Initialization of node and edge features: Node vector: Visual features, category embedding, and attribute word embedding are concatenated and mapped by a linear layer; Edge vector: Relation type embedding and spatial orientation embedding are concatenated.

[0122] 2. First-level local aggregation: Nodes collect the attributes of their one-hop neighbors, integrate them, update their own semantic representation, and at the same time perform a slight update on the edge vector to eliminate attribute noise.

[0123] 3. Second-layer relationship reinforcement: Introduce relationship-specific attention to enable the relationship vector to actively modulate the information of the two end nodes; after the edge is updated, the new edge vector is injected back into the two end nodes to strengthen the action or spatial connection between objects.

[0124] 4. Third-layer global context fusion: Attention pooling is performed on all nodes in the graph to obtain graph-level context vectors, which are then fed back to each node; non-adjacent nodes are allowed to establish long-range dependencies through graph-level vectors to capture multi-hop link semantics.

[0125] 5. Residuals and Normalization: The residuals of the output and input of each layer are added together and then normalized to suppress oversmoothing; the DropEdge technique is used to randomly discard some edges to enhance generalization ability.

[0126] 6. Multi-granularity readout: Node and edge vectors are aggregated into triplet embeddings through learnable aggregation; connected triplets are further pooled to obtain subgraph embeddings; global pooling of all graph nodes generates full graph embeddings, preparing for similarity scoring and interpretation display.

[0127] After completing the above three-layer message passing, the system obtains a multi-granularity graph embedding that simultaneously contains local object information, explicit relation features, and global structural context, laying a semantically complete and structurally sensitive representation foundation for subsequent multi-dimensional similarity fusion.

[0128] Step S102: Receive the user's query command and decompose the query command into multiple subqueries; select triples related to the subqueries from the scene graph of each image and concatenate them into candidate subgraphs; concatenate the subqueries into query subgraphs, calculate the similarity between each candidate subgraph and the query subgraph, and sort the images in descending order of the similarity scores of the candidate subgraphs and then feed them back to the user.

[0129] Optionally, this embodiment employs a progressive reasoning retrieval process:

[0130] I. Query, Understanding, and Breakdown:

[0131] Complex queries entered by users are first processed by intelligent decomposition:

[0132] 1. Semantic Analysis: Utilizing dependency syntax and part-of-speech tagging, the system locates the subject, object, action verbs, and spatial prepositions in the query. Qualifiers such as color, material, brand, and posture are recorded as attributes to identify the core objects, actions, and relationships within the query.

[0133] 2. Intent Understanding: Determine whether the query is a single-object description, a two-object relationship description, or a complex multi-object scenario. If the query contains logical operators (such as "simultaneously" or "and"), mark it as a combined intent to determine the user's search intent and key focus.

[0134] 3. Query Decomposition: Complex queries are decomposed into multiple interrelated subqueries using the "object-relationship-object (+attribute)" template. Each subquery maintains semantic integrity and avoids disrupting action or spatial direction information.

[0135] For example, for the query "a woman holding a green skincare bottle to demonstrate the product", the system breaks it down into:

[0136] Subquery 1: Female + Young characteristics

[0137] -Subquery 2: Handheld gestures + Display posture

[0138] -Subquery 3: Green skincare bottle + product attributes

[0139] 4. Subquery standardization: Synonyms are merged ("holding" and "grasping" are merged into "grasping"); locative words are standardized by direction ("top" and "above" are merged into "above").

[0140] II. Triple Retrieval and Matching:

[0141] 1. Three-level index structure

[0142] For each subquery, the system retrieves relevant triple information from the scene graph database pre-built in the candidate set:

[0143] - Preliminary screening based on semantic similarity

[0144] - Precise filtering based on structure matching

[0145] - Context-dependent reordering

[0146] The retrieval process employs a multi-level index structure. The first level is based on object category indexing—filtering scene images by subject or object category. The second level is based on relationship type indexing—filtering by action / location relationship. The third level is based on attribute feature indexing—performing precise matching based on attributes such as color and material. This hierarchical indexing improves retrieval efficiency by 73.5%.

[0147] 2. Preliminary Candidate Selection

[0148] For each subquery, retrieve the set of triples that satisfy the object + relation constraints from the index to form a candidate list.

[0149] 3. Semantic similarity coarse ranking

[0150] Perform a text-vector cosine comparison on the candidate triples and subqueries, and retain the top n scores.

[0151] 4. Structural Filtering

[0152] Filter triples where the subject, object, and direction are inconsistent with the subquery (e.g., "bottle above hand" ≠ "hand above bottle").

[0153] III. Subgraph Construction and Optimization:

[0154] 1. Stitching Strategy: Relevant triples containing shared nodes in the same candidate image (subject or object) are merged into a connected subgraph using an intelligent algorithm; if a shared node is missing, it is filled in using the shortest hop edge, but no more than once.

[0155] 2. Semantic Consistency Check: When opposite directions or actions occur in the same object-relation-object combination, the most reliable one is retained based on distance score or confidence level, ensuring that the triples in the subgraph are semantically mutually supportive.

[0156] 3. Structural integrity verification: Ensure that all nodes required by the subquery can be connected in the subgraph via a path of no more than two hops. If a break occurs, discard the split part or downgrade it to a partial matching candidate to ensure that the subgraph has a reasonable graph structure.

[0157] 4. Redundant information filtering: Delete leaf nodes that contribute nothing to the query and have a degree of 1 in the entire graph; remove irrelevant or duplicate triples to ensure a streamlined subgraph.

[0158] IV. Similarity Calculation and Ranking:

[0159] 1. Three-dimensional similarity calculation

[0160] Semantic similarity: based on cosine similarity between the query side graph text vector and the candidate side graph text vector. Structural similarity: comparing the node-edge morphology of the two subgraphs and using normalized graph edit distance.

[0161] Attribute similarity: The intersection-union ratio (IUR) of attribute sets of nodes with the same name.

[0162] 2. Weight fusion

[0163] The final similarity score is obtained through weighted fusion:

[0164] Final score = α × semantic similarity + β × structural similarity + γ × attribute similarity

[0165] The weight parameters were optimized using a grid search: α = 0.5, β = 0.3, γ = 0.2. The final score = 0.5 × semantics + 0.3 × structure + 0.2 × attribute.

[0166] 3. Result sorting

[0167] Sort by final score in descending order; if scores are the same, prioritize those with higher structural integrity. Return all triples in the final matched subgraph in natural language for front-end highlighting or display.

[0168] Experimental results show that, compared with the traditional direct matching method, the progressive retrieval mechanism improves the accuracy of complex queries by 34.6% and the retrieval speed by 28.3%.

[0169] Optionally, this embodiment also provides detailed steps for query understanding and decomposition:

[0170] 1. Text preprocessing: Perform word segmentation, part-of-speech tagging, dependency syntax analysis, and named entity recognition on user input.

[0171] 2. Target element extraction: Identify explicitly appearing object nouns, attribute adjectives, action or spatial relation verbs.

[0172] 3. Clause segmentation: Segment complex sentences based on dependency trees; generate candidate clauses for coordinate, subordinate, and adverbial structures.

[0173] 4. Semantic Role Slot Filling: Mapping each clause to a (subject, relation, object, attribute) semantic frame—filling in missing subjects or objects.

[0174] 5. Subquery redundancy removal: merge homogeneous clauses and remove semantically empty phrases to obtain m≤5 subqueries.

[0175] 6. Query subgraph generation: Use the triple encoder from steps 3-4 to convert each subquery into a triple embedding and connect them to form a query subgraph; at the same time, read out the vector q_sub_emb and the structure and attribute cache.

[0176] Optionally, this embodiment also provides refined steps for triple retrieval and matching:

[0177] 1. Subquery parsing

[0178] Extract the following from the subquery: candidate categories of subject / object, relation words, and key attribute words (color, material, etc.). If the subquery itself contains multiple relations, further refine it into multiple "triple-level sub-subqueries".

[0179] 2. Filter by index hierarchy (three-level or similar multi-level index)

[0180] Level 1: Object Category Index – First, find triples that match the subject / object categories.

[0181] Level 2: Relation type index – further filtering out triples that do not match the relation type.

[0182] Level 3: Attribute Index – Precise filtering using attribute terms (color, brand, etc.).

[0183] 3. Semantic similarity and orientation verification

[0184] For the remaining candidate triples, calculate the semantic similarity (cosine, etc.) between their embedding vectors and subquery vectors. Also, verify the relation direction ("hand above bottle" ≠ "bottle above hand"), and discard any that do not match.

[0185] 4. Context reordering

[0186] By rearranging the triples based on their neighborhood quality in the original scene graph (whether they share nodes with other hit triples or form a more complete link), we obtain the set of "related triples".

[0187] Optionally, this embodiment also provides detailed steps for similarity calculation and ranking:

[0188] In the similarity calculation and ranking process, the final score is based on the similarity between the two subgraphs, specifically the query side graph and the candidate side graph. In detail:

[0189] 1. Querying the Q-Subgraph

[0190] Starting point: The query decomposer breaks down a complex query into several subqueries (S1, S2, ... Sn).

[0191] The system semantically "merges" these subqueries: combining their respective triples or object-relation sets into a connected, deduplicated, and logically conflict-free query subgraph. This step is like assembling several puzzle pieces into a complete picture, resulting in a Q-Subgraph.

[0192] Then, a text encoder is used to read the Q-Subgraph into a vector q_sub_emb and save its structure / attribute information.

[0193] 2. Candidate Graph Side Subgraph (G-Subgraph)

[0194] In each candidate graph, first use each subquery S i Find the set of corresponding triples. Concatenate the matched triples according to the rules of shared node connectivity and semantic consistency to obtain a candidate subgraph that best fits the entire set of subqueries. Similarly, read the vector g_sub_emb and retain the topology and attribute set of the subgraph.

[0195] 3. Similarity Calculation

[0196] Semantic vector cosine - cos(q_sub_emb, g_sub_emb)

[0197] Structural similarity – comparing the node-edge topology of two subgraphs

[0198] Attribute similarity – comparing the attribute sets of two adjacent nodes

[0199] The final score is obtained by weighting the results using α·semantic, β·structural, and γ·attributive factors.

[0200] Therefore, this step involves each subquery first locating a local part of the candidate graph, then stitching them together to form a query side graph and a candidate side graph, and finally comparing the similarity between the two "merged subgraphs".

[0201] Optionally, this embodiment also includes a preliminary search (full-image nearest neighbor search), the steps of which are as follows:

[0202] 1. Query vector generation: The complete query sentence is fed into the text encoder to obtain q_full_emb.

[0203] 2. ANN retrieval: Perform a K (~1000) nearest neighbor search in the full graph HNSW index using q_full_emb.

[0204] 3. Coarse filtering: Remove images with scores below the global threshold; if the user specifies metadata (brand, resolution, time), filter immediately.

[0205] 4. Adaptive candidate set size: When the recall is less than n_min, the threshold is automatically relaxed or an inverted index is used to supplement; when the number of candidates exceeds n_max, the Top-n_max is retained.

[0206] Optionally, this embodiment also includes joint fine-tuning of the image encoder and the text encoder, that is, using the CLIP architecture to train the parameters of the image encoder and the text encoder simultaneously, in order to focus on optimizing the encoder's ability to understand text descriptions and corresponding image content at different granularities.

[0207] The image encoder and text encoder are key modules for "image-to-text conversion, text-to-text conversion, and vector alignment with visual representation." They are trained offline and then online to convert user queries, subqueries, and images into vectors in real time. This supports steps such as triple encoding, full-image recall, triple matching, subgraph similarity, and result interpretation, and runs throughout the entire training and inference chain of the scheme. To improve the brevity of the scheme description, descriptions of some encoding operations are omitted in this paper.

[0208] The training data for this joint fine-tuning employs a carefully designed matching strategy:

[0209] - The ratio of full image description to triplet description is set to 4:1.

[0210] - Full-image descriptions are randomly extracted from a multi-granularity description library, including scene-level descriptions, object-level descriptions, and detail-level descriptions.

[0211] - Triple descriptions are randomly extracted from a relation interpretation database, focusing on the precise expression of relationships between objects.

[0212] The training sample construction process in this stage is as follows: For each image-text pair, a corresponding scene graph is first generated, and then all triplet relations are extracted from the scene graph. Each triplet is converted into descriptive text using a natural language template, such as "(hand, holding, beer can)" being converted into "The hand is holding a tilted beer can and is making a pouring motion".

[0213] Experimental results show that, compared with the traditional 1:1 data ratio, the 4:1 ratio strategy improved the accuracy of the model in text understanding tasks by 12.3% and in understanding complex relationships by 21.7%.

[0214] Optionally, the joint fine-tuning of the image encoder and text encoder can be refined into the following steps:

[0215] 1. Multi-source corpus aggregation: The system first automatically collects three types of text for each image: whole-image descriptions generated by a multimodal large language model; object-level and attribute-level short sentences obtained by parsing scene graph nodes; and reference descriptions introduced from public datasets. Subsequently, the system uses a large language model to perform synonym rewriting and style diversification, so that the same image has at least three to five semantically equivalent but worded whole-image descriptions.

[0216] 2. Automatic expansion of relational corpus: Expanding the context... Figure Three Tuples are filled into a preset relation template to obtain basic sentence patterns. Then, the language model is called to generate two to three rewritten sentences for each sentence pattern, and a semantic consistency discriminator is used to filter out ambiguous or overgeneralized versions, forming a relation interpretation library.

[0217] 3. Hierarchical Labeling and Quality Inspection: Each piece of text is labeled with granularity (scene-level, object-level, attribute-level, relation-level) and its consistency with the corresponding key information in the image is verified. Semantic duplication detection and language fluency detection are used to remove redundant or low-quality sentences.

[0218] 4. Contrastive Learning Batch Construction: Scene-level text and relation-level text are mixed into the same batch at a ratio of 4:1; object-level and attribute-level text are randomly inserted to increase semantic span. For each positive text, three types of negative examples are randomly selected: a completely unrelated image, an image of the same type of object but with incorrect relationships, and a sentence containing different relationships for the same image. These samples are input into the text encoder and image encoder to generate corresponding text embedding vectors and image embedding vectors.

[0219] 5. Multi-stage parameter optimization: First, freeze the image encoder on the visual side and fine-tune the text encoder separately for 100,000 to 150,000 steps to enable the text encoder to initially align the image embedding vector. Then, unfreeze the visual-text alignment projection layer (used for vector alignment of text and images) and perform joint fine-tuning for 30,000 to 50,000 steps until the accuracy of text-image retrieval exceeds the preset threshold on the validation set.

[0220] After fine-tuning, the text encoder is compatible with the semantic expression of long sentences, short sentences, and single relational sentences, and can convert them into text vectors aligned with the image embedding vectors, providing a unified text vector output for subsequent retrieval chains.

[0221] Optionally, this embodiment also includes specialized training on graph retrieval capabilities, focusing on improving the model's image retrieval ability. This training employs the following strategies:

[0222] -Constructing queries- Embedding data pairs across the entire graph to train the model's global retrieval capabilities

[0223] -Construct subqueries-subgraph embedding data pairs to train the model's local exact matching capability.

[0224] - Subgraph construction logic: Through semantic relevance analysis, related ternaries are combined to form semantically complete subgraphs.

[0225] The specific implementation of the subgraph construction algorithm: For a complex query "hand holding beer and pouring it into a glass", the system first identifies the core objects (hand, beer, glass) and relationships (holding, pouring), then retrieves triples containing these elements in the scene graph, and finally combines the relevant triples to form a complete subgraph representation.

[0226] After this training phase, the model's recall@5 metric for complex multi-object queries improved from 0.42 in the traditional method to 0.78, an improvement of 85.7%.

[0227] Optionally, the specialized training for graph retrieval capabilities can be broken down into the following steps:

[0228] 1. Query-Full Image Pairing Training: The whole sentence query vector encoded by the text encoder is paired with the whole image vector of the corresponding image to form a positive pair; the whole image vector of random sampling is paired with the query vector to form a negative pair; the global recall accuracy of the model in a million-level index is enhanced by contrast loss.

[0229] 2. Subquery generation and subgraph pairing: Complex queries are analyzed using syntax tree and key phrases are extracted to decompose them into several subqueries; based on the scene graph, the cluster of connected triples that is semantically closest to the subqueries is searched within the graph to generate positive subgraphs; negative subgraphs with three difficulty gradients are constructed: incorrect relations with the same object, incorrect objects with the same relation, and random unrelated subgraphs.

[0230] 3. Joint comparison and consistency loss: The comparison loss is calculated separately for the whole graph pair and the subgraph pair using an independent tower structure; additional consistency regularization is introduced to ensure that the matching direction of the same query is consistent in the global and local channels, avoiding the drift of the two paths.

[0231] 4. Nearest Neighbor Index Offline Construction: The optimized whole graph embedding is written to the full graph HNSW index; the subgraph embedding is written to the multi-segment inverted bucket vector index, and the subgraph-image-triple mapping is recorded.

[0232] After this training phase, the system can complete first-level recall in milliseconds on a graph database of millions of graphs and has the ability to quickly match subqueries locally.

[0233] As can be seen from the above description, the scene graph-based multimodal image retrieval method provided in this application can utilize the powerful understanding capabilities of a multimodal large language model. After receiving a preprocessed image, it extracts image features through its built-in visual encoder and then uses the generation capabilities of the language model to directly output a structured scene graph description. This enables the model to achieve fine-grained semantic matching during the retrieval process through a progressive query retrieval mechanism of query decomposition and subgraph construction, ultimately achieving high-precision text-image retrieval.

[0234] To improve the accuracy of text-image retrieval, this application provides an embodiment of a scene graph-based multimodal image retrieval device for implementing all or part of the aforementioned scene graph-based multimodal image retrieval method. See [link to embodiment]. Figure 7 The scene graph-based multimodal image retrieval device specifically includes the following components:

[0235] The model training module 10 is used to acquire images and preprocess them, extract features from the preprocessed images to obtain image feature vectors, match the image feature vectors with a pre-trained language knowledge base to generate a full-image description of the image, and convert the full-image description into a structured scene graph and write it into a scene graph database. The scene graph is represented in the form of triples. The triples include a subject object, a relation type, and an object object.

[0236] The online retrieval module 20 is used to receive the user's query command and decompose the query command into multiple subqueries; select triples related to the subqueries from the scene graph of each image and concatenate them into candidate subgraphs; concatenate the subqueries into query subgraphs, calculate the similarity between each candidate subgraph and the query subgraph, and sort the images in descending order of the similarity scores of the candidate subgraphs and then feed them back to the user.

[0237] As can be seen from the above description, the scene graph-based multimodal image retrieval device provided in this application embodiment can utilize the powerful understanding capabilities of the multimodal large language model. After receiving a preprocessed image, it extracts image features through its built-in visual encoder and then uses the generation capabilities of the language model to directly output a structured scene graph description. This enables the model to achieve fine-grained semantic matching through a progressive query retrieval mechanism of query decomposition and subgraph construction during the retrieval process, ultimately achieving high-precision text-image retrieval.

[0238] From a hardware perspective, in order to improve the accuracy of text-image retrieval, this application provides an embodiment of an electronic device for implementing all or part of the scene graph-based multimodal image retrieval method, wherein the electronic device specifically includes the following:

[0239] The system comprises a processor, memory, a communications interface, and a bus; wherein the processor, memory, and communications interface communicate with each other via the bus; the communications interface is used to realize information transmission between the scene graph-based multimodal image retrieval device and core business systems, user terminals, and related databases and other related devices; the logic controller can be a desktop computer, tablet computer, or mobile terminal, etc., and this embodiment is not limited to these. In this embodiment, the logic controller can be implemented with reference to the embodiments of the scene graph-based multimodal image retrieval method and the scene graph-based multimodal image retrieval device in the embodiments, the content of which is incorporated herein, and repeated details will not be described again.

[0240] It is understood that the user terminal may include smartphones, tablet computers, network set-top boxes, portable computers, desktop computers, personal digital assistants (PDAs), in-vehicle devices, smart wearable devices, etc. Among these, the smart wearable devices may include smart glasses, smartwatches, smart bracelets, etc.

[0241] In practical applications, parts of the scene graph-based multimodal image retrieval method can be executed on the electronic device side as described above, or all operations can be completed in the client device. The specific choice depends on the processing power of the client device and the limitations of the user's usage scenario. This application does not impose any limitations on this. If all operations are completed in the client device, the client device may further include a processor.

[0242] The aforementioned client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission. The server may include a server on the task scheduling center side; in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, a server cluster consisting of multiple servers, or a distributed server structure.

[0243] Figure 8 This is a schematic block diagram illustrating the system configuration of the electronic device 9600 according to an embodiment of this application. Figure 8 As shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It is worth noting that... Figure 8 This is an example; other types of structures can also be used to supplement or replace this structure to achieve telecommunications functions or other functions.

[0244] In one embodiment, the scene graph-based multimodal image retrieval method can be integrated into the central processing unit 9100. The central processing unit 9100 can be configured to perform the following control:

[0245] Step S101: Acquire an image and preprocess it; extract features from the preprocessed image to obtain an image feature vector; match the image feature vector with a pre-trained language knowledge base to generate a full-image description of the image; convert the full-image description into a structured scene graph and write it into a scene graph database, wherein the scene graph is represented in the form of triples; the triples include a subject object, a relation type, and an object object.

[0246] Step S102: Receive the user's query command and decompose the query command into multiple subqueries; select triples related to the subqueries from the scene graph of each image and concatenate them into candidate subgraphs; concatenate the subqueries into query subgraphs, calculate the similarity between each candidate subgraph and the query subgraph, and sort the images in descending order of the similarity scores of the candidate subgraphs and then feed them back to the user.

[0247] As can be seen from the above description, the electronic device provided in this application embodiment utilizes the powerful understanding capabilities of a multimodal large language model. After receiving a preprocessed image, it extracts image features through its built-in visual encoder and then uses the generation capabilities of the language model to directly output a structured scene graph description. This enables the model to achieve fine-grained semantic matching through a progressive query retrieval mechanism of query decomposition and subgraph construction during the retrieval process, ultimately achieving high-precision text-image retrieval.

[0248] In another embodiment, the scene graph-based multimodal image retrieval device can be configured separately from the central processing unit 9100. For example, the scene graph-based multimodal image retrieval device can be configured as a chip connected to the central processing unit 9100, and the scene graph-based multimodal image retrieval method function can be implemented through the control of the central processing unit.

[0249] like Figure 8 As shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It is worth noting that the electronic device 9600 does not necessarily need to include these components. Figure 8 All components shown; in addition, the electronic device 9600 may also include Figure 8 For components not shown, please refer to existing technology.

[0250] like Figure 8As shown, the central processing unit 9100, sometimes also referred to as a controller or operating control, may include a microprocessor or other processor device and / or logic device, which receives inputs and controls the operation of various components of the electronic device 9600.

[0251] The memory 9140 may be, for example, one or more of a cache, flash memory, hard drive, removable media, volatile memory, non-volatile memory, or other suitable devices. It may store the aforementioned failure-related information, and also store a program for executing that information. The central processing unit 9100 may execute the program stored in the memory 9140 to perform information storage or processing, etc.

[0252] Input unit 9120 provides input to central processing unit 9100. Input unit 9120 may be, for example, a keypad or touch input device. Power supply 9170 provides power to electronic device 9600. Display 9160 displays images and text. Display may be, for example, an LCD display, but is not limited thereto.

[0253] The memory 9140 can be a solid-state memory, such as a read-only memory (ROM), random access memory (RAM), a SIM card, etc. It can also be a memory that retains information even when power is off, can be selectively erased, and contains more data; examples of this type of memory are sometimes referred to as EPROMs. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 may include an application / function storage unit 9142 for storing application programs and function programs or processes for executing the operation of the electronic device 9600 via the central processing unit 9100.

[0254] The memory 9140 may also include a data storage unit 9143 for storing data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 may include various drivers for the electronic device's communication functions and / or for performing other functions of the electronic device (such as messaging applications, address book applications, etc.).

[0255] The communication module 9110 is a transmitter / receiver that sends and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which is the same as in a conventional mobile communication terminal.

[0256] Based on different communication technologies, multiple communication modules 9110 can be configured in the same electronic device, such as cellular network modules, Bluetooth modules, and / or wireless LAN modules. The communication module 9110 (transmitter / receiver) is also coupled to a speaker 9131 and a microphone 9132 via an audio processor 9130 to provide audio output via the speaker 9131 and receive audio input from the microphone 9132, thereby realizing typical telecommunications functions. The audio processor 9130 may include any suitable buffer, decoder, amplifier, etc. Additionally, the audio processor 9130 is coupled to a central processing unit 9100, enabling on-device recording via the microphone 9132 and on-device playback of stored audio via the speaker 9131.

[0257] Embodiments of this application also provide a computer-readable storage medium capable of implementing all steps of the scene graph-based multimodal image retrieval method with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all steps of the scene graph-based multimodal image retrieval method with a server or client as the execution subject in the above embodiments. For example, when the processor executes the computer program, it implements the following steps:

[0258] Step S101: Acquire an image and preprocess it; extract features from the preprocessed image to obtain an image feature vector; match the image feature vector with a pre-trained language knowledge base to generate a full-image description of the image; convert the full-image description into a structured scene graph and write it into a scene graph database, wherein the scene graph is represented in the form of triples; the triples include a subject object, a relation type, and an object object.

[0259] Step S102: Receive the user's query command and decompose the query command into multiple subqueries; select triples related to the subqueries from the scene graph of each image and concatenate them into candidate subgraphs; concatenate the subqueries into query subgraphs, calculate the similarity between each candidate subgraph and the query subgraph, and sort the images in descending order of the similarity scores of the candidate subgraphs and then feed them back to the user.

[0260] As can be seen from the above description, the computer-readable storage medium provided in this application embodiment utilizes the powerful understanding capabilities of a multimodal large language model. After receiving a preprocessed image, it extracts image features through its built-in visual encoder and then uses the generation capabilities of the language model to directly output a structured scene graph description. This enables the model to achieve fine-grained semantic matching during the retrieval process through a progressive query retrieval mechanism of query decomposition and subgraph construction, ultimately achieving high-precision text-image retrieval.

[0261] Embodiments of this application also provide a computer program product capable of implementing all steps of the scene graph-based multimodal image retrieval method described above, where the execution subject is a server or client. When executed by a processor, this computer program / instruction implements the steps of the scene graph-based multimodal image retrieval method. For example, the computer program / instruction implements the following steps:

[0262] Step S101: Acquire an image and preprocess it; extract features from the preprocessed image to obtain an image feature vector; match the image feature vector with a pre-trained language knowledge base to generate a full-image description of the image; convert the full-image description into a structured scene graph and write it into a scene graph database, wherein the scene graph is represented in the form of triples; the triples include a subject object, a relation type, and an object object.

[0263] Step S102: Receive the user's query command and decompose the query command into multiple subqueries; select triples related to the subqueries from the scene graph of each image and concatenate them into candidate subgraphs; concatenate the subqueries into query subgraphs, calculate the similarity between each candidate subgraph and the query subgraph, and sort the images in descending order of the similarity scores of the candidate subgraphs and then feed them back to the user.

[0264] As can be seen from the above description, the computer program product provided in this application embodiment utilizes the powerful understanding capabilities of a multimodal large language model. After receiving a preprocessed image, it extracts image features through its built-in visual encoder and then uses the generation capabilities of the language model to directly output a structured scene graph description. This enables the model to achieve fine-grained semantic matching through a progressive query retrieval mechanism of query decomposition and subgraph construction during the retrieval process, ultimately achieving high-precision text-image retrieval.

[0265] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0266] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure One One or more processes and / or boxes Figure One A device that provides the functions specified in one or more boxes.

[0267] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure One One or more processes and / or boxes Figure One The function specified in one or more boxes.

[0268] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure One One or more processes and / or boxes Figure One The steps of the function specified in one or more boxes.

[0269] Specific embodiments have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A multimodal image retrieval method based on scene graphs, applied to a multimodal large language model, characterized in that, The method includes: The process involves acquiring an image and preprocessing it, then extracting features from the preprocessed image to obtain an image feature vector. This image feature vector is then matched with a pre-trained language knowledge base to generate a full-image description. The full-image description is converted into a structured scene graph and written into a scene graph database. The scene graph is represented in the form of triples, where each triple includes a subject object, a relation type, and an object object. The system receives the user's query command and decomposes the query command into multiple subqueries; it selects triples related to the subqueries from the scene graph of each image and concatenates them into candidate subgraphs; it concatenates the subqueries into query subgraphs, calculates the similarity between each candidate subgraph and the query subgraph, and then sorts the images in descending order of the similarity scores of the candidate subgraphs and returns them to the user.

2. The multimodal image retrieval method based on scene graphs according to claim 1, characterized in that, It also includes joint adjustment of the text encoder and the image encoder, the joint adjustment steps of which include: Construct a text corpus containing multi-granular descriptions at the scene, object, and attribute levels, as well as a relation description corpus obtained by templating and expanding scene graph triples with a large language model; in the training batch, positive samples are extracted at a sampling ratio of 4:1 for full graph descriptions and relation descriptions, and negative samples are collected at the same time. Positive and negative samples are input into a text encoder to obtain a text embedding vector, and an image is input into an image encoder to obtain an image embedding vector. The text embedding vector and the image embedding vector are aligned in the same semantic space. The text embedding vector and the image embedding vector are used together in the contrastive learning loss calculation to maximize the similarity of positive samples and minimize the similarity of negative samples; the text encoder parameters are iteratively updated until the set text-image retrieval accuracy threshold is reached.

3. The multimodal image retrieval method based on scene graphs according to claim 1, characterized in that, It also includes training of graph retrieval capabilities, which includes: Construct query-full graph embedding pairs and subquery-subgraph embedding pairs; the query-full graph embedding pair uses a full graph natural language description or a real user query as text, and is paired with a corresponding full graph embedding as a positive example; the subquery-subgraph embedding pair is to decompose the query into several subqueries, and use the connected subgraph embedding that matches the semantics of each subquery as a positive example; A dual-tower or multi-tower contrastive learning loss function is used to jointly optimize full-graph matching and subgraph matching, and a hard example mining mechanism is introduced to dynamically update negative samples; After training, an approximate nearest neighbor index for full graph embedding and subgraph embedding is constructed offline.

4. The multimodal image retrieval method based on scene graphs according to claim 1, characterized in that, The encoding strategy for the triples includes: Analyze the complexity of triples and, based on the analysis results, classify the triples into simple relations and complex relations; For simple relations, triplet encoding is performed using a holistic encoding method; for complex relations, triplet encoding is performed using a separate encoding method. The overall encoding method is as follows: the three parts of subject object, relation type and object object are connected using a special delimiter, and then the overall encoding is performed by a pre-trained text encoder; The method of separation encoding is as follows: the subject object, relation type and object object are encoded independently, and then combined through feature fusion strategy.

5. The multimodal image retrieval method based on scene graphs according to claim 1, characterized in that, The scene graph database is a multi-layer graph neural network structure, including: The scene graph is processed using a multi-layer graph neural network structure, which includes a graph neural network layer that focuses on learning local features of nodes, a graph neural network layer that learns direct relationships between nodes, and a graph neural network layer that captures global graph structure information.

6. The multimodal image retrieval method based on scene graphs according to claim 1, characterized in that, It also includes a preliminary search, which includes: Candidate images are obtained by retrieving related images from the scene graph database using query commands; After decomposing the query command into multiple subqueries, triples related to the subqueries are selected from the scene graph of each candidate graph and then concatenated to form a candidate subgraph.

7. The multimodal image retrieval method based on scene graphs according to claim 1, characterized in that, The step of decomposing the query command into multiple subqueries includes: Using dependency syntax and part-of-speech tagging, the query command is segmented, tagged with part-of-speech, and identified by dependency syntax and named entity recognition to extract object nouns, attribute adjectives, action or spatial relation verbs; Based on the query command described in the dependency tree segmentation, candidate clauses are generated for parallel, subordinate, and adverbial structures; each clause is mapped to a semantic framework containing subject, relation, object, and attribute. Merge homogeneous clauses and remove semantically empty phrases to obtain the subquery; The step of selecting triples related to the subquery from the scene graph of each image and concatenating them into a candidate subgraph includes: The subquery vectors are sent in parallel to the subgraph index of the image to retrieve relevant triple information in the scene graph of the image; Candidate triples are obtained by filtering triples according to the index hierarchy, which includes object category index, relation type index, and attribute index. Calculate the semantic similarity between the candidate triples and the subquery vector, check the relation direction, and remove triples that do not meet the requirements; By combining the neighborhood quality of the triples in the original scene graph, the context is reordered to obtain the relevant set of triples; Relevant ternary pairs that share nodes in the same retrieved image, whether for the subject or object, are combined into candidate subgraphs.

8. A multimodal image retrieval device based on scene graphs, characterized in that, The device includes: The model training module is used to acquire images and preprocess them, extract features from the preprocessed images to obtain image feature vectors, match the image feature vectors with a pre-trained language knowledge base to generate a full-image description of the image, and convert the full-image description into a structured scene graph and write it into a scene graph database. The scene graph is represented in the form of triples, and the triples include a subject object, a relation type, and an object object. The online retrieval module is used to receive the user's query command and decompose the query command into multiple subqueries; select triples related to the subqueries from the scene graph of each image and concatenate them into candidate subgraphs; concatenate the subqueries into query subgraphs, calculate the similarity between each candidate subgraph and the query subgraph, and sort the images in descending order of the similarity scores of the candidate subgraphs before feeding them back to the user.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the multimodal image retrieval method based on scene graphs as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the scene graph-based multimodal image retrieval method according to any one of claims 1 to 7.

Images