Picture retrieval method and system based on multi-heterogeneous graph fusion

Abstract

The invention provides a picture retrieval method and system based on multi-heterogeneous graph fusion, and the method comprises the following steps: obtaining query information outputted by a user, optimizing the query information, and generating an optimized query statement; encoding the optimized query information to obtain a user query code, and constructing a heterogeneous graph containing different types of heterogeneous nodes based on the user query code and the types of the summary information in the database; according to the similar relation between the heterogeneous nodes of the abstract codes in the heterogeneous graph and the nodes of the user query codes, only the heterogeneous nodes with the semantic relevancy with the user query codes exceeding a set threshold value are reserved, and one or more heterogeneous graph sub-graphs are formed; carrying out global importance sorting on the picture abstract nodes, and generating a semantic similarity ranking and an importance score ranking; and selecting a picture set meeting a set condition from the sorting result as a final query result. According to the method, a more accurate picture retrieval result with a better context understanding capability can be realized.

Description

Technical Field

[0001] This invention relates to the field of image retrieval technology, specifically to an image retrieval method based on multi-heterogeneous graph fusion, and also to an image retrieval system that implements the image retrieval method based on multi-heterogeneous graph fusion. Background Technology

[0002] Current image retrieval technologies mainly rely on the following two types of solutions:

[0003] I. Traditional keyword-based matching methods, such as manually labeling images (e.g., "landscape" or "people") or using shallow semantic models (e.g., TF-IDF (Term Frequency-Inverse Document Frequency, a common weighting technique for information retrieval and text mining) or bag-of-words models) to establish associations between text and images. While these methods can achieve basic retrieval, they are severely limited by the fixed vocabulary used for labeling and the shallow semantic parsing of the text. They cannot handle complex natural language queries (e.g., "city streets filled with reflections after rain, conveying a sense of loneliness") because they lack the ability to dynamically understand context, abstract descriptions, and implicit semantics.

[0004] Second, deep learning-based cross-modal retrieval models, such as dual-tower models like CLIP and ALIGN, map text and images to the same vector space for similarity matching through contrastive learning. However, these models perform poorly in long-tail scenarios (such as "19th-century steampunk interior design") or queries requiring multi-step reasoning (such as "find illustrations related to the theme of 'environmental protection' and suitable for children's education"). The root cause is that existing model architectures do not fully integrate the semantic generation and reasoning capabilities of Large Language Models (LLMs), relying only on static feature interactions. Traditional methods for image feature extraction largely depend on pre-trained CNN or ViT models, without optimizing for the text understanding characteristics of LLMs, further limiting retrieval accuracy. Summary of the Invention

[0005] To address the issue of prioritizing dynamic understanding capabilities while resulting in low retrieval accuracy in existing technologies, this invention provides an image retrieval method based on multi-heterogeneous graph fusion, and also provides an image retrieval system that implements the aforementioned multi-heterogeneous graph fusion-based image retrieval method. By modeling the relationship between user queries and images using heterogeneous graphs, this method can fully explore the semantic structure of image clusters behind the relevant images, achieving more accurate and context-aware image retrieval results.

[0006] This invention provides an image retrieval method based on multi-heterogeneous graph fusion, comprising the following steps:

[0007] S1: Optimize query statement: Obtain the query information output by the user, optimize the query information, and generate an optimized query statement;

[0008] S2: Construct a heterogeneous graph: Encode the optimized query information to obtain the user query code, and construct one or more heterogeneous graphs containing different types of heterogeneous nodes based on the user query code and the types of summary information in the database.

[0009] S3: Extracting Heterogeneous Graph Subgraphs: Based on the similarity relationship between heterogeneous nodes in the summary encoding information of the heterogeneous graph and user query encoding nodes, only retain heterogeneous nodes whose semantic relevance to the user query encoding exceeds a set threshold, forming one or more heterogeneous graph subgraphs.

[0010] S4: Global importance ranking of image summary nodes: Based on the information of the heterogeneous graph subgraph, the global importance ranking of image summary nodes is generated, and semantic similarity ranking and importance score ranking are generated;

[0011] S6: Select query results: Select the set of images that meet the set conditions from the sorted results and output them to the user as the final query results.

[0012] Furthermore, after step S4 is executed and before step S6 is executed, a sorting result fusion step S5 is included: the sorting results are fused using the Reverse Ranking Fusion Algorithm (RRF) to obtain a fusion score, wherein the calculation formula of the Reverse Ranking Fusion Algorithm (RRF) is:

[0013]

[0014] Where R represents the set of all available sorted lists; rank r (d) represents the rank of image d in the sort list r; k represents the smoothing coefficient, which is used to control the degree of influence of the ranking and prevent low-ranked results from having excessively large values; RRF(d) represents the fusion score of image d in all sorting results.

[0015] In step S6, based on the fusion score, the final ranking result after RRF fusion processing is obtained, and then M images with the highest scores are selected from the final ranking result as a candidate set, where M is a preset positive integer.

[0016] Further, in step S1, the optimization process includes query cleaning, query rewriting, and query enhancement. The query cleaning is used to correct typos, filter meaningless characters, and standardize the expression format. The query rewriting is used to improve the semantic alignment between the query statement and the image summary using natural language processing technology. The query enhancement is used to optimize the expression of the query, making its expression more focused on the query target. According to claim 1, the image retrieval method based on multi-heterogeneous graph fusion is characterized in that: before step S1, a database construction step is included, wherein the database stores image encoding information and original image paths for relevance matching between user encoding information and image encoding information.

[0017] Furthermore, before step S1 is executed, a database construction step is also included. The database stores image encoding information and original image paths for correlation matching between user encoding information and image encoding information.

[0018] Furthermore, the database construction step includes the following sub-steps:

[0019] A1: Deploy and train a multimodal large language model as an image summarization generation engine;

[0020] A2: Obtain each image input by the user and generate three types of hierarchical summaries: short summaries to summarize the core content, long summaries to describe details, and tag summaries covering specific entities and abstract concepts;

[0021] A3: Encode each summary to generate image summary codes;

[0022] A4: For each type of summary, calculate the cosine similarity between all image summary codes of that type;

[0023] A5: Store the original image path, the generated summary, the image summary codes corresponding to the three types of summaries, and the cosine similarity between the summaries in the database.

[0024] Furthermore, in step S2, the method for constructing the heterogeneous graph is as follows:

[0025] S201: Encode the optimized user query to obtain its semantic vector representation;

[0026] S202: Calculate the cosine similarity between the user query code and the image summary code in the database. Construct an edge of type "user query - image summary" based on the calculated cosine similarity. The user query code and the image summary code are two heterogeneous nodes, representing user intent and image semantics, respectively. This edge of type "user query - image summary" can reflect the relationship between user intent and image content.

[0027] S203: Using a sparse adjacency list as the core data structure, it retains several image summaries with the highest similarity between each image and the user query code, and constructs long summary-user query heterogeneous graphs, short summary-user query heterogeneous graphs, and tag summary-user query heterogeneous graphs.

[0028] Furthermore, in step S3, the method for extracting heterogeneous graph subgraphs is as follows:

[0029] S301: Introduce a similarity threshold filtering mechanism, and preset a semantic similarity score threshold;

[0030] S302: Using the user query node as the seed node, remove all "user query-image summary" edges connected to it with similarity below the threshold, and only retain the image summary node that is highly related to the query semantics;

[0031] S303: Using the retained image summary nodes connected to the user query node as seed nodes, remove the "image summary-image summary" edges in the heterogeneous graph where the similarity between the user query node and other image summary nodes is lower than the threshold, forming the final heterogeneous graph subgraph, which contains the user query node and related image summary nodes.

[0032] Furthermore, in step S4, the PageRank algorithm is used to globally rank the importance of the summary nodes. The constructed heterogeneous graph has undirected edges, representing bidirectional semantic similarity. The calculation formula is as follows:

[0033]

[0034] Among them, PR(v i ) represents the PageRank score of the node, N(v i ) indicates the relationship with node v i The set of all adjacent nodes, deg(v) j ) indicates the relationship with node v j The number of connected edges, PR(v) j ) represents the PageRank score of the neighboring node Vj of node Vi, N represents the total number of nodes in the heterogeneous graph subgraph, and d represents the damping factor used to control the probability of random jumps.

[0035] The present invention also provides an image retrieval system for implementing the image retrieval method based on multi-heterogeneous graph fusion, comprising:

[0036] Query statement optimization module: Used to optimize query information and generate optimized query statements;

[0037] Heterogeneous graph construction module: used to encode the optimized query information, obtain the user query code, and construct a heterogeneous graph containing more than one type of heterogeneous nodes based on the user query code and the types of summary information in the database;

[0038] Heterogeneous graph subgraph extraction module: Based on the similarity relationship between heterogeneous nodes in the summary encoding information of the heterogeneous graph and user query encoding nodes, only retain heterogeneous nodes whose semantic relevance to the user query encoding exceeds a set threshold, forming one or more heterogeneous graph subgraphs;

[0039] The image summary node global importance ranking module is used to rank the global importance of image summary nodes based on the information of the heterogeneous graph subgraph, and generate semantic similarity ranking and importance score ranking.

[0040] The query result selection module is used to select a set of images that meet the set conditions from the sorted results as the final query result.

[0041] System visualization module: Used to obtain user input query information and display the query results to the user in a visual interface.

[0042] Furthermore, it also includes a sorting result fusion and reordering module: used to fuse the sorting results using the Reverse Ranking fusion algorithm (RRF) to obtain a fusion score, and then reorder the query results based on this fusion score. Each module is functionally independent yet highly collaborative, together constructing a closed-loop process from image data management to precise retrieval and visualization.

[0043] Compared with the prior art, the beneficial effects of the present invention are:

[0044] 1. A heterogeneous graph modeling method is introduced, which designs two types of entities (user query and image summary) and two types of edges ("user query-image summary" and "image summary-image summary"). By modeling the relationship between user query and images through heterogeneous graphs, the semantic structure of image clusters behind the query-related images can be fully explored, resulting in more accurate image retrieval results with better contextual understanding.

[0045] 2. This paper proposes a graph traversal and ranking mechanism in heterogeneous graphs, employing the PageRank algorithm to score the importance of image nodes. Combining the connections between query nodes and image nodes, as well as the semantic edges between image nodes, the global influence score of each image node in the overall graph is calculated. Finally, the image nodes are ranked according to their PageRank scores, thereby selecting key images that are closely related to the query semantics and have strong centrality in the graph structure.

[0046] 3. The Regression-Ranking (RRF) algorithm is used to fuse the multi-path ranking results. By taking the reciprocal of the rankings of images from multiple ranking lists and summing them in weights, a unified fusion score is obtained. This method requires no training parameters, is robust, and can significantly improve the robustness and diversity of the overall ranking while preserving the effectiveness of each ranking source. By effectively integrating semantic similarity ranking with structural importance ranking, the RRF algorithm helps generate a more relevant and representative set of image retrieval results.

[0047] 4. This invention generates text summaries of images through a multimodal large language model, transforming image retrieval into text summary retrieval, achieving accurate cross-modal alignment, and effectively solving the modal gap problem existing in traditional image retrieval.

[0048] 5. The system generates three types of image summaries (short summary, long summary, and tag summary) using multi-granularity summarization technology, enabling it to simultaneously meet the needs of queries with varying complexity. Short summaries provide a summary of the core content, ensuring that retrieval focuses on key elements; long summaries contain detailed descriptions, improving the matching accuracy of complex queries; and tag summaries cover keywords, enhancing adaptability to long-tail scenarios. This multi-layered semantic representation significantly improves the flexibility of retrieval.

[0049] 6. The innovative query rewriting mechanism uses a large language model to intelligently optimize user input, including query cleaning, query rewriting, and query enhancement. This effectively bridges the semantic differences between user expressions and system indexes, significantly improving query accuracy.

[0050] 7. A multi-heterogeneous graph fusion approach with multi-path recall is employed. Short summary heterogeneous graphs ensure that user queries match core elements, long summary heterogeneous graphs capture detailed semantics, and tag summary heterogeneous graphs cover precise matching of structured tags. This hybrid retrieval scheme using multi-heterogeneous graphs satisfies both "search accuracy" and "semantic generalization".

[0051] 8. The method demonstrates strong adaptability to long-tail scenarios. The dynamic tag generation technology utilizes the open-domain knowledge of large language models to automatically expand emerging and niche concepts. Attached Figure Description

[0052] To more clearly illustrate the solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0053] Figure 1 This is a flowchart of the image retrieval method based on multi-heterogeneous graph fusion according to the present invention;

[0054] Figure 2 A flowchart illustrating an embodiment of database construction methods;

[0055] Figure 3 This is a flowchart illustrating an embodiment of the image retrieval method of the present invention.

[0056] Figure 4 A schematic diagram illustrating the method for constructing heterogeneous graphs and extracting subgraphs from heterogeneous graphs. Detailed Implementation

[0057] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings are used to distinguish different objects, not to describe a particular order.

[0058] In this invention, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment to other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this invention can be combined with other embodiments.

[0059] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0060] like Figure 1 As shown, the image retrieval method based on multi-heterogeneous graph fusion of the present invention includes the following steps:

[0061] S1: Optimize query statement: Obtain the query information output by the user, optimize the query information, and generate an optimized query statement;

[0062] S2: Construct a heterogeneous graph: Encode the optimized query information to obtain the user query code, and construct one or more heterogeneous graphs containing different types of heterogeneous nodes based on the user query code and the types of summary information in the database.

[0063] S3: Extracting Heterogeneous Graph Subgraphs: Based on the similarity relationship between heterogeneous nodes in the summary encoding information of the heterogeneous graph and user query encoding nodes, only retain heterogeneous nodes whose semantic relevance to the user query encoding exceeds a set threshold, forming one or more heterogeneous graph subgraphs.

[0064] S4: Global importance ranking of image summary nodes: Based on the information of the heterogeneous graph subgraph, the global importance ranking of image summary nodes is generated, and semantic similarity ranking and importance score ranking are generated;

[0065] S6: Select query results: Select the set of images that meet the set conditions from the sorted results and output them to the user as the final query results.

[0066] Preferably, after step S4 is executed and before step S6 is executed, a sorting result fusion step S5 is also included: the sorting results are fused using the Reverse Ranking Fusion Algorithm (RRF) to obtain a fusion score.

[0067] Before step S1 is executed, a database construction step is also included. The database stores image encoding information and original image paths for matching the correlation between user encoding information and image encoding information.

[0068] Most existing image retrieval methods are based on similarity matching between queries and images, typically relying on semantic or visual feature alignment between images and queries, such as vector retrieval based on text-image co-embedding spaces. However, these methods often overlook potential semantic associations, contextual connections, or co-occurrence relationships between images. Specifically, when a query is strongly associated with a particular image, the system cannot further utilize the potential semantic structure between that image and other images, thus limiting the richness and contextual relevance of the retrieval results. This invention introduces a heterogeneous graph modeling method, designing two types of entities (user query, image summary) and two types of edges ("user query-image summary" and "image summary-image summary"). By modeling the relationship between user queries and images using heterogeneous graphs, the semantic structure of image clusters behind query-related images can be fully explored, achieving more accurate and context-aware image retrieval results.

[0069] Furthermore, an image traversal and sorting mechanism is introduced, combining the connection edges between query nodes and image nodes as well as the semantic edges between image nodes to calculate the global influence score of each image node in the overall graph. Finally, the image nodes are sorted according to the score, thereby selecting key images that are closely related to the query semantics and have strong centrality in the graph structure, making the selected images more accurate.

[0070] like Figure 2 As shown in the preferred embodiment of the present invention, the present invention generates three types of image summaries (short summary, long summary, and tag summary) through multi-granularity summary generation technology, thereby enabling the database of the present invention to simultaneously meet the needs of queries with different levels of complexity. Of course, finer-grained division can also be achieved, or the data can be divided and stored in the form of summaries and tags.

[0071] The database construction steps described in this invention include the following sub-steps:

[0072] A1: Deploy and train multimodal large language models (such as Microsoft's Phi-4-multimodal large language model, Alibaba's Qwen2.5-VL generalized large model, etc.) as image summarization generation engines;

[0073] A2: Obtain each image input by the user. In this example, it is the image path. The large model can obtain the corresponding image information through the image path and use the prompting engineering to generate three types of hierarchical summaries: short summaries for summarizing the core content, long summaries for describing details, and tag summaries covering specific entities and abstract concepts.

[0074] A3: Encode each summary to generate image summary encoding. Encoding methods can include bge image encoding, m3e text embedding model encoding, etc.

[0075] A4: For each type of summary, calculate the cosine similarity between all image summary codes of that type;

[0076] A5: Store the original image path, the generated summary, the image summary codes corresponding to the three types of summaries, and the cosine similarity between the summaries in the database.

[0077] In addition to using cosine similarity to obtain the similarity relationship between images, this example can also use the dot product method to calculate the similarity between images, or the similarity between text concatenation and all image vectors in the database.

[0078] This invention provides short summaries that encapsulate core content, ensuring that searches focus on key elements; long summaries contain detailed descriptions, improving the matching accuracy of complex queries; and tag summaries cover keywords, enhancing adaptability to long-tail scenarios. This multi-layered semantic representation method greatly improves the flexibility of retrieval.

[0079] This invention generates text summaries for images using a multimodal large language model, transforming image retrieval into text summary retrieval. This achieves precise cross-modal alignment and effectively solves the modality gap problem inherent in traditional image retrieval. The method leverages the powerful capabilities of large language models in image-text joint understanding and open-domain generation to perform deep semantic understanding of the input image and automatically generate tag summaries that highly match the image content. The open-domain generation capability of the multimodal large language model is used to generate tag summaries and dynamically expand the tag library.

[0080] like Figure 3 As shown, the image retrieval method in this example is:

[0081] 1. Optimize query statements.

[0082] When a user enters a query, the system first invokes a multimodal large language model to intelligently rewrite the query through prompting engineering. This includes query cleaning, query rewriting, and query enhancement. Query cleaning aims to correct typos, filter meaningless characters, and standardize the expression format to ensure the query's accuracy. Query rewriting employs natural language processing techniques such as synonym replacement and concept expansion to improve the semantic alignment between the query and the image summary, achieving more accurate semantic matching. Query enhancement optimizes the expression of the query, making it more focused on the query's objective. After these processes, the system finally generates an optimized query that accurately reflects the user's search intent while fully adapting to the system's search mechanism.

[0083] The innovative query rewriting mechanism of this invention uses a large language model to intelligently optimize user input, effectively bridging the semantic differences between user expression and system index, and significantly improving query accuracy.

[0084] 2. Construct a heterogeneous graph.

[0085] like Figure 4 As shown, the heterogeneous graph construction process of the present invention mainly includes the following steps:

[0086] (1) First, the database already stores image summary information, three different types of summary codes generated by the coding model, and similarity scores between the pre-calculated image summary codes;

[0087] (2) Encode the optimized user query to obtain its semantic vector representation. Figure 3 In subgraph (a), the triangle represents the user query code. The four green dots represent the image summary code, but it should be noted that the green dots are of the same type. The user query code and the image summary code are considered as two heterogeneous nodes, representing the user intent and the image semantics, respectively.

[0088] (3) In Figure 3 In the subgraph (b), given the user query code as q and the image summary code as v, the cosine similarity sim(q,v) is calculated as shown in formula (1).

[0089] Formula (1)

[0090] The cosine similarity is sim(q,v). Based on the calculated cosine similarity, edges of the type "user query - image summary" are constructed to reflect the association between user intent and image content.

[0091] (4) To achieve efficient support for semantic association modeling between images and cross-modal retrieval, this invention uses a sparse adjacency list as the core data structure to organize the similarity relationships between image summaries and between user queries and image summaries. First, during the image storage stage, the system generates corresponding summary information for each image and converts the summary into a dense vector representation through an encoding model. Next, the cosine similarity between any two image summary vectors is calculated to measure the semantic relevance of the images. To avoid constructing a large-scale fully connected similarity matrix, the system only retains a few image summaries with the highest similarity between each image and itself (e.g., Top-k or images exceeding a set threshold), forming a sparse graph structure.

[0092] The graph structure described above is implemented as an adjacency list, where each image summary node corresponds to a unique identifier, serving as the key in the adjacency list. The key's value is a list of tuples, each recording the identifiers of other image summaries semantically similar to the given image summary, along with their corresponding cosine similarity scores. This sparse adjacency list can be maintained in memory as a hash table or dictionary, or it can be serialized into JSON, Pickle, or other formats for persistent storage, facilitating subsequent querying and loading.

[0093] In supporting cross-modal retrieval, user queries are also encoded as dense vectors, and their cosine similarity is calculated with image summary vectors to construct "image summary-image summary" type edges to filter out the most semantically relevant image summaries. The system also uses a sparse adjacency list structure to record these "user query-image summary" association edges. The key in the table is the unique identifier of the query, and the values ​​are the identifiers of the related image summaries and their similarity scores. In this way, the system can efficiently construct and maintain semantically relevant graphs in large-scale data environments, significantly reducing computational and storage overhead, while providing structured data support for downstream tasks such as graph construction, graph search, or graph neural network processing.

[0094] Through the above steps, three heterogeneous graphs containing multiple types of nodes and various relational edges are finally formed, such as... Figure 3 As shown in subgraph (c), it provides a structural foundation for subsequent graph representation learning and multimodal information fusion.

[0095] 3. Extract heterogeneous graph subgraphs.

[0096] The extraction of heterogeneous graph subgraphs in this invention mainly includes the following steps:

[0097] (1) In order to construct a more compact and highly relevant heterogeneous graph subgraph, a similarity threshold filtering mechanism is introduced, with a preset semantic similarity score threshold.

[0098] (2) Using the user query node as the seed node, remove all “user query-image summary” edges connected to it with similarity below the threshold, and retain only the image summary node that is highly related to the query semantics.

[0099] (3) Then, using the remaining image summary nodes connected to the user query nodes as seed nodes, remove the "image summary-image summary" edges that have a similarity of less than a threshold with other image summary nodes.

[0100] like Figure 3 As shown in subgraph (d), through the two-stage edge removal of the heterogeneous graph, each heterogeneous graph will eventually generate a subgraph containing user query nodes and related image summary nodes, resulting in the final "User Query - Long Summary" heterogeneous graph subgraph, "User Query - Short Summary" heterogeneous graph subgraph, and "User Query - Tag Summary" heterogeneous graph subgraph. Using the image summary nodes retained in these subgraphs, the system can accurately locate target images highly relevant to the user query semantics, achieving precise retrieval.

[0101] 4. Sort the image summary nodes by global importance.

[0102] To measure the overall influence of each summary node in the heterogeneous graph, this invention introduces the PageRank algorithm to rank the summary nodes globally by importance. In this algorithm, the edges of the constructed heterogeneous graph are undirected, representing bidirectional semantic similarity relationships.

[0103] The iterative formula for the PageRank algorithm is shown in formula (2) below.

[0104]

[0105] Where PR(v) i ) represents the PageRank score of the node, N(v i ) indicates the relationship with node v i The set of all adjacent nodes, deg(v) j ) indicates the relationship with node v j The number of connected edges, PR(v) j Let represent the PageRank score of node Vi's neighboring node Vj, N represent the total number of nodes in the graph, and d represent the damping factor used to control the probability of random jumps. Initially, the PageRank score of each node is set to... The calculation is iterative until convergence. By applying PageRank to the undirected heterogeneous graph, the relative position and connection strength of the image summary nodes in the overall graph structure can be fully considered, thus identifying key image nodes with high weights in semantic propagation.

[0106] This invention employs the Regression-Ranking Function (RRF) algorithm to fuse multi-path ranking results. A unified fusion score is obtained by taking the reciprocal of the image's ranking in multiple ranking lists and then summing the results in a weighted manner. For each image, its final score is determined by its ranking position in each ranking list; a higher ranking contributes more, while a lower ranking has a smaller impact. This method requires no training parameters, is robust, and significantly improves the robustness and diversity of the overall ranking while preserving the effectiveness of each ranking source. By effectively integrating semantic similarity ranking with structural importance ranking, the RRF algorithm helps generate a more relevant and representative set of image retrieval results.

[0107] 5. Merging of sorting results.

[0108] After obtaining the heterogeneous graph subgraphs of "user query - long summary", "user query - short summary", and "user query - tag summary", the system obtains two sorting scores in each heterogeneous graph subgraph: one is the sorting based on the semantic similarity between the user query and the summary, and the other is the sorting based on the importance of the image summary nodes calculated by the PageRank algorithm. After obtaining multiple sorting results, RRF (Reciprocal Rank Fusion) is introduced to fuse the sorting results. The core idea of ​​its fusion is to avoid over-reliance on the sorting results from a single source by introducing the inverse ranking. The calculation method of RRF is shown in the following formula (3).

[0109]

[0110] Where R represents the set of all available sorting lists (such as semantic similarity sorting, PageRank sorting); rank r (d) represents the ranking of document d in the sorting list r (starting from 1); k represents the smoothing coefficient, which is used to control the degree of influence of ranking and prevent low-ranked results from having excessively large values; RRF(d) represents the fusion score of image d in all sorting results.

[0111] This invention employs a multi-heterogeneous graph fusion approach with multi-path recall. Short-summary heterogeneous graphs ensure matching of user queries with core elements, long-summary heterogeneous graphs capture detailed semantics, and tag-summary heterogeneous graphs cover precise matching of structured tags. This hybrid retrieval scheme using multi-heterogeneous graphs satisfies both "search accuracy" and "semantic generalization."

[0112] 6. Select search results

[0113] After obtaining the final ranking list after RRF fusion processing, the system selects the top M images with the highest scores from the ranking results as a candidate set, based on a pre-set parameter M, where M is a positive integer. This step not only effectively controls the computational resource consumption of subsequent processing but also ensures that the selected images are highly representative in terms of semantic relevance and structural importance, providing more focused and high-quality image input for subsequent tasks such as content generation, summary extraction, or tag expansion.

[0114] The method of this invention demonstrates strong adaptability to long-tail scenarios. The dynamic tag generation technology utilizes the open-domain knowledge of a large language model to automatically expand emerging and niche concepts.

[0115] To implement the above image retrieval method, this invention also provides an image retrieval system based on multi-heterogeneous image fusion, which mainly includes eight modules, specifically:

[0116] 1. Database Construction Module: Integrates a multimodal large language model and encoding model, responsible for generating three types of image summaries, encoding the summaries, and calculating the pairwise cosine similarity between all image summary codes under each summary type. Then, the summaries, summary codes, cosine similarity between summary codes of the same type, and image storage paths are stored in the database in a predefined format.

[0117] 2. Query statement optimization module: Includes query cleaning, query rewriting, and query enhancement to optimize query statements and achieve semantic alignment between query statements and image summaries.

[0118] 3. Heterogeneous Graph Construction Module: This module encodes the optimized query information to obtain the user query code. Based on the user query code and the types of summary information in the database, it constructs one or more heterogeneous graphs containing different types of heterogeneous nodes.

[0119] This invention treats user query encoding and image summary encoding as two types of heterogeneous nodes. It uses cosine similarity to measure the semantic relevance between user query encoding and image summary encoding to construct edges of the type "user query-image summary". Similarly, it calculates the semantic similarity between image summaries based on cosine similarity to construct edges of the type "image summary-image summary", thereby modeling the potential semantic association between images.

[0120] 4. Heterogeneous Graph Subgraph Extraction Module: A preset semantic similarity score threshold is used. Starting with the user query node, all "user query - image summary" edges with similarity below the threshold are filtered out, retaining only image summary nodes highly relevant to the query semantics. Subsequently, using the retained image summary nodes as the center, further "image summary - image summary" edges with similarity below the threshold between them and other image summaries are removed. Ultimately, each heterogeneous graph yields a subgraph containing both the user query node and the image summary node. The target image is located using the retained image summary nodes within the subgraph.

[0121] 5. Image Summary Node Global Importance Ranking Module: This module ranks the global importance of image summary nodes based on the information of the heterogeneous graph subgraph, generating semantic similarity rankings and importance score rankings. This invention introduces the PageRank algorithm to perform global importance ranking of summary nodes, identifying key image nodes with high weight in semantic propagation.

[0122] 6. Sorting Result Fusion and Reordering Module: This module uses the Reverse Ranking Fusion Algorithm (RRF) to merge the sorting results, obtain a fusion score, and then reorder the query results based on this fusion score.

[0123] 7. Query Result Selection Module: This module selects a set of images from the sorted results that meet set criteria, serving as the final query result. In this example, based on a pre-defined parameter M, the module selects the top M images with the highest scores from the sorted results as the candidate set.

[0124] 8. System Visualization Module: This module is used to acquire user-input query information and display the query results to the user in a visual interface. In this example, Grado is used to build the visualization interface, supporting user input queries and image import operations, and demonstrating the algorithm's effectiveness through a visual interface.

[0125] The system of this invention comprises eight core functional modules that are both functionally independent and highly collaborative, jointly constructing a closed-loop process from image data management to precise retrieval and visualization. Based on a multimodal large language model and semantic encoding model, the system generates multi-perspective summaries (long, short, and tagged summaries) and performs structured storage during the image input stage. After a user initiates a search request, the query processing module cleans, rewrites, and semantically enhances the query statement to accurately express the user's intent. Subsequently, the system constructs a heterogeneous graph structure based on the semantic relationship between the optimized query and image summary encoding, further modeling the semantic interaction between images.

[0126] In heterogeneous graphs, the subgraph extraction module filters edge weights based on semantic relevance thresholds to obtain a subset of images strongly correlated with the user query. To measure the importance of summary nodes in the graph, the PageRank algorithm is introduced to obtain the global influence score of image nodes in the overall structure. Simultaneously, the system integrates semantic similarity ranking and structural importance ranking, using the RRF algorithm to construct a unified fusion list of multi-source ranking results. Based on the set parameter M, the system outputs several top-ranked candidate images as the final search results. Finally, the system's visualization module intuitively displays the rearranged search results to the user, completing a full closed loop from query to result display. This invention not only solves the problem of ignoring semantic relationships between images in traditional image retrieval but also achieves higher accuracy and scalability in image semantic retrieval capabilities through the combination of heterogeneous graphs and multimodal generative models. The entire process fully reflects the professional division of labor and close collaboration among the modules, significantly improving retrieval accuracy and user experience while ensuring system response speed, demonstrating good practicality and scalability.

[0127] Furthermore, this invention demonstrates strong adaptability to long-tail scenarios. The dynamic tag generation technology leverages the open-domain knowledge of large language models to automatically expand upon emerging and niche concepts.

[0128] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the present invention are within the protection scope of the present invention.

Claims

1. An image retrieval method based on multi-heterogeneous graph fusion, characterized in that, Includes the following steps: S1: Optimize query statement: Obtain the query information output by the user, optimize the query information, and generate an optimized query statement; S2: Construct a heterogeneous graph: Encode the optimized query information to obtain the user query code, and construct one or more heterogeneous graphs containing different types of heterogeneous nodes based on the user query code and the types of summary information in the database. S3: Extracting Heterogeneous Graph Subgraphs: Based on the similarity relationship between heterogeneous nodes in the summary encoding information of the heterogeneous graph and user query encoding nodes, only retain heterogeneous nodes whose semantic relevance to the user query encoding exceeds a set threshold, forming one or more heterogeneous graph subgraphs. S4: Global importance ranking of image summary nodes: Based on the information of the heterogeneous graph subgraph, the global importance ranking of image summary nodes is generated, and semantic similarity ranking and importance score ranking are generated; S6: Select query results: Select the set of images that meet the set conditions from the sorted results and output them to the user as the final query results.

2. The image retrieval method based on multi-heterogeneous graph fusion according to claim 1, characterized in that: After step S4 is executed and before step S6 is executed, a sorting result fusion step S5 is included: the sorting results are fused using the Reverse Ranking Fusion Algorithm (RRF) to obtain a fusion score, wherein the calculation formula of the Reverse Ranking Fusion Algorithm (RRF) is: Where R represents the set of all available sorted lists; rank r (d) represents the rank of image d in the sort list r; k represents the smoothing coefficient, used to control the degree of influence of the ranking and prevent low-ranked results from having excessively large values; RRF(d) represents the fusion score of image d among all sorted results. In step S6, based on the fusion score, the final ranking result after RRF fusion processing is obtained, and then M images with the highest scores are selected from the final ranking result as a candidate set, where M is a preset positive integer.

3. The image retrieval method based on multi-heterogeneous graph fusion according to claim 1, characterized in that: In step S1, the optimization process includes query cleaning, query rewriting, and query enhancement. Query cleaning is used to correct typos, filter meaningless characters, and standardize the expression format. Query rewriting is used to improve the semantic alignment between the query statement and the image summary using natural language processing technology. Query enhancement is used to optimize the expression of the query, making it more focused on the query target. According to claim 1, the image retrieval method based on multi-heterogeneous graph fusion is characterized in that: before step S1, a database construction step is included, wherein the database stores image encoding information and original image paths for relevance matching between user encoding information and image encoding information.

4. The image retrieval method based on multi-heterogeneous graph fusion according to any one of claims 1-3, characterized in that: Before step S1 is executed, a database construction step is also included. The database stores image encoding information and original image paths for matching the correlation between user encoding information and image encoding information.

5. The image retrieval method based on multi-heterogeneous graph fusion according to claim 4, characterized in that: The database construction steps include the following sub-steps: A1: Deploy and train a multimodal large language model as an image summarization generation engine; A2: Obtain each image input by the user and generate three types of hierarchical summaries: short summaries to summarize the core content, long summaries to describe details, and tag summaries covering specific entities and abstract concepts; A3: Encode each summary to generate image summary codes; A4: For each type of summary, calculate the cosine similarity between all image summary codes of that type; A5: Store the original image path, the generated summary, the image summary codes corresponding to the three types of summaries, and the cosine similarity between the summaries in the database.

6. The image retrieval method based on multi-heterogeneous graph fusion according to claim 5, characterized in that: In step S2, the method for constructing the heterogeneous graph is as follows: S201: Encode the optimized user query to obtain its semantic vector representation; S202: Calculate the cosine similarity between the user query code and the image summary code in the database. Construct an edge of type "user query - image summary" based on the calculated cosine similarity. The user query code and the image summary code are two heterogeneous nodes, representing user intent and image semantics, respectively. This edge of type "user query - image summary" can reflect the relationship between user intent and image content. S203: Using a sparse adjacency list as the core data structure, it retains several image summaries with the highest similarity between each image and the user query code, and constructs long summary-user query heterogeneous graphs, short summary-user query heterogeneous graphs, and tag summary-user query heterogeneous graphs.

7. The image retrieval method based on multi-heterogeneous graph fusion according to claim 6, characterized in that: In step S3, the method for extracting heterogeneous graph subgraphs is as follows: S301: Introduce a similarity threshold filtering mechanism, and preset a semantic similarity score threshold; S302: Using the user query node as the seed node, remove all "user query-image summary" edges connected to it with similarity below the threshold, and only retain image summary nodes that are highly related to the query semantics; S303: Using the retained image summary nodes connected to the user query node as seed nodes, remove the "image summary-image summary" edges in the heterogeneous graph where the similarity between the user query node and other image summary nodes is lower than the threshold, forming the final heterogeneous graph subgraph, which contains the user query node and related image summary nodes.

8. The image retrieval method based on multi-heterogeneous graph fusion according to claim 7, characterized in that: In step S4, the PageRank algorithm is used to globally rank the importance of the summary nodes. The constructed heterogeneous graph has undirected edges, representing bidirectional semantic similarity. The calculation formula is as follows: Among them, PR(v i N(v) represents the PageRank score of a node. i ) indicates the relationship with node v i The set of all adjacent nodes, deg(v) j ) indicates the relationship with node v j The number of connected edges, PR(v) j ) represents the PageRank score of the neighboring node Vj of node Vi, N represents the total number of nodes in the heterogeneous graph subgraph, and d represents the damping factor used to control the probability of random jumps.

9. An image retrieval system based on multi-heterogeneous graph fusion, used to implement the image retrieval method based on multi-heterogeneous graph fusion as described in any one of claims 1-8, characterized in that, include: Query statement optimization module: Used to optimize query information and generate optimized query statements; Heterogeneous graph construction module: used to encode the optimized query information, obtain the user query code, and construct a heterogeneous graph containing more than one type of heterogeneous nodes based on the user query code and the types of summary information in the database; Heterogeneous graph subgraph extraction module: Based on the similarity relationship between heterogeneous nodes in the summary encoding information of the heterogeneous graph and user query encoding nodes, only retain heterogeneous nodes whose semantic relevance to the user query encoding exceeds a set threshold, forming one or more heterogeneous graph subgraphs; The image summary node global importance ranking module is used to rank the global importance of image summary nodes based on the information of the heterogeneous graph subgraph, and generate semantic similarity ranking and importance score ranking. The query result selection module is used to select a set of images that meet the set conditions from the sorted results as the final query result. System visualization module: Used to obtain user input query information and display the query results to the user in a visual interface.

10. The image retrieval system based on multi-heterogeneous graph fusion according to claim 9, characterized in that: It also includes a sorting result fusion and reordering module: used to fuse the sorting results using the Reverse Ranking Fusion Algorithm (RRF) to obtain a fusion score, and then reorder the query results based on the fusion score.

Images