A cross-modal multi-precision hybrid index retrieval method and system for large models
By constructing a multi-layered hybrid index and dynamic index routing, the problems of high storage cost, high response latency, and poor retrieval adaptability in large-scale multimodal retrieval systems are solved, achieving efficient and adaptable cross-modal retrieval, and improving system response speed and the adaptability of the generative model.
Patent Information
- Application Number
- CN202611154892.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-31
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multimodal retrieval systems suffer from high storage costs, high response latency, and poor retrieval adaptability in large-scale data scenarios. In particular, cross-modal queries require a large amount of real-time similarity calculations and the fixed process cannot dynamically select the optimal execution path, resulting in low overall system response efficiency.
A multi-layered hybrid index is constructed, including a structured attribute index layer, a multi-precision vector index layer, and a cross-modal association graph index layer. Through dynamic index routing and progressive retrieval mechanisms, combined with a cost model and cumulative confidence decay conditions, the retrieval path is optimized to improve response speed and adaptability.
By using multi-level hybrid indexing and dynamic index routing, unnecessary index scanning and computational overhead are reduced, improving retrieval response speed, enriching candidate sources for cross-modal retrieval, and enhancing the adaptability of retrieval results to the generative model.
Smart Images

Figure CN123115481A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information and network technology, and in particular to a cross-modal multi-precision hybrid index retrieval method and system for large models. Background Technology
[0002] With the development of artificial intelligence and big data technologies, multimodal data such as text, images, videos, and audio are widely used in scenarios such as intelligent question answering, content retrieval, traffic monitoring, knowledge management, and model training data management. Different modalities of data differ significantly in data format, feature representation, and semantic organization. To achieve cross-modal retrieval, it is usually necessary to first map the data of different modalities to a unified semantic space, and then perform similarity retrieval based on structured indexes, keyword indexes, or vector indexes.
[0003] In existing multimodal retrieval systems, a common practice is to extract feature vectors from text, images, videos, and audio data separately, and then store these feature vectors in a vector database or an approximate nearest neighbor index. For structured attributes such as time, source, category, and quality level, structured indexes are typically used for filtering. After a user initiates a query, the system performs a search in the corresponding indexes based on the query content, and merges and sorts the results returned by different indexes to obtain multimodal retrieval results relevant to the user's query.
[0004] However, in large-scale, multimodal data scenarios, existing retrieval methods still have the following shortcomings. First, existing vector indexes typically store all data with a fixed dimension, saving the vector representation with the same precision regardless of how frequently the data is accessed. This results in cold, low-frequency data occupying a large amount of storage space, making it difficult to balance retrieval accuracy and storage cost. Especially in datasets with hundreds of millions or even larger scales, fixed-dimensional vector indexes significantly increase storage and maintenance overhead.
[0005] Secondly, existing cross-modal retrieval methods mostly rely on real-time calculation of similarity between different modalities during querying, or on retrieving data from each modal index separately and then merging the results. These methods lack pre-organization of semantic relationships between modalities at the index level, resulting in extensive real-time similarity calculations and high response latency during cross-modal queries. Furthermore, the relationships between results from different modalities are not clearly defined, making it difficult to support further expansion from one modal result to other related modal results.
[0006] Furthermore, existing retrieval processes are typically quite fixed, such as performing structured filtering first, then vector similarity retrieval, or calling multiple indexes in parallel and then merging them uniformly. Such fixed processes cannot dynamically select the optimal execution path based on query modality, query complexity, accuracy requirements, candidate set size, and index running status. When the user query type changes, fixed retrieval processes are prone to unnecessary index scanning and computational overhead, affecting the overall system response efficiency.
[0007] In the process of cross-modal association expansion, simply expanding the association results according to a fixed number of hops or fixed rules can easily lead to two types of problems: first, insufficient expansion, resulting in some relevant modal results not being recalled; second, over-expansion, causing a large number of irrelevant results to enter the candidate set, increasing the burden of subsequent ranking and filtering. Therefore, how to improve cross-modal recall capability while controlling the scope of multi-hop expansion is a problem that large-scale cross-modal retrieval systems need to solve.
[0008] As search results are increasingly used in downstream tasks such as large model generation, intelligent question answering, and summarization, these results not only need to be semantically relevant to the query but also suitable for organization and expression by generative models. Existing retrieval systems typically focus more on similarity ranking and less on the information density, diversity, contextual coherence, and adaptability to generative tasks of the search results. This results in some search results having high similarity but not necessarily contributing to the generation of high-quality responses. Summary of the Invention
[0009] This invention provides a cross-modal multi-precision hybrid index retrieval method and system for large models, in order to solve the problems of slow response speed and poor retrieval adaptability of existing multimodal retrieval technologies.
[0010] This invention is achieved through the following technical solution, providing a cross-modal multi-precision hybrid index retrieval method for large models, comprising the following steps: A multi-layered hybrid index is constructed, comprising a structured attribute index layer, a multi-precision vector index layer, and a cross-modal association graph index layer. The structured attribute index layer contains structured attribute indexes built from structured attribute information of the multimodal data. The multi-precision vector index layer contains multi-precision embedding vectors generated for the multimodal data, which are stored hierarchically according to different dimensions based on data access frequency, forming a multi-precision vector index. The cross-modal association graph index layer contains a cross-modal association graph index, with multimodal samples as nodes and cross-modal associations as edges, and edge weights based on semantics. Similarity, co-occurrence statistics, and structural consistency calculations are performed. Query requests are received, and index routes are dynamically determined based on query features. Each index route specifies at least one index and its execution order. Based on the index routes, a progressive retrieval process is performed: structured attribute filtering is performed based on the structured attribute index to obtain a first candidate set; approximate nearest neighbor coarse screening is performed on the first candidate set using low-dimensional vectors to obtain a second candidate set; the corresponding full-dimensional vectors are called or loaded on the second candidate set for fine ranking to obtain a third candidate set; multi-hop association expansion is performed on the third candidate set through a cross-modal association graph to obtain the final retrieval result.
[0011] Specifically, multi-precision embedding vectors are generated using the MRL method, supporting truncation on multiple preset dimensions while retaining representational capabilities; hierarchical storage divides data into four popularity levels—hot data, warm data, cold data, and archived data—based on the number of times the data is queried within a preset time window, corresponding to the storage of embedding vectors in the first high dimension, second high dimension, first low dimension, and second low dimension, respectively. The first high dimension is higher than the second high dimension, the second high dimension is higher than the first low dimension, and the first low dimension is higher than the second low dimension.
[0012] Specifically, during a query, the dimension of the low-dimensional vector is determined based on the precision requirement dimension according to the query type, while the target popularity dimension is determined based on the lowest popularity level of the data in the candidate set. The actual query dimension is then determined based on the precision requirement dimension and the target popularity dimension. When the highest similarity score of the coarse screening result is lower than the preset reliability threshold, a dimension enhancement mechanism is triggered, which enhances the corresponding candidate subset to a higher dimension for re-retrieval.
[0013] Specifically, in the cross-modal association graph index, the formula for calculating edge weights is:
[0014] Among them, w ij Let be the edge weight of the cross-modal association edge between node i and node j. Let represent the semantic similarity between the embedding vectors corresponding to node i and node j, and let cooccur(i, j) represent the frequency of co-occurrence of node i and node j in the historical search log. The structure consistency measure of the local nearest neighbor set of node i and node j in their respective modalities is α, β, and γ, which are weight coefficients and α+β+γ=1. Only edges with weights higher than the preset association threshold are retained to keep the cross-modal association graph sparse.
[0015] Specifically, multi-hop association expansion uses a stopping condition based on cumulative confidence decay:
[0016] in, Let h be the cumulative confidence score for reaching node j in the h-th hop. w represents the cumulative confidence score for reaching node i in the (h-1)th hop. ij Let be the edge weight of the cross-modal association edge between node i and node j, and η be the preset hop decay factor. The initial confidence level for query node q; The expansion will stop when any of the following conditions are met: The cumulative confidence of all candidate nodes in the current jump is lower than the first stopping threshold; The sum of the cumulative confidence scores of the newly expanded nodes is lower than the preset proportional threshold of the sum of the cumulative confidence scores of the newly added nodes in the previous hop; The number of jumps has reached the preset maximum number of jumps limit.
[0017] Specifically, dynamically determining index routes further includes: The query request is decomposed into multiple subqueries, and a directed acyclic graph is constructed based on the dependencies between the subqueries. In the directed acyclic graph, the nodes are subqueries, and the edges represent the dependencies between the subqueries. The cost model for each candidate index is calculated, taking into account the cost of index data reading, processor computing cost, memory usage cost, and network transmission cost. With the goal of minimizing the total execution cost, the optimal index allocation and execution order for each subquery are determined under the constraints of the dependencies of the directed acyclic graph and the concurrency limit of a single index.
[0018] Specifically, the dynamic determination of index routing takes into account the following factors: query modality type, query complexity, user-specified precision requirements, and the actual response time and hit rate statistics of each index in historical queries; text type queries are given priority in allocating structured attribute indexes and multi-precision vector indexes, image type queries are given priority in allocating image modality vector indexes in multi-precision vector indexes, and queries with high precision requirements are forced to use full-dimensional vectors in multi-precision vector indexes.
[0019] Specifically, the method also includes a retrieval and generation co-optimization step: A suitability score is calculated for the final search results. The suitability score is a combination of the search relevance score, the result diversity score, and the model suitability score. The search results are reordered based on the generated suitability score, and the reordered results are used as the input context for the generative model.
[0020] Specifically, the retrieval and generation co-optimization steps also include: obtaining the intermediate representation output by the generation model during the generation process, extracting key information from the intermediate representation as query expansion signals; re-inputting the expanded query into the progressive retrieval process, performing full-dimensional vector fine ranking or cross-modal association expansion to supplement the supporting information required for generation.
[0021] A cross-modal, multi-precision hybrid index retrieval system for large models includes: The hybrid index building module is used to construct multi-layer hybrid indexes, which include a structured attribute index layer, a multi-precision vector index layer, and a cross-modal association graph index layer. The structured attribute index layer contains structured attribute indexes built based on structured attribute information of multimodal data. The multi-precision vector index layer contains multi-precision embedding vectors generated for multimodal data. The multi-precision embedding vectors are stored hierarchically according to different dimensions based on data access popularity to form a multi-precision vector index. The cross-modal association graph index layer contains a cross-modal association graph index. The cross-modal association graph index uses multimodal samples as nodes and cross-modal associations as edges. The edge weights are calculated based on semantic similarity, co-occurrence statistics, and structural consistency. The dynamic routing module is used to receive query requests and dynamically determine index routes based on query characteristics. The index route specifies at least one index and the execution order. The progressive retrieval module is used to perform structured attribute filtering based on index routing to obtain a first candidate set, use low-dimensional vectors to perform approximate nearest neighbor coarse screening on the first candidate set to obtain a second candidate set, call or load the corresponding full-dimensional vectors on the second candidate set to perform fine ranking to obtain a third candidate set, and use cross-modal association graphs to perform multi-hop association expansion on the third candidate set to obtain the final retrieval result.
[0022] Compared with the prior art, the present invention has the following beneficial effects: (1) The candidate set is filtered layer by layer from coarse to fine through a four-layer progressive retrieval mechanism, and the execution path is adaptively selected for different queries by combining dynamic index routing based on cost model, which reduces unnecessary index scanning and computation overhead and improves retrieval response speed. (2) By pre-computing and storing the semantic associations between modalities through the cross-modal association graph, and using the cumulative confidence decay as the stopping condition for multi-hop expansion, the association results of other modalities are supplemented while controlling the expansion range, providing richer candidate sources for cross-modal retrieval; (3) By integrating multidimensional scores of retrieval relevance, result diversity and generative model adaptability, the retrieval results are re-ranked and the query is expanded by combining the intermediate representation of the generative model, thereby improving the adaptability of the retrieval results to the generative model. Attached Figure Description
[0023] Figure 1 The flowchart shows a cross-modal multi-precision hybrid index retrieval method for large models according to the present invention. Figure 2 This is a block diagram illustrating the index architecture principle of a cross-modal multi-precision hybrid index retrieval method for large models according to the present invention. Figure 3 This is a retrieval timing diagram of a cross-modal multi-precision hybrid index retrieval method for large models according to the present invention; Figure 4 This is a flowchart illustrating the dynamic index routing process of a cross-modal multi-precision hybrid index retrieval method for large models according to the present invention. Detailed Implementation
[0024] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0025] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature; in the description of this application, unless otherwise stated, "multiple" means two or more.
[0026] Please see Figures 1-4 This embodiment provides a cross-modal, multi-precision hybrid index retrieval system for large models, including a hybrid index construction module, a dynamic routing module, a progressive retrieval module, and a collaborative optimization generation module. The functions and structures of each module are as follows: The hybrid index building module is used to construct multi-layer hybrid indexes, which include a structured attribute index layer, a multi-precision vector index layer, and a cross-modal association graph index layer. The structured attribute index layer contains structured attribute indexes built based on structured attribute information of multimodal data. The multi-precision vector index layer contains multi-precision embedding vectors generated for multimodal data. The multi-precision embedding vectors are stored hierarchically according to different dimensions based on data access popularity to form a multi-precision vector index. The cross-modal association graph index layer contains a cross-modal association graph (CMAG) index. The cross-modal association graph index uses multimodal samples as nodes and cross-modal associations as edges. The edge weights are calculated based on semantic similarity, co-occurrence statistics, and structural consistency.
[0027] The dynamic routing module receives query requests and dynamically determines index routes based on query characteristics. The index route specifies at least one index and its execution order.
[0028] The progressive retrieval module is used to perform structured attribute filtering based on index routing to obtain a first candidate set, use low-dimensional vectors to perform approximate nearest neighbor (ANN) coarse screening on the first candidate set to obtain a second candidate set, call or load the corresponding full-dimensional vectors on the second candidate set for fine ranking to obtain a third candidate set, and use cross-modal association graphs to perform multi-hop association expansion on the third candidate set to obtain the final retrieval result.
[0029] The collaborative optimization module is used to reorder the final search results using the generated suitability score, expand the original query based on the intermediate representation output by the generative model and trigger the progressive retrieval module to re-retrieve, and synthesize virtual query and document pairs using the generative model, which are then used as training positive samples to expand the index coverage after being verified by the retrieval system.
[0030] The modules mentioned above work together to complete the entire process from index building and routing decisions to progressive retrieval.
[0031] This embodiment also provides a cross-modal multi-precision hybrid index retrieval method for large models. This embodiment uses the example of "finding historical images, text descriptions, and audio recordings similar to traffic congestion in a certain video" to illustrate the implementation process of the method. The historical database contains approximately 100 million multimodal records, including 30 million text news articles, 40 million images, 20 million videos, and 10 million audio recordings. For example... Figure 1 As shown, this method includes the following steps: S1: Construct a multi-layered hybrid index, which includes a structured attribute index layer, a multi-precision vector index layer, and a cross-modal association graph index layer. The structured attribute index layer contains a structured attribute index built based on the structured attribute information of the multimodal data. The multi-precision vector index layer contains multi-precision embedding vectors generated for the multimodal data. The multi-precision embedding vectors are stored hierarchically according to different dimensions based on the data access popularity to form a multi-precision vector index. The cross-modal association graph index layer contains a cross-modal association graph index. The cross-modal association graph index uses multimodal samples as nodes and cross-modal associations as edges. The edge weights are calculated based on semantic similarity, co-occurrence statistics, and structural consistency.
[0032] In this embodiment, as Figure 2 As shown, the hybrid index building module executes S1 to build a three-level index structure.
[0033] First layer: Construction of structured attribute index For each data record in the multimodal database, the system organizes its structured attribute information, including modality type (text / image / video / audio), creation timestamp, data source identifier, content category tag, quality rating, etc., using a B+ tree or Log-Structured Merge-Tree (LSM tree). The LSM tree is a write-optimized key-value storage structure that supports efficient range queries and equality queries. This index forms the basis for the first layer of structured attribute filtering.
[0034] The second layer involves the construction of multi-precision vector indexes. This embodiment employs the Matryoshka Representation Learning (MRL) method to generate embedding vectors for each piece of multimodal data. During training, this method calculates the contrastive learning loss for each dimension d in the truncated dimension set D = {128, 256, 512, 1024}, and then sums these losses using a weighted average to obtain the total loss.
[0035] in, For the weights of the losses in each dimension, this embodiment uses an equal-weight configuration. , This represents the InfoNCE loss calculated on the d-dimensional truncation vector, where q is the query sample. Let N be the positive samples and N be the set of negative samples. The InfoNCE loss function maximizes the similarity of positive sample pairs and minimizes the similarity of negative sample pairs through contrastive learning. Through the above training, a single embedding vector can maintain strong semantic representation ability even after being truncated in multiple preset dimensions. For example, a 1024-dimensional full-dimensional vector, after taking its first 128, 256, or 512 dimensions and performing L2 normalization, can still effectively express the semantic information of the original sample.
[0036] This invention extends the above-mentioned MRL method from textual modality to multimodal retrieval scenarios. It generates a full-dimensional vector (1024 dimensions in this embodiment) for each sample in a unified semantic space, and obtains sub-vectors of different dimensions by truncating the first d dimensions and performing L2 normalization, so that the same vector can be used at four precision levels: 128 dimensions, 256 dimensions, 512 dimensions, and 1024 dimensions.
[0037] Furthermore, this embodiment designs an adaptive dimension selection rule based on data access popularity. The system divides the data into four popularity levels and stores them hierarchically according to the number of times the data is queried within a preset time window: Hot data (queried ≥10 times within 7 days): Stores the highest-dimensional (1024-dimensional) full-precision vectors, constructing a hierarchical navigable small world (HNSW) index that resides in memory. HNSW is an efficient ANN search algorithm based on graph structure to organize high-dimensional vectors.
[0038] Warm data (queried ≥3 times within 30 days but not meeting the hot data standard): Stored as a precision vector in the second highest dimension (512 dimensions), with the HNSW graph index residing in memory.
[0039] Cold data (data other than hot and warm data): Stores the first low-dimensional (256-dimensional) low-precision vector, and the HNSW graph index is stored in the disk cache.
[0040] Archived data (not accessed within 90 days): Stores the second lowest dimension (128 dimensions) of the lowest precision vector, retaining only the structured index information, and the vector is loaded on demand.
[0041] The first high-dimensional dimension (1024 dimensions) is higher than the second high-dimensional dimension (512 dimensions), the second high-dimensional dimension is higher than the first low-dimensional dimension (256 dimensions), and the first low-dimensional dimension is higher than the second low-dimensional dimension (128 dimensions). This hierarchical storage strategy enables the storage cost of the vector index to adaptively match the data access frequency, and can significantly reduce storage space usage compared to fixed full-precision storage in scenarios with a high proportion of cold data.
[0042] The third layer involves the construction of a cross-modal association graph index. A cross-modal association graph (CMAG) is constructed on top of the HNSW vector index. The CMAG is a graph structure with multimodal samples as nodes and semantic associations between modalities as edges, and its structure is as follows:
[0043] Where V is the set of all multimodal sample nodes, Let W be the set of cross-modal edges (connecting only nodes belonging to different modalities), and let W be the edge weight matrix.
[0044] For any two nodes i and j belonging to different modalities, the weight of their cross-modal association edge is calculated using a multi-layer evidence fusion method:
[0045] The meanings and calculation methods of each factor are as follows: (1) The semantic similarity factor is the cosine similarity between the full-dimensional embedding vectors (1024 dimensions) of node i and node j.
[0046] (2) cooccur(i, j) is the co-occurrence statistical factor, which is calculated based on the system's historical retrieval logs to determine the frequency of two nodes appearing together in the same query result. The calculation formula is as follows:
[0047] Where R(q) is the set of search results for query q, and ε is the Laplace smoothing term with ε=1. This factor reflects the statistical regularity of the common demand for the two samples in actual user queries.
[0048] (3) The structural consistency factor measures the degree of local structural similarity between two nodes in their respective modalities within the HNSW graph. The calculation formula is as follows:
[0049] in, Let k be the set of k nearest neighbors of node i in the HNSW graph within its respective modality. In this embodiment, k=20. This factor reflects the structural similarity between two nodes in their respective modal spaces.
[0050] The fusion weight coefficients of the above three factors are α=0.6, β=0.25, and γ=0.15, satisfying α+β+γ=1. These values are determined by parameter search on the validation set.
[0051] After edge construction is complete, an edge pruning strategy is executed: only cross-modal edges with weights higher than a preset association threshold are retained. The remaining edges are deleted to maintain the sparsity of the cross-modal association graph, reducing storage and computational overhead. By pre-computing and storing the semantic associations between modalities, real-time similarity calculation can be avoided during subsequent cross-modal queries, significantly reducing response latency.
[0052] S2: Receive query requests and dynamically determine index routes based on query characteristics. Index routes specify at least one index and execution order.
[0053] In this embodiment, S2 is executed by the dynamic routing module. The user uploads a 30-second road monitoring video (containing video footage and ambient audio) to the system, and retrieves: (1) historical traffic images with similar content; (2) text records describing congested road conditions; and (3) audio clips containing features such as car horns and braking sounds.
[0054] like Figure 4 As shown, after receiving the query request, the dynamic routing module performs the following operations: S21: Query decomposition and construction of directed acyclic graph The composite query is decomposed into independently executable subqueries, which is then decomposed into a Directed Acyclic Graph (DAG). A DAG is a graph structure consisting of nodes and directed edges that does not contain cycles.
[0055] in, Subquery Depends on The execution results (such as structured filtering followed by vector retrieval).
[0056] In this embodiment, the query request is decomposed into independently executable subqueries q1 (image retrieval subquery), q2 (text retrieval subquery), and q3 (audio retrieval subquery), and a DAG, G is constructed. exec =({q1, q2, q3}, {dep ij}), where edge dep ij Indicates subquery q j Depends on q i The execution result is as follows. In this embodiment, the three subqueries are independent of each other and have no dependency relationship, so they can be executed in parallel.
[0057] S22: Routing Cost Model Calculation For each subquery q i and candidate index I k Calculate the routing cost using the combination of:
[0058] in, :index Number of pages to be read (I / O amount); In the index Execution The estimated CPU computation time; :index Memory usage; :index Network transmission overhead of the node (in a distributed deployment scenario); Weighting coefficients for each cost item (determined by fitting historical query logs). S23: Optimal Routing Decision With the goal of minimizing the total execution cost, the optimal index allocation and execution order of each subquery are determined under the constraints of DAG dependency relationship and the upper limit of the number of concurrent queries per index.
[0059] The decision-making process considers the following factors: the modality of the query (this query input is video, requiring cross-modal retrieval of images, text, and audio); query complexity (cross-modal semantic similarity retrieval, a complex query, prioritizing the use of a combination of vector indexes and relational graph indexes); user precision requirements (normal precision, balancing speed and precision, not mandatory full-dimensional); and the actual response time and hit rate statistics of each index in historical queries. For text subqueries, a combination of structured attribute indexes and text multi-precision vector indexes is prioritized; for image subqueries, multi-precision vector indexes corresponding to the image are prioritized; for audio subqueries, multi-precision vector indexes corresponding to the audio are prioritized.
[0060] The final index route is determined as follows: first layer structured attribute filtering → second layer 512-dimensional MRL vector coarse screening → third layer full-dimensional (1024-dimensional) vector fine sorting → fourth layer cross-modal association graph multi-hop expansion.
[0061] S3: Based on index routing, perform progressive retrieval: Filter structured attributes based on the structured attribute index to obtain the first candidate set; perform approximate nearest neighbor coarse screening on the first candidate set using low-dimensional vectors to obtain the second candidate set; call or load the corresponding full-dimensional vectors on the second candidate set for fine ranking to obtain the third candidate set; perform multi-hop association expansion on the third candidate set through cross-modal association graphs to obtain the final retrieval result.
[0062] In this embodiment, S3 is executed by the progressive retrieval module. For example... Figure 3 As shown, progressive retrieval employs a four-layer filtering process from coarse to fine. Each layer performs a more refined retrieval operation on the candidate subset selected in the previous layer, progressively reducing the size of the candidate set: The first layer is structured attribute filtering, which uses a structured attribute index managed by a B+ tree or LSM tree to quickly coarsely screen structured metadata such as modality type, time range, data source, quality level, and content category. The time budget is controlled within 10 milliseconds, and the candidate set size starts from 10. 8 The magnitude was reduced to 10 6 Magnitude.
[0063] The second layer is the MRL vector coarse screening. On the candidate set after structured filtering, a low-dimensional vector (256-dimensional or 512-dimensional, determined by the dimension selection rule) is used to perform an approximate nearest neighbor search on the HNSW graph index, with a time budget controlled within 50 milliseconds. The candidate set size starts from 10. 6 The magnitude was reduced to 10 4 Magnitude.
[0064] The third layer is full-dimensional vector fine-grained ranking. It calls or loads the corresponding full-dimensional vectors (1024 dimensions) from the second-layer candidate set for fine-grained similarity calculation and ranking, with a time budget controlled within 100 milliseconds. The candidate set size starts from 10... 4 The magnitude was reduced to 10 3 Magnitude.
[0065] The fourth layer is cross-modal association expansion. Based on the Top-K results obtained from the third layer's fine ranking, multi-hop association expansion is performed through the cross-modal association graph CMAG to supplement relevant results from other modalities. The expansion range is controlled by the cumulative confidence decay stopping condition, and the time budget is controlled within 50 milliseconds. Finally, the multimodal fusion Top-K retrieval results are output.
[0066] The total response time for the four-layer progressive retrieval is controlled within 210 milliseconds. The specific execution process of this embodiment is described layer by layer below.
[0067] First layer: Structured attribute filtering.
[0068] Based on the structured attribute index (B+ tree) in the structured attribute index layer, the following filtering conditions (Table 1) are applied: Table 1 Filtering conditions
[0069] The filtering time was controlled within 10 milliseconds, and approximately 1 million candidate records meeting the criteria were selected from 100 million full records. Modality filtering was the main reduction factor, and the results included approximately 400,000 images, approximately 350,000 text records, and approximately 250,000 audio records, forming the first candidate set.
[0070] Second layer: MRL vector coarse screening.
[0071] The dimension of the low-dimensional vector during a query is determined by the query type and the required precision dimension. At the same time, the target popularity dimension is determined based on the lowest popularity level of the candidate data. The actual query dimensions are determined based on the accuracy requirements and target popularity dimensions, using the following formula:
[0072] Among them, the accuracy requirement dimension Determined by query type: When performing an exact search (the user requires high-precision results) =1024, during rapid pre-screening (coarse-grained recall) =256, during regular retrieval (balancing accuracy and speed) =512. Target popularity dimension The popularity level is determined by the lowest popularity level of the data in the candidate set: if the candidate set consists entirely of hot data. =1024, if the candidate set contains temperature data then =512, if the candidate set contains cold data then =256.
[0073] This example is a conventional search. =512), and the candidate set contains temperature data ( =512), therefore the actual query dimension =min(512, 512)=512 dimensions.
[0074] The system inputs the user-uploaded video into the video encoder to extract a 1024-dimensional full-dimensional vector, truncates the first 512 dimensions and performs L2 normalization to obtain the query vector, and performs ANN search on the 512-dimensional HNSW graph index of the three candidate subsets: images (about 400,000), text (about 350,000), and audio (about 250,000), respectively, and retrieves the top-500 results for each.
[0075] Monitoring confidence during the coarse screening process: When the highest similarity score of the coarse screening results falls below the preset confidence threshold... When the similarity score is 0.75, the dimension upscaling mechanism is triggered, and the corresponding candidate subset is upscaled to a higher dimension for re-searching. In this embodiment, the image subset has a coarse-screened Top-1 similarity score of 0.82 (above 0.75), so no dimension upscaling is needed; the text subset has a coarse-screened Top-1 similarity score of 0.68 (below 0.75), triggering the dimension upscaling mechanism, and the text candidate subset is upscaled to 1024 dimensions for supplementary retrieval; the audio subset has a coarse-screened Top-1 similarity score of 0.71 (below 0.75), also triggering the dimension upscaling mechanism. Approximately 1500 candidates are obtained (500 images, 500 texts, and 500 audios), forming the second candidate set. The time is controlled within 50 milliseconds, and the candidate set size starts from 10... 6 The magnitude was reduced to 103 Scale. The dimensionality enhancement mechanism ensures search accuracy while avoiding the computational overhead of full-scale high-dimensional searches.
[0076] The third layer: full-dimensional vector fine sorting.
[0077] The full-dimensional vectors (1024 dimensions) of the aforementioned 1500 candidate records are called or loaded, and cosine similarity is calculated and sorted with the full-dimensional vector of the query video. The Top-100 records for each modality are retained: 100 images (similarity range 0.72~0.91), 100 text records (similarity range 0.65~0.84), and 100 audio records (similarity range 0.61~0.79), forming a third candidate set of 300 records, with the time controlled within 100 milliseconds.
[0078] Fourth layer: Multi-hop expansion of the cross-modal association graph. Starting with the top-ranked result from the third-layer ranking (query node), multi-hop expansion is performed on the cross-modal association graph (CMAG). The expansion process employs a cumulative confidence decay mechanism; the cumulative confidence of reaching node j in the h-th hop is calculated using the following formula:
[0079] in, Let h be the cumulative confidence score for reaching node j in the h-th hop. w represents the cumulative confidence score for reaching node i in the (h-1)th hop. ij Let be the edge weight of the cross-modal association edge between node i and node j, and η be the preset hop decay factor, taken as η=0.85. Query the initial confidence of the node. =1.0.
[0080] The extension terminates when any of the following stopping conditions are met: (1) Confidence threshold stopping condition: The cumulative confidence of all candidate nodes in the current hop is lower than the first stopping threshold θ. stop =0.2, meaning the process stops when the cumulative confidence of all candidate nodes for a given hop is below 0.2; (2) Marginal gain stopping condition: That is, the process stops when the total confidence of the newly expanded nodes is less than 10% of that of the previous hop. This represents the sum of the cumulative confidence scores of the newly expanded nodes. The sum of cumulative confidence scores for the newly added nodes in the previous hop; (3) Maximum jump count stopping condition: The current jump count h reaches the preset maximum jump count limit H. max =3.
[0081] In this embodiment, the first hop (video → image) finds 12 high-weight associated edges with a weight greater than 0.65, and adds 9 image records. The highest cumulative confidence score after the first hop is... Lowest cumulative confidence level The minimum value is higher than 0.2, so condition (1) is not triggered and the expansion continues.
[0082] The second hop (image → text) uses the image node associated with the first hop as a springboard, finds 8 related edges, and adds 6 new text records. The highest cumulative confidence score after the second hop is... Lowest cumulative confidence level The lowest cumulative confidence level of 0.065 is lower than the first stopping threshold of 0.2, triggering condition (1).
[0083] After the second hop, the marginal gain is checked simultaneously: the cumulative confidence of the newly extended text nodes is 0.89, and the cumulative confidence of the newly added image nodes in the previous hop is 1.47. The gain ratio is 0.89 / 1.47≈60.5%, which is higher than 10%, so condition (2) is not triggered. The current hop number h=2 has not reached Hmax=3, so condition (3) is not triggered.
[0084] Since condition (1) has been triggered, the expansion process is terminated. 15 new cross-modal association records (9 images + 6 texts) are added, and the final candidate set is expanded from 300 to 315 (109 images, 106 texts, and 100 audios), with the time controlled within 50 milliseconds.
[0085] The overall response time for the progressive retrieval process is kept within 200 milliseconds. The cross-modal association graph avoids the overhead of real-time similarity calculation during cross-modal queries by pre-computing semantic associations between modalities, significantly reducing the response latency of cross-modal retrieval; multi-hop inference expands the retrieval scope through association propagation, improving the recall rate of cross-modal retrieval.
[0086] S4: Generate collaborative optimization steps, including calculating and re-ranking the fitness scores of the final retrieval results, expanding the query using intermediate representations of the generative model, and mining virtual positive samples.
[0087] In one implementation, the system further includes a collaborative optimization module. When the system interfaces with a downstream generative model (such as a large language model), it employs a Retrieval-Augmented Generation (RAG) framework, using retrieval results as the input context for the generative model to improve the accuracy of generated facts. This invention further improves the unidirectional information flow between retrieval and generation into a bidirectional enhanced closed-loop mechanism, performing the following collaborative optimization operations.
[0088] S41: Generate quality feedback-driven retrieval reordering For each candidate ri in the final search results obtained from S3, a suitability score is calculated and generated. :
[0089] in, To obtain the relevance score, the normalized cosine similarity calculated in the third-level full-dimensional vector fine ranking stage is directly used; To score the diversity of results, the algorithm is calculated based on the Maximum Marginal Relevance (MMR) algorithm. MMR is a re-ranking algorithm that takes into account both relevance and diversity, and is used to avoid returning results with overly similar content. To generate a model fit score, information density and fluency are considered for text results, image clarity and content completeness are considered for image results, and signal-to-noise ratio and sound clarity are considered for audio results. Weighting coefficients λ1, λ2, and λ3 are preset parameters that satisfy λ1+λ2+λ3=1. In this embodiment, λ1=0.5, λ2=0.3, and λ3=0.2.
[0090] In this embodiment, the 315 candidate results are reordered based on the aforementioned suitability score:
[0091] The reordered Top-20 multimodal fusion results were used as the input context for the generative model. The final results included: 5 images (photos of congested traffic from different angles), 8 text descriptions (traffic accident reports, traffic broadcasts, and social media records), and 7 audio clips (including car horns, braking sounds, and ambient sounds of traffic congestion). Cross-modal relationships were established between the results through CMAG association edges.
[0092] S42: Retrieval-oriented query expansion During the content generation process of the generative model based on the retrieval results, the intermediate representation output by the generative model during the generation process is obtained, and key information is extracted from the intermediate representation as query expansion signals. Specifically, the expanded query is constructed using the following formula:
[0093] in, For the original query, The ExtractKeywords function extracts key information from the intermediate representations output by the generative model during the generation process (e.g., generated summary text, extracted keyword list, decoder intermediate layer hidden states, etc.). Conca represents concatenating the original query with the extracted key information to form an expanded query. .
[0094] The expanded query is then re-entered into the progressive retrieval process to perform full-dimensional vector ranking or cross-modal association expansion to supplement the generation of the necessary supporting information.
[0095] S43: Virtual Positive Sample Mining The generative model can further synthesize virtual query-document pairs (e.g., generating new query text and corresponding multimodal content descriptions based on existing data descriptions), and input these synthesized pairs into the retrieval system for reverse validation—that is, verifying whether the synthesized pairs can be effectively retrieved through a multi-precision vector index. Successfully matched synthesized pairs serve as new positive training samples to expand the training data coverage of the HNSW graph index, especially to compensate for the lack of training samples for long-tail queries and emerging concepts.
[0096] In this embodiment, each of the above modules can be implemented by a computer program, which is stored in a computer-readable storage medium and performs the functions of each of the above steps when executed by a processor.
[0097] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. This application is not limited to the exact structures described above and illustrated in the accompanying drawings, and it should not be considered that the specific implementation of this application is limited to these descriptions. For those skilled in the art, various changes and modifications made without departing from the concept of this application should be considered to fall within the protection scope of this application.
Claims
1. A cross-modal multi-precision hybrid index retrieval method for large models, characterized in that, Includes the following steps: A multi-layered hybrid index is constructed, comprising a structured attribute index layer, a multi-precision vector index layer, and a cross-modal association graph index layer. The structured attribute index layer contains a structured attribute index built based on structured attribute information of multimodal data. The multi-precision vector index layer contains multi-precision embedding vectors generated for multimodal data. These multi-precision embedding vectors are stored hierarchically according to different dimensions based on data access popularity, forming a multi-precision vector index. The cross-modal association graph index layer contains a cross-modal association graph index, which uses multimodal samples as nodes and cross-modal associations as edges. The edge weights are calculated based on semantic similarity, co-occurrence statistics, and structural consistency. Receive a query request and dynamically determine the index route based on the query characteristics. The index route specifies at least one index and its execution order. Based on the index routing, a progressive retrieval is performed: structured attribute filtering is performed based on the structured attribute index to obtain a first candidate set; a coarse screening of the first candidate set is performed using low-dimensional vectors to obtain a second candidate set; the corresponding full-dimensional vectors are called or loaded on the second candidate set for fine ranking to obtain a third candidate set; and multi-hop association expansion is performed on the third candidate set through the cross-modal association graph to obtain the final retrieval result.
2. The cross-modal multi-precision hybrid index retrieval method for large models according to claim 1, characterized in that: The multi-precision embedding vector is generated using the MRL method, which supports truncation on multiple preset dimensions while retaining representational capabilities. The hierarchical storage divides the data into four popularity levels—hot data, warm data, cold data, and archived data—based on the number of times the data is queried within a preset time window. These levels correspond to the storage of embedding vectors in the first high dimension, second high dimension, first low dimension, and second low dimension, respectively. The first high dimension is higher than the second high dimension, the second high dimension is higher than the first low dimension, and the first low dimension is higher than the second low dimension.
3. The cross-modal multi-precision hybrid index retrieval method for large models according to claim 2, characterized in that: During the query, the dimension of the low-dimensional vector is determined based on the precision requirement dimension according to the query type, and the target popularity dimension is determined based on the lowest popularity level of the data in the candidate set. The actual query dimension is determined based on the precision requirement dimension and the target popularity dimension. When the highest similarity score of the coarse screening result is lower than the preset reliability threshold, the dimension enhancement mechanism is triggered to enhance the corresponding candidate subset to a higher dimension for re-retrieval.
4. The cross-modal multi-precision hybrid index retrieval method for large models according to claim 1, characterized in that: In the cross-modal association graph index, the edge weight is calculated using the following formula: Among them, w ij Let be the edge weight of the cross-modal association edge between node i and node j. Let represent the semantic similarity between the embedding vectors corresponding to node i and node j, and let cooccur(i, j) represent the frequency of co-occurrence of node i and node j in the historical search log. The structure consistency measure of the local nearest neighbor set of node i and node j in their respective modes is α, β, and γ, which are weight coefficients and α+β+γ=1; only edges with weights higher than the preset association threshold are retained to keep the cross-modal association graph sparse.
5. The cross-modal multi-precision hybrid index retrieval method for large models according to claim 4, characterized in that: The multi-hop association extension uses a stopping condition based on cumulative confidence decay: in, Let h be the cumulative confidence score for reaching node j in the h-th hop. w represents the cumulative confidence score for reaching node i in the (h-1)th hop. ij Let be the edge weight of the cross-modal association edge between node i and node j, and η be the preset hop decay factor. The initial confidence level for query node q; The expansion will stop when any of the following conditions are met: The cumulative confidence of all candidate nodes in the current jump is lower than the first stopping threshold; The sum of the cumulative confidence scores of the newly expanded nodes is lower than the preset proportional threshold of the sum of the cumulative confidence scores of the newly added nodes in the previous hop; The number of jumps has reached the preset maximum number of jumps limit.
6. The cross-modal multi-precision hybrid index retrieval method for large models according to claim 1, characterized in that, The dynamic determination of index routes further includes: The query request is decomposed into multiple subqueries, and a directed acyclic graph is constructed based on the dependencies between the multiple subqueries, wherein the nodes in the directed acyclic graph are subqueries, and the edges represent the dependencies between the subqueries. The cost model for each candidate index is calculated, and the cost model comprehensively considers the index data reading cost, processor computing cost, memory usage cost and network transmission cost. With the goal of minimizing the total execution cost, and under the constraints of the dependencies of the directed acyclic graph and the single index concurrency limit, the optimal index allocation and execution order of each subquery are determined.
7. The cross-modal multi-precision hybrid index retrieval method for large models according to claim 6, characterized in that, The dynamic determination of index routing takes into account the following factors: query modality type, query complexity, user-specified precision requirements, and the actual response time and hit rate statistics of each index in historical queries; text type queries are given priority in allocating structured attribute indexes and multi-precision vector indexes, image type queries are given priority in allocating image modality vector indexes in the multi-precision vector indexes, and queries requiring high precision are forced to use full-dimensional vectors in the multi-precision vector indexes.
8. The cross-modal multi-precision hybrid index retrieval method for large models according to claim 1, characterized in that, The method also includes a retrieval and generation of collaborative optimization steps: A suitability score is calculated for the final search results. The suitability score integrates the search relevance score, the result diversity score, and the model suitability score. The search results are reordered based on the generated suitability score, and the reordered results are used as the input context for the generative model.
9. The cross-modal multi-precision hybrid index retrieval method for large models according to claim 8, characterized in that, The retrieval and generation collaborative optimization step also includes: Obtain the intermediate representation output by the generative model during the generation process, and extract key information from the intermediate representation as a query extension signal; The expanded query is re-entered into the progressive retrieval process to perform the full-dimensional vector ranking or the cross-modal association expansion to supplement the generation of the required supporting information.
10. A cross-modal, multi-precision hybrid index retrieval system for large models, characterized in that, include: A hybrid index construction module is used to construct a multi-layer hybrid index, which includes a structured attribute index layer, a multi-precision vector index layer, and a cross-modal association graph index layer. The structured attribute index layer contains a structured attribute index built based on the structured attribute information of multimodal data. The multi-precision vector index layer contains multi-precision embedding vectors generated for multimodal data. The multi-precision embedding vectors are stored hierarchically according to different dimensions based on data access popularity to form a multi-precision vector index. The cross-modal association graph index layer contains a cross-modal association graph index. The cross-modal association graph index uses multimodal samples as nodes and cross-modal associations as edges. The edge weights are calculated based on semantic similarity, co-occurrence statistics, and structural consistency. The dynamic routing module is used to receive query requests and dynamically determine index routes based on query characteristics. The index routes specify at least one index and the execution order. The progressive retrieval module is used to perform structured attribute filtering based on the index routing to obtain a first candidate set, perform approximate nearest neighbor coarse screening on the first candidate set using low-dimensional vectors to obtain a second candidate set, call or load the corresponding full-dimensional vectors on the second candidate set for fine ranking to obtain a third candidate set, and perform multi-hop association expansion on the third candidate set through the cross-modal association graph to obtain the final retrieval result.
