Text retrieval-oriented adaptive length embedding method and system

Abstract

The invention provides a text retrieval-oriented adaptive length embedding method and system, and the method comprises the steps: encoding an original document into a high-dimensional embedding vector by using a trained embedding model, and obtaining an original document embedding matrix X belonging to Rn * d; carrying out matrix learning transformation on the embedded vector through a transformation matrix fitting module to obtain a transformed embedded vector; inputting the converted embedded vector into a hybrid coding module for hybrid coding, dividing the converted embedded vector of each document into a fixed-length dense part and a variable-length sparse part, dynamically adjusting the length of the sparse part according to the semantic complexity of the document, and then performing similarity calculation by combining the dense part and the sparse part to obtain a similarity value; and thus, self-adaptive text retrieval is realized. According to the method, the resource utilization efficiency of the system is remarkably improved, and the retrieval accuracy and robustness are also ensured. And the method is particularly suitable for a large-scale retrieval system and an application environment with strict requirements on storage and computing resources.

Description

Technical Field

[0001] The present invention relates to the field of text retrieval technology, and in particular to a text retrieval-oriented adaptive length embedding method and system. Background Art

[0002] Text retrieval refers to the process of searching for and retrieving relevant documents within large text collections for a specific query. This technology not only supports a wide range of applications, including search engines, recommender systems, and question-answering systems, but also plays a key role in information management, log retrieval, document recommendation, and enterprise knowledge bases. The core goal of a text retrieval system is to efficiently and accurately select the most relevant documents from a vast text collection based on the diverse needs of users. In recent years, with the rise of large language models (LLMs), text retrieval has become a key component of retrieval-augmented generation (RAG) systems, supporting complex capabilities such as intelligent question answering and content generation. The overall performance of text retrieval technology depends heavily on how text data is represented and the criteria used to measure the relevance between queries and documents. These representations and relevance determination mechanisms determine whether the system can accurately understand query intent, identify semantic associations, and maintain good performance in a variety of application scenarios.

[0003] Among numerous text retrieval methods, dense retrieval has become the mainstream technology in the current text retrieval field due to its ability to effectively capture high-level semantic features of text. Compared to traditional sparse feature methods (such as keyword-based TF-IDF and BM25), dense retrieval methods utilize neural network models to encode text content into dense vector representations, offering significant advantages in semantic understanding and relevance capture. The performance of dense embedding models is closely related to the model's parameter scale and the dimensionality of the vector. Similar to the scaling characteristics of large language models, for embedding models, increasing the dimensionality of the embedding vector often enhances the model's ability to express complex semantic relationships, thereby improving retrieval accuracy. For documents with complex semantic structures and rich conceptual relationships, using high-dimensional vector representations can better capture the deep semantic information in the text and meet the needs of complex retrieval tasks. For example, in proprietary domains, where document content involves a large amount of specialized terminology and complex semantics, high-dimensional vectors can more meticulously characterize document features.

[0004] However, the use of high-dimensional vectors also brings significant resource consumption problems. First, the system needs to store more vector dimensions for each document, and the overall storage overhead increases significantly. Secondly, during the retrieval process, the computational complexity of commonly used approximate nearest neighbor algorithms (such as LSH, IVFPQ, etc.) during querying is often linearly related to the vector dimension d, which means that high-dimensional vectors will lead to a decrease in retrieval speed and affect the response efficiency of the system. For application scenarios that require real-time processing or run in resource-constrained environments such as edge devices and mobile terminals, the storage and computational overhead of high-dimensional vectors becomes a major obstacle to actual deployment. Currently, mainstream dense embedding models generally use fixed-length encoding methods, which cannot be dynamically adjusted according to the semantic complexity of the document. This not only causes a large amount of waste of storage and computing resources, but also limits the flexibility and efficiency of the model in diverse application environments.

[0005] Existing text retrieval methods can be broadly categorized into keyword-based models, dense retrieval models, and retrieval models based on large language models. The following summarizes the characteristics and shortcomings of these models.

[0006] (1) Keyword-based model

[0007] This type of model typically encodes each document as a sparse vector over a vocabulary, with each dimension reflecting statistical features such as term frequency. The most common methods include TF-IDF and its improved BM25 algorithm. TF-IDF assigns weights to terms based on their importance within a single document and their distribution across the entire corpus. Term frequency (TF) measures how often a term appears in a document, while inverse document frequency (IDF) downweights terms that appear frequently across many documents. Despite its relative simplicity, TF-IDF remains an effective baseline method in text retrieval. BM25 improves on TF-IDF by introducing a probabilistic ranking function and considering document length normalization. BM25 uses hyperparameters k and b to control term saturation and length normalization, respectively. Due to its high efficiency and effectiveness in ranking documents based on keyword matching, this method has been widely used in various search engines and information retrieval systems. However, keyword-based models can only achieve exact keyword matching, struggle to handle synonyms, and fail to capture semantic information at the sentence and paragraph levels. Therefore, such models perform poorly in retrieval scenarios that require deep semantic understanding.

[0008] (2) Dense retrieval model

[0009] Dense retrieval models use deep neural networks, particularly pre-trained language models such as BERT, to map queries and documents into continuous vector representations. These models overcome the limitations of traditional keyword-matching models, enabling semantic-level retrieval and uniformly mapping queries and documents into a shared dense embedding space. DPR, a pioneering work in the field of dense retrieval, proposed a dual-tower encoding structure. By pre-encoding all documents, building an index, and combining it with an approximate nearest neighbor algorithm, it enables efficient, real-time retrieval of large document collections. The ANCE model subsequently improved model performance by mining hard negative examples and enhancing contrastive learning with high-scoring negative examples returned by the approximate nearest neighbor algorithm, thereby narrowing the data distribution gap between training and inference. RocketQA introduced knowledge distillation, employing a cross-encoder as a teacher model to generate soft labels for the dual-tower architecture to improve training performance. These improved methods have been widely adopted and proven effective in training dense retrieval models. However, the vector representations generated by these models are typically fixed-dimensional and cannot adapt to the semantic complexity of the text.

[0010] (3) Retrieval model based on large language model

[0011] With the rapid development of large language models, the latest dense retrieval methods have begun to use pre-trained large language models as base models and fine-tune them for retrieval tasks, thereby leveraging the large language models' ability to understand general knowledge. Promptreps designs specific prompt words to allow large language models to summarize text using a single token, and uses hidden states and probability distributions as dense and sparse vector representations, respectively. Other methods perform supervised fine-tuning on large language models to further improve retrieval performance. For example, NV-EMBED replaces the standard attention layer with a bidirectional attention mechanism to enhance expressiveness. BGE-ICL utilizes the contextual learning capabilities of LLM to improve embedding generation by constructing examples in queries. GTE-QWEN adopts a curriculum learning strategy to fine-tune the Qwen2 model to extend the maximum context length. Although LLM-based models generally outperform traditional dense models in retrieval performance, their high-dimensional embedding representations bring challenges to storage and retrieval efficiency. Summary of the Invention

[0012] In view of the defects in the prior art, the present invention aims to provide a text retrieval-oriented adaptive length embedding method and system.

[0013] According to the present invention, an adaptive length embedding method for text retrieval is provided, comprising:

[0014] Step S1: Use the trained embedding model to encode the original document into a high-dimensional embedding vector to obtain the original document embedding matrix X∈R n×d ;

[0015] Step S2: The embedding vector is subjected to a learning transformation matrix by a transformation matrix fitting module to obtain a transformed embedding vector;

[0016] Step S3: The converted embedding vector is input into a hybrid coding module for hybrid coding. The converted embedding vector of each document is divided into a fixed-length dense part and a variable-length sparse part. The length of the sparse part is dynamically adjusted according to the semantic complexity of the document. The similarity calculation is then performed on the dense and sparse parts to achieve adaptive text retrieval.

[0017] Preferably, the learned transformation matrix makes the transformed embedding as linearly independent as possible in each dimension by fitting the original high-dimensional embedding, while maintaining the similarity between the text embeddings before and after the transformation;

[0018] The step S2 comprises:

[0019] Step S2.1: Perform k-means clustering on all high-dimensional embedding vectors;

[0020] Step S2.2: Randomly select subsets according to the proportion of the number of samples in each cluster as the data for fitting the transformation matrix P, and calculate the transformation matrix P; where P∈R d×d is the matrix to be solved;

[0021] Step S2.3: After fitting the transformation matrix, use the current matrix to transform all the initial high-dimensional embeddings to obtain the linearly independent embeddings of each dimension after the transformation. The new embedding matrix X′∈R is obtained by linear transformation X′=XP n×d , where n represents the number of documents and d represents the dimension of each embedding vector.

[0022] Preferably, the transformation X′=XP needs to satisfy two conditions:

[0023] First, the dimensions are linearly independent, that is, the covariance matrix of each column of X′ is a diagonal matrix, thus ensuring that the dimensions are orthogonal after the transformation;

[0024] Second, the transformed similarity matrix remains consistent with the original similarity matrix;

[0025] The formal constraints are as follows:

[0026]

[0027] Among them, |·| F represents the Frobenius norm, λ1,λ2,...,λ d represents the elements on the diagonal of the covariance matrix;

[0028] Then the covariance matrix of the selected samples is decomposed into eigenvalues, and the formula is as follows:

[0029] X T X=VΛV T (2)

[0030] Where V∈R d×d represents the eigenvector orthogonal matrix, Λ represents the eigenvalue diagonal matrix;

[0031] Taking P = V can simultaneously satisfy the two constraints in formula (1). The transformed matrix X′ is equivalent to performing principal component analysis (PCA) on X, and the principal components are concentrated in the first several dimensions.

[0032] Preferably, the fixed-length dense part and the variable-length sparse part are stored using a vector database and an inverted list respectively.

[0033] Preferably, the hybrid encoding module adaptively adjusts the length of its embedding vector according to the semantic complexity of each document, by selecting the minimum dimension to fully express the semantics, including:

[0034] Set a preset threshold θ, and for each transformed document embedding, first sort it by the absolute value of each dimension from large to small, and then select dimensions in order from large to small until the ratio of the sum of squares of the selected dimensions to the total sum of squares exceeds the threshold θ.

[0035] Preferably, the dense parts of all document embeddings are unified into the first k dimensions of the transformed vector, where k is a hyperparameter;

[0036] The remaining dimensions select the minimum number of dimensions as the sparse part as needed to ensure that the sum of the squares of the dense part and the sparse part exceeds the preset threshold.

[0037] Preferably, the similarity calculation in step S3 includes:

[0038] Step S3.1: For each vector x i ′, define the dense part as

[0039] Step S3.2: Calculate the similarity of the dense part as the standard dot product of the first k dimensions, as follows:

[0040]

[0041] Step S3.3: Calculate the similarity of the sparse parts. The similarity of the sparse parts is the dot product of the corresponding components in the intersection of the selected dimensions of the respective sparse parts. The calculation formula is as follows:

[0042]

[0043] Among them, S i and Sj Represents x i ′ and x j The index set of the selected dimension of the sparse part of ′ is specifically defined as:

[0044]

[0045] Step S3.4: Embed two documents x i ′ and x j The total similarity of the similarity of ′ is:

[0046] sim(x i ′,x j ′)=sim (dense) (x i ′,x j ′)+sim (sparse) (x i ′,x j ′).

[0047] According to the present invention, an adaptive length embedding system for text retrieval is provided, comprising:

[0048] Module M1: Use the trained embedding model to encode the original document into a high-dimensional embedding vector to obtain the original document embedding matrix X∈R n×d ;

[0049] Module M2: The embedding vector is subjected to a learning transformation matrix by a transformation matrix fitting module to obtain a transformed embedding vector;

[0050] Module M3: Input the converted embedding vector into the hybrid coding module for hybrid coding, divide the converted embedding vector of each document into a fixed-length dense part and a variable-length sparse part, and dynamically adjust the length of the sparse part according to the semantic complexity of the document. Then, similarity calculation is performed on the dense and sparse parts to achieve adaptive text retrieval.

[0051] Preferably, the learned transformation matrix makes the transformed embedding as linearly independent as possible in each dimension by fitting the original high-dimensional embedding, while maintaining the similarity between the text embeddings before and after the transformation;

[0052] The module M2 includes:

[0053] Module M2.1: Perform k-means clustering on all high-dimensional embedding vectors;

[0054] Module M2.2: Randomly select subsets according to the proportion of the number of samples in each cluster as data for fitting the transformation matrix P, and calculate the transformation matrix P; where P∈R d×d is the matrix to be solved;

[0055] Module M2.3: After fitting the transformation matrix, use the current matrix to transform all the initial high-dimensional embeddings to obtain linearly independent embeddings of each dimension after transformation. The new embedding matrix X′∈R is obtained by linear transformation X′=XP n×d , where n represents the number of documents and d represents the dimension of each embedding vector.

[0056] Preferably, the fixed-length dense part and the variable-length sparse part are stored using a vector database and an inverted list respectively.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. The adaptive length embedding method of the present invention solves the problem of the difficulty in balancing resource utilization and expressiveness in fixed-length dense embedding, and realizes the adaptive matching of embedding length and document semantic complexity.

[0059] 2. To address the high time and space costs of fitting a transformation matrix when the amount of text data is large, this paper proposes a clustering-based sample subset selection algorithm. By performing k-means clustering on the original embeddings and randomly selecting several samples from each cluster, the distribution of the selected samples is made closer to the original data distribution, and the number of samples required for fitting is reduced.

[0060] 3. To address the problem of misaligned dimensions between variable-length embeddings, making direct similarity calculation difficult, this paper proposes a dense + sparse hybrid encoding scheme. By leveraging the fact that the principal components of the transformed embeddings are distributed in the front dimensions, the embedding is divided into a fixed-length dense portion and a variable-length sparse portion, and an inverted index is used to accelerate retrieval of the sparse portion.

[0061] 4. The present invention realizes an efficient text encoding and retrieval framework, which can be widely used in search engines, intelligent question answering, retrieval enhancement generation and other systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0063] Figure 1 It is a schematic flow chart of the working method of the present invention. DETAILED DESCRIPTION

[0064] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0065] The present invention makes full use of the redundancy in high-dimensional embedding vectors and converts the high-dimensional vectors into a set of linearly independent components through linear transformation. For each document, the method can analyze its semantic complexity and retain only the minimum necessary dimensions sufficient to express its semantic information, thereby achieving adaptive matching of the embedding length and the semantic complexity of the document. Specifically, the present invention first uses the existing high-dimensional embedding model to obtain the original vector representation, and then maps the high-dimensional vector to a linearly independent subspace by learning the transformation matrix. On this basis, a hybrid coding strategy is adopted: the embedding vector of each document is divided into a shorter dense part and a sparse part of variable length, and the length of the sparse part is dynamically adjusted according to the actual semantic requirements of different documents. This can not only ensure that the model has sufficient expressive power when expressing complex semantics, but also greatly reduce the overall storage and retrieval costs.

[0066] In the actual retrieval stage, this method combines the information of the dense part and the sparse part, and realizes efficient and adaptive matching of queries and documents by comprehensively calculating the similarity. The core is to decorrelate the dimensions of the embedding vector and adaptively select the minimum subset of dimensions that can fully express its semantic features for each document. This adaptive length embedding strategy not only significantly improves the resource utilization efficiency of the system, but also ensures the accuracy and robustness of the retrieval. By flexibly adjusting the embedding dimensions, the adaptive length embedding method can take into account efficiency, accuracy and scalability, and is particularly suitable for large-scale retrieval systems and application environments with strict requirements on storage and computing resources. Therefore, the method provided by the present invention brings a new technical path to the field of text retrieval, which has important theoretical value and broad application prospects.

[0067] Example 1

[0068] According to the present invention, an adaptive length embedding method for text retrieval is provided, such as Figure 1 Shown, including:

[0069] Step S1: Use the trained embedding model to encode the original document into a high-dimensional embedding vector.

[0070] Step S2: The embedding vector is passed through a transformation matrix fitting module to learn a transformation matrix to obtain a transformed embedding vector. The learned transformation matrix fits the original high-dimensional embedding, making the transformed embedding as linearly independent as possible in all dimensions while maintaining the similarity between the text embeddings before and after the transformation. To improve efficiency, this method estimates the transformation matrix P by selecting s samples. Step S2 includes:

[0071] Step S2.1: Perform k-means clustering on all high-dimensional embedding vectors.

[0072] Step S2.2: Randomly select subsets based on the number of samples in each cluster as the data for fitting the transformation matrix. Through the above steps S2.1 and S2.2, even when the number of samples is limited, it can be ensured that the selected subsets are representative of the overall data distribution.

[0073] Step S2.3: After fitting the transformation matrix, use the current matrix to transform all the initial high-dimensional embeddings to obtain the transformed embeddings whose dimensions are linearly independent.

[0074] Specifically, there are n documents, and their original d-dimensional embedding matrix is ​​recorded as X∈R n × d This method calculates a transformation matrix P such that the transformed matrix X′ = XP maintains the original embedding similarity structure, leaving the similarity between any two documents unchanged, while also ensuring linear independence between the dimensions. For each transformed document embedding x′, a minimal subset of dimensions is selected such that the proportion of its squared sum to the total squared sum of x′ exceeds a preset threshold θ.

[0075] The transformation matrix fitting module aims to embed the original document into a matrix X∈R n×d , where n represents the number of documents, d represents the dimension of each embedding vector, and the new embedding matrix X′∈R is obtained by linear transformation X′=XP n×d , where P∈R d×d is the matrix to be solved.

[0076] The transformation X′=XP must satisfy two conditions. First, the dimensions must be linearly independent, that is, X′TX′ (i.e., the covariance matrix of each column of X′) is a diagonal matrix, thus ensuring that the dimensions are orthogonal after the transformation. Second, the transformed similarity matrix should be consistent with the original similarity matrix. The formal constraints are as follows:

[0077]

[0078] Among them, |·| F represents the Frobenius norm. λ1,λ2,...,λ d Represents the elements on the diagonal of the covariance matrix.

[0079] Then the covariance matrix of the selected samples is decomposed into eigenvalues, and the formula is as follows:

[0080] X T X=VΛV T (2)

[0081] Where V∈R d×dDenotes an orthogonal matrix of eigenvectors, and Λ denotes a diagonal matrix of eigenvalues. Taking P = V can simultaneously satisfy the two constraints in formula (1). The transformed matrix X′ is equivalent to performing principal component analysis (PCA) on X, with the principal components concentrated in the first several dimensions.

[0082] To reduce computational costs, this module uses a clustering-based sample selection strategy, using a selected sample subset to calculate P. To ensure that the subset is representative, a clustering-based sampling strategy is adopted. That is, the K-means algorithm is first used to cluster the dataset X into m clusters:

[0083]

[0084] Then, from each cluster C i Randomly select a subset in proportion to the number of samples. Suppose a total of s samples are selected, |C i | is the number of samples in the ith cluster, then from C i The number of samples drawn from is:

[0085]

[0086] The samples selected from each cluster are combined to form the sampling matrix X sub ∈R s×d , used to fit the transformation matrix P.

[0087] However, the above method faces two major challenges. First, directly fitting the transformation matrix using the embedding vectors of all documents will result in huge computational and storage overhead. Real-world datasets often contain millions of documents. To ensure retrieval results, the original embedding dimension is usually high. Since the transformation matrix solution complexity is O(nd 2 ), when both the number of documents n and the embedding dimension d are large, computing P will be extremely time-consuming and memory-intensive. Second, while adaptive-length embedding can retain sufficient semantic information for each document, the number of dimensions retained varies across documents, making direct similarity calculation impossible.

[0088] To solve the above problems, the present invention proposes a clustering-based sample selection strategy and a hybrid coding scheme.

[0089] Step S3: The converted embedding vector is input into a hybrid coding module for hybrid coding, and the converted embedding vector of each document is divided into a shorter dense part and a sparse part of variable length, i.e., a fixed-length dense part and a variable-length sparse part, and the length of the sparse part is dynamically adjusted according to the semantic complexity of the document. Then, similarity calculation is performed on the dense and sparse parts, thereby realizing adaptive text retrieval. After the document embedding is transformed, its main components (representing common and significant features) are concentrated in the first several dimensions, while the components that can express the individual characteristics of the document are distributed in the subsequent dimensions. Therefore, ALE divides the converted embedding vector into a fixed-length dense part and a variable-length sparse part, and adopts hybrid similarity calculation during retrieval. The fixed-length dense part and the variable-length sparse part are stored using a vector database and an inverted list respectively to improve storage and retrieval efficiency. During retrieval, the total similarity is the weighted sum of the similarity of the dense part and the similarity of the sparse part.

[0090] After the transformation matrix P is applied to the original high-dimensional embedding, the document embedding dimensions become linearly independent. The hybrid encoding module aims to adaptively adjust the length of each document's embedding vector based on its semantic complexity, selecting the minimum number of dimensions to fully express the semantics. This is achieved by setting a preset threshold θ and, for each transformed document embedding, sorting the dimensions from largest to smallest by absolute value. Dimensions are then selected sequentially from largest to smallest until the proportion of the squares of the selected dimensions to the total squares exceeds the threshold θ.

[0091] However, because the dimensions of variable-length vectors are not aligned, it is difficult to directly calculate similarity between different documents. Different documents retain different numbers of principal components due to their different semantic complexity. Moreover, the transformation matrix is ​​estimated based on a subset of the data. The front dimensions of the transformed space mainly capture the global principal components, but important components of some documents may still be distributed in subsequent dimensions, reflecting individual semantic characteristics. Therefore, the variable-length vectors of different documents are not aligned in terms of dimensions, and direct dot products cannot be performed. Although the selected dimension indices and their values ​​for each document can be stored in a sparse matrix format, the compression and reconstruction process will weaken the space and time efficiency advantages brought by variable-length coding.

[0092] To this end, the present invention proposes a hybrid encoding method. Leveraging the characteristic that the principal components of the transformed embedding are concentrated in the first few dimensions, each transformed embedding vector is divided into a dense part and a sparse part. The dense part of all document embeddings is unified as the first k dimensions of the transformed vector, where k is a hyperparameter. For the remaining dimensions, the aforementioned greedy strategy is used to select the minimum number of dimensions as the sparse part, ensuring that the sum of the squares of the dense and sparse parts exceeds a preset threshold.

[0093] Specifically, for each vector x i ′, define the dense part as The sparse part is a number of dimensions selected from the remaining dimensions according to the above strategy, forming a variable-length sparse vector. i ′ and x j The similarity of ′ consists of the following two parts:

[0094] The similarity of the dense part is the standard dot product of the first k dimensions:

[0095]

[0096] The similarity of the sparse parts is the dot product of the corresponding components in the intersection of the selected dimensions of the respective sparse parts:

[0097]

[0098] Among them S i and S j x i ′ and x j The index set of the selected dimension of the sparse part of ′ is specifically defined as:

[0099]

[0100] Finally, the total similarity is:

[0101] sim(x i ′,x j ′)=sim (dense) (x i ′,x j ′)+sim (sparse) (x i ′,x j ′)(8)

[0102] To speed up retrieval, the present invention establishes an inverted index in the sparse part, storing only the values ​​of the selected dimensions and their document IDs. During a search query, the query embeds the corresponding dimension selected in the sparse part and searches the corresponding inverted index, achieving efficient similarity calculation for the sparse part without reconstructing the full-length vector. The dense part can be efficiently calculated using the approximate nearest neighbor (ANN) algorithm.

[0103] Furthermore, the present invention applies the length adaptive embedding method to perform vector conversion and hybrid encoding based on the two basic embedding models BGE-EN-ICL (4096 dimensions) and GTE-QWEN2-INSTRUCT (3584 dimensions), and then uses the generated variable-length vectors for retrieval experiments. The datasets used in the experiment include four commonly used datasets in the field: FIQA, Natural QA, MSMARCO and Hotpot QA. The comparison models are fixed-length embedding models from the BGE and GTE series, and the generated embedding dimensions are 768 and 1024 respectively. The experiment measures the retrieval accuracy from two indicators: NDCG@10 score and RECALL@1000 score. The experimental results are shown in Table 1-2. Among them, the method proposed in the present invention is abbreviated as ALE (Adaptive Length Embedding) and is represented in bold in the table.

[0104] Tables 1 and 2 show the retrieval results of the proposed method compared to other fixed-length dense embedding models using the NDCG@10 score and RECALL@1000 score as evaluation metrics, respectively. The dimension in this method refers to the number of dimensions in the dense part, while the threshold θ for the sparse part is set at 0.75. Taking the NDCG@10 score as an example, at 768 dimensions, the average NDCG@10 scores of the BGE and GTE series improved by 20.46% and 14.83%, respectively, after applying the adaptive length embedding method. At 1024 dimensions, the average NDCG@10 scores of the BGE and GTE series improved by 17.5% and 6.23%, respectively. Even when the dense part dimension is only 512, the proposed method still achieves higher NDCG@10 scores than the 768- and 1024-dimensional fixed-length dense embeddings. Since the time and space costs of retrieval are primarily related to the dimensionality of the dense vector, this demonstrates that the proposed method can achieve higher retrieval accuracy than dense embedding models with higher dimensionality while being less expensive.

[0105] Table 1 NDCG@10 score comparison experimental results

[0106]

[0107] Table 2. Comparative experimental results of RECALL@1000 scores

[0108]

[0109] This paper aims to address the challenges of using fixed-length dense embeddings to balance resource utilization and expressiveness. These challenges include: 1) keyword-based models can only achieve precise word matching and cannot handle synonyms or the overall semantics of paragraphs; 2) dense retrieval models can only encode text into fixed-length embedding vectors, which introduces information redundancy and cannot adjust the encoding length based on the semantic complexity of the text, resulting in unnecessary storage and retrieval costs; and 3) embedding vectors generated by retrieval models based on large language models are typically very high-dimensional, posing challenges to storage and retrieval efficiency.

[0110] First, to address the problem that keyword-based models cannot handle synonyms and paragraph-level semantics, the proposed method further transforms the embeddings generated by the neural network model. The resulting hybrid encoding inherits the properties of the original embeddings and can express the overall semantics at the paragraph level.

[0111] Secondly, to address the problems of dense retrieval models that can only encode text into fixed-length embedding vectors, resulting in information redundancy and an inability to adjust the encoding length based on the semantic complexity of the text, as well as the high dimensionality of embedding vectors based on large models, this paper proposes an adaptive length embedding method. The core concept of this method is to fully utilize the redundancy in high-dimensional embedding vectors and, through the design of a well-designed linear transformation, convert the high-dimensional vectors into a set of linearly independent components. For each document, this method analyzes its semantic complexity and retains only the minimum necessary dimensions to express its semantic information, thereby adaptively matching the embedding length to the document's semantic complexity. Specifically, this method first utilizes an existing high-dimensional embedding model to obtain the original vector representation. Then, by learning a transformation matrix, it maps the high-dimensional vector into a linearly independent subspace. Based on this, a hybrid encoding strategy is adopted: each document's embedding vector is divided into a shorter dense component and a variable-length sparse component. The length of the sparse component is dynamically adjusted to meet the actual semantic requirements of different documents. This ensures that the model has sufficient expressive power to express complex semantics while significantly reducing overall storage and retrieval costs.

[0112] Example 2

[0113] The present invention also provides an adaptive length embedding system for text retrieval, which can be implemented by executing the process steps of the adaptive length embedding method for text retrieval, that is, those skilled in the art can understand the adaptive length embedding method for text retrieval as a preferred implementation of the adaptive length embedding system for text retrieval.

[0114] According to the present invention, an adaptive length embedding system for text retrieval is provided, comprising:

[0115] Module M1: Use the trained embedding model to encode the original document into a high-dimensional embedding vector to obtain the original document embedding matrix X∈R n×d .

[0116] Module M2: The embedding vector is subjected to a learning transformation matrix by a transformation matrix fitting module to obtain a transformed embedding vector. The learning transformation matrix makes the transformed embedding as linearly independent as possible in each dimension by fitting the original high-dimensional embedding, while maintaining the similarity between the text embeddings before and after the transformation. The module M2 includes: Module M2.1: k-means clustering of all high-dimensional embedding vectors. Module M2.2: Randomly extract a subset according to the proportion of the number of samples in each cluster as the data used to fit the transformation matrix P, and calculate the transformation matrix P. Where P∈R d×d is the square matrix to be solved. Module M2.3: After fitting the transformation matrix, use the current matrix to transform all the initial high-dimensional embeddings to obtain linearly independent embeddings of each dimension after transformation. The new embedding matrix X′∈R is obtained through the linear transformation X′=XP. n×d , where n represents the number of documents and d represents the dimension of each embedding vector. The transformation X′=XP must satisfy two conditions: first, the dimensions are linearly independent, that is, the covariance matrix of each column of X′ is a diagonal matrix, thus ensuring that the dimensions are orthogonal after the transformation. Second, the transformed similarity matrix remains consistent with the original similarity matrix. The formal constraints are as follows:

[0117]

[0118] Among them, |·| F represents the Frobenius norm, λ1,λ2,...,λ d Represents the elements on the diagonal of the covariance matrix. Then the covariance matrix of the selected sample is decomposed by eigenvalue, and the formula is as follows:

[0119] X T X=VΛV T (2)

[0120] Where V∈R d×d Denotes an orthogonal matrix of eigenvectors, and Λ denotes a diagonal matrix of eigenvalues. Taking P = V can simultaneously satisfy the two constraints in formula (1). The transformed matrix X′ is equivalent to performing principal component analysis (PCA) on X, with the principal components concentrated in the first several dimensions.

[0121] Module M3: Inputs the transformed embedding vector into a hybrid coding module for hybrid coding. The transformed embedding vector of each document is divided into a fixed-length dense part and a variable-length sparse part. The length of the sparse part is dynamically adjusted according to the semantic complexity of the document. The similarity calculation is then performed on the dense and sparse parts, thereby achieving adaptive text retrieval. The fixed-length dense part and the variable-length sparse part are stored using a vector database and an inverted list, respectively. The hybrid coding module adaptively adjusts the length of its embedding vector based on the semantic complexity of each document, and selects the minimum dimension to fully express the semantics, including: setting a preset threshold θ, sorting each transformed document embedding from large to small by the absolute value of each dimension, and then selecting dimensions in descending order until the proportion of the sum of the squares of the selected dimensions to the total sum of squares exceeds the threshold θ. The dense parts of all document embeddings are unified as the first k dimensions of the transformed vector, where k is a hyperparameter. The remaining dimensions select the minimum dimension as the sparse part as needed to ensure that the sum of the squares of the dense and sparse parts exceeds the preset threshold. The similarity calculation in module M3 includes: Module M3.1: For each vector x i ′, define the dense part as Module M3.2: Calculate the similarity of the dense part as the standard dot product of the first k dimensions, the formula is as follows:

[0122]

[0123] Module M3.3: Calculate the similarity of the sparse parts. The similarity of the sparse parts is the dot product of the corresponding components in the intersection of the selected dimensions of the sparse parts. The calculation formula is as follows:

[0124]

[0125] Among them, S i and S j Represents x i ′ and x j The index set of the selected dimension of the sparse part of ′ is specifically defined as:

[0126]

[0127] Module M3.4: Two-document embedding i ′ and x j The total similarity of the similarity of ′ is:

[0128] sim(x i ′,x j ′)=sim (dense) (x i ′,x j ′)+sim (sparse) (x i ′,x j ′).

[0129] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0130] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. An adaptive length embedding method for text retrieval, characterized in that: include: Step S1: Use the trained embedding model to encode the original document into a high-dimensional embedding vector to obtain the original document embedding matrix X∈R n×d ; Step S2: The embedding vector is subjected to a learning transformation matrix by a transformation matrix fitting module to obtain a transformed embedding vector; Step S3: The converted embedding vector is input into a hybrid coding module for hybrid coding. The converted embedding vector of each document is divided into a fixed-length dense part and a variable-length sparse part. The length of the sparse part is dynamically adjusted according to the semantic complexity of the document. The similarity calculation is then performed on the dense and sparse parts to achieve adaptive text retrieval.

2. The text retrieval-oriented adaptive length embedding method according to claim 1, characterized in that: The learned transformation matrix fits the original high-dimensional embedding so that the transformed embedding is as linearly independent as possible in each dimension while maintaining the similarity between the text embeddings before and after the transformation; The step S2 comprises: Step S2.1: Perform k-means clustering on all high-dimensional embedding vectors; Step S2.2: Randomly select subsets according to the proportion of the number of samples in each cluster as the data for fitting the transformation matrix P, and calculate the transformation matrix P; where P∈R d×d is the matrix to be solved; Step S2.3: After fitting the transformation matrix, use the current matrix to transform all the initial high-dimensional embeddings to obtain the linearly independent embeddings of each dimension after the transformation. The new embedding matrix X′∈R is obtained by linear transformation X′=XP n×d , where n represents the number of documents and d represents the dimension of each embedding vector.

3. The text retrieval-oriented adaptive length embedding method according to claim 2, characterized in that: The transformation X′=XP must satisfy two conditions: First, the dimensions are linearly independent, that is, the covariance matrix of each column of X′ is a diagonal matrix, thus ensuring that the dimensions are orthogonal after the transformation; Second, the transformed similarity matrix remains consistent with the original similarity matrix; The formal constraints are as follows: Among them, |·| F represents the Frobenius norm, λ1,λ2,...,λ d represents the elements on the diagonal of the covariance matrix; Then the covariance matrix of the selected samples is decomposed into eigenvalues, and the formula is as follows: X T X=VΛV T (2) Where V∈R d×d represents the eigenvector orthogonal matrix, Λ represents the eigenvalue diagonal matrix; Taking P = V can simultaneously satisfy the two constraints in formula (1). The transformed matrix X′ is equivalent to performing principal component analysis (PCA) on X, and the principal components are concentrated in the first several dimensions.

4. The text retrieval-oriented adaptive length embedding method according to claim 1, characterized in that: The fixed-length dense part and the variable-length sparse part are stored using a vector database and an inverted list respectively.

5. The text retrieval-oriented adaptive length embedding method according to claim 1, characterized in that: The hybrid encoding module adaptively adjusts the length of the embedding vector of each document based on its semantic complexity, selecting the minimum number of dimensions to fully express the semantics, including: Set a preset threshold θ, and for each transformed document embedding, first sort it by the absolute value of each dimension from large to small, and then select dimensions in order from large to small until the ratio of the sum of squares of the selected dimensions to the total sum of squares exceeds the threshold θ.

6. The text retrieval-oriented adaptive length embedding method according to claim 1, characterized in that: The dense parts of all document embeddings are unified into the first k dimensions of the transformed vector, where k is a hyperparameter; The remaining dimensions select the minimum number of dimensions as the sparse part as needed to ensure that the sum of the squares of the dense part and the sparse part exceeds the preset threshold.

7. The text retrieval-oriented adaptive length embedding method according to claim 1, characterized in that: The similarity calculation in step S3 includes: Step S3.1: For each vector x i ′, define the dense part as x′ i(dense) =(x′ i,1 ,…,x′ i,k ); Step S3.2: Calculate the similarity of the dense part as the standard dot product of the first k dimensions, as follows: Step S3.3: Calculate the similarity of the sparse parts. The similarity of the sparse parts is the dot product of the corresponding components in the intersection of the selected dimensions of the respective sparse parts. The calculation formula is as follows: Among them, x i and x j Represents x i ′ and x j The index set of the selected dimension of the sparse part of ′ is specifically defined as: Step S3.4: Two documents are embedded in x i ′ and x j The total similarity of the similarity of ′ is: yes(x) i ′,x j ′)=yes (dense) (x i ′,x j ′)+yes (sparse) (x i ′,x j ′)。 8. An adaptive length embedding system for text retrieval, characterized by: include: Module M1: Use the trained embedding model to encode the original document into a high-dimensional embedding vector to obtain the original document embedding matrix X∈R n×d ; Module M2: The embedding vector is subjected to a learning transformation matrix by a transformation matrix fitting module to obtain a transformed embedding vector; Module M3: Input the converted embedding vector into the hybrid coding module for hybrid coding, divide the converted embedding vector of each document into a fixed-length dense part and a variable-length sparse part, and dynamically adjust the length of the sparse part according to the semantic complexity of the document. Then, similarity calculation is performed on the dense and sparse parts to achieve adaptive text retrieval.

9. The text retrieval-oriented adaptive length embedding system according to claim 8, characterized in that: The learned transformation matrix fits the original high-dimensional embedding so that the transformed embedding is as linearly independent as possible in each dimension while maintaining the similarity between the text embeddings before and after the transformation; The module M2 includes: Module M2.1: Perform k-means clustering on all high-dimensional embedding vectors; Module M2.2: Randomly select subsets according to the proportion of the number of samples in each cluster as data for fitting the transformation matrix P, and calculate the transformation matrix P; where P∈R d×d is the matrix to be solved; Module M2.3: After fitting the transformation matrix, use the current matrix to transform all the initial high-dimensional embeddings to obtain linearly independent embeddings of each dimension after transformation. The new embedding matrix X′∈R is obtained by linear transformation X′=XP n×d , where n represents the number of documents and d represents the dimension of each embedding vector.

10. The text retrieval-oriented adaptive length embedding system according to claim 8, characterized in that: The fixed-length dense part and the variable-length sparse part are stored using a vector database and an inverted list respectively.

Images