A cross-document type retrieval method and system based on differential nested encoding

Abstract

The application discloses a cross-document type retrieval method and system based on differential nested coding. The method includes six steps: document preprocessing, document type identification, differential coding, index construction and optimization, online retrieval and result fusion. First, the document is standardized and preprocessed to extract structured information. Second, the BERT classifier is used to analyze the document features and identify the document type. Then, according to the document type, the corresponding encoder is selected to generate the original coding vector. Next, the differential nested coding index is constructed and optimized. In the online retrieval stage, the user query is preprocessed, the relevant document type is predicted, and the approximate nearest neighbor retrieval is performed. Finally, the retrieval results are sorted by a stepwise weight fusion strategy, and the top-100 retrieval results are returned. Through differential coding and index optimization, the application realizes efficient retrieval of different types of documents, and has significant practical value and application prospect.

Description

Technical Field

[0001] This application relates to the fields of information retrieval and natural language processing, and in particular to a cross-document type retrieval method and system based on differential nested encoding. Background Technology

[0002] Current vector retrieval systems commonly employ fixed-dimensional vector representation, meaning that all document types are encoded using the same vector dimensions. This uniform encoding approach has the following significant problems:

[0003] Significant resource waste: For short documents with simple structure and single content (such as news flashes, brief Q&As, etc.), using high-dimensional vectors (such as 2048-dimensional or 4096-dimensional vectors) will result in a serious waste of storage space and computing resources. For example, a short notice containing only a few dozen words, when represented by a 4096-dimensional vector, has extremely high information redundancy in most dimensions, and the actual effective information may only require 256 dimensions to be fully expressed.

[0004] Insufficient expressive power: For long documents with complex content and high information density (such as academic papers, patent documents, technical specifications, etc.), fixed low-dimensional vectors (such as 768 dimensions) cannot fully capture their rich semantic information, resulting in a decrease in retrieval accuracy. Complex documents contain multi-level semantic relationships, professional terminology, logical structures and other fine-grained features. These features are over-compressed in low-dimensional space, causing information loss.

[0005] Inefficient retrieval: In hybrid document repositories, simple documents are forced to use high-dimensional representations, increasing unnecessary vector computation overhead and index storage costs; while complex documents use insufficient-dimensional representations, requiring additional post-processing (such as reordering and multi-stage retrieval) to compensate for accuracy loss. It is difficult to simultaneously optimize the overall system's retrieval efficiency and quality.

[0006] Cost control is challenging: the storage and query costs of large-scale vector databases increase linearly with dimensionality. Adopting a fixed high-dimensional strategy (e.g., 4096 dimensions) leads to persistently high overall costs (significantly increasing storage space, memory usage, and computation time); while adopting a fixed low-dimensional strategy (e.g., 256 dimensions) sacrifices retrieval quality, particularly resulting in a significant drop in recall for complex queries and long documents. Enterprises struggle to find a balance between cost and quality. Summary of the Invention

[0007] Based on this, embodiments of this application provide a cross-document type retrieval method and system based on differentiated nested encoding. By intelligently identifying document types and dynamically allocating optimal vector dimensions, it solves problems such as waste of storage resources, low retrieval efficiency, and insufficient expressive power in the prior art, and achieves the optimal balance between retrieval quality, storage cost, and computational efficiency.

[0008] Firstly, a cross-document type retrieval method based on differential nested encoding is provided. This method includes six steps: document preprocessing, document type identification, differential encoding, index construction and optimization, online retrieval, and result fusion. Specifically, it includes:

[0009] Document preprocessing: Read the original document collection and perform standardized preprocessing on the documents, including removing special characters, unifying the encoding format, word segmentation, removing duplicate documents, and extracting the structured information of the documents;

[0010] Document type identification: The trained BERT classifier is used to identify the type of the document, analyze the multidimensional features of the document, output the probability distribution of the document's type, and select the type with the highest probability as the document's category label;

[0011] Differential encoding: Based on the document type, select the corresponding encoder to vectorize the document and generate a 4096-dimensional original encoded vector;

[0012] Index building and optimization: Based on the document type, the corresponding prefix of the original encoding vector is extracted as a nested encoding, which is then inserted into the corresponding type of FAISS index. The complete 4096-dimensional encoding of the document is stored in an independent vector storage system to optimize the index.

[0013] Online retrieval: Receives user queries, preprocesses and extracts features from the queries, uses the BERT classifier to predict the most likely document type related to the query, generates query vectors for the predicted document types using the corresponding encoder, performs approximate nearest neighbor retrieval in parallel in the corresponding index, and returns candidate documents and their similarity scores;

[0014] Result fusion: The candidate documents returned by different indexes are merged into the result pool. For duplicate documents, the record with the highest similarity score is retained. Based on the confidence score predicted by the document type and the original similarity score retrieved, a step-weighted fusion strategy is used to calculate the fusion score, the candidate documents are sorted, and the sorted retrieval results are returned.

[0015] Optionally, the differential coding step specifically includes:

[0016] Establish a mapping relationship between document types and encoding dimensions, and assign nested encodings of different lengths to different types of documents;

[0017] A nested encoding structure is designed to ensure that all encoder-generated vectors adopt a nested structure, wherein the high-dimensional vector contains all the information of the low-dimensional vector. The encoders include a news encoder, a web page encoder, a technical document encoder, a thesis encoder, and a patent encoder, each of which is optimized for its corresponding document type. The 4096-dimensional original encoded vector is mapped through a linear projection layer and then subjected to L2 normalization.

[0018] Based on the document type, only the corresponding length of encoding is stored in the index; specifically, documents identified as news are encoded with 512 dimensions to news_index_512, documents identified as web pages are encoded with 768 dimensions to web_index_768, documents identified as technology documents are encoded with 1024 dimensions to tech_index_1024, documents identified as papers are encoded with 2048 dimensions to paper_index_2048, and documents identified as patents are encoded with 2560 dimensions to patent_index_2560.

[0019] Optionally, the differential encoding step further includes adaptive multi-granularity text embedding representation learning based on gradient optimization, specifically including:

[0020] During the initialization phase, a pre-trained large language model, Qwen3-8b, is used as the base encoder. A learnable linear projection layer is added on top of the encoder to project the hidden state dimension onto the target dimension of 4096. Xavier uniform initialization or normal distribution initialization is used. Training is then performed, specifically including:

[0021] In the first phase, the focus is on optimizing the representation quality of the first 512 dimensions;

[0022] In the second phase, the optimization scope is expanded to the first 768 dimensions while maintaining the performance of the first 512 dimensions.

[0023] In the third stage, the optimization scope is expanded to the first 1024 dimensions;

[0024] In the fourth stage, the optimization scope is expanded to the first 2048 dimensions, with a focus on academic paper documents.

[0025] In the final stage, the complete 2560-dimensional representation is optimized, mainly for patent documents;

[0026] In each training phase, the gradient is backpropagated through all 4096 dimensions of parameters, and the loss function is set to focus only on the dimension range of the current optimization.

[0027] Optionally, the online retrieval and result fusion step includes:

[0028] Query preprocessing involves standardizing the query text input by the user, including text cleaning, case normalization, word segmentation, stop word filtering, and query expansion.

[0029] Query feature extraction: Extract multidimensional features from the query for subsequent type prediction.

[0030] Document type prediction: Use a trained BERT classifier to predict the document type most likely to be relevant to the query.

[0031] Multi-path vector encoding generates a query vector of the corresponding dimension for each document type being predicted;

[0032] Parallel index retrieval performs approximate nearest neighbor retrieval in parallel within the index corresponding to the predicted document type;

[0033] A tiered weighted fusion strategy is employed, which considers the predicted confidence of document type and the original similarity score of the retrieval to scientifically integrate results from multiple sources.

[0034] All candidate documents are sorted in descending order based on the fusion score, and the Top-100 are selected as the initial recall results.

[0035] The query preprocessing includes removing invalid characters, unifying case, word segmentation, removing stop words, and query expansion.

[0036] The word segmentation process selects different word segmentation tools based on the language type of the document; jieba is used for Chinese and space is used for English.

[0037] The feature extraction includes extracting the number of words in the query, character length, frequency of professional terms, distribution of named entity types, query intent type, and domain features;

[0038] The query intent types include information search, question answering, and exploratory search.

[0039] Optionally, the result fusion step further includes a multi-scale re-ranking strategy, specifically including the following steps:

[0040] Encoding retrieval involves retrieving the original 4096-dimensional encoding vectors from the storage system for the 100 initially recalled candidate documents. The vector index stores the truncated encoding for fast recall, while a separate vector store holds the complete 4096-dimensional encoding for fine ranking. Retrieval optimization includes batch retrieval of encoding vectors to reduce I / O operations.

[0041] Generate a multi-scale query vector, and truncate the 4096-dimensional code of the query into sub-codes of different lengths;

[0042] Calculate multi-scale similarity: for each candidate document, calculate its similarity to the query across different dimensions;

[0043] The tiered weight fusion adopts a tiered weight strategy similar to that of the recall phase, but with different weight configurations.

[0044] The final sorting is performed in descending order based on the reordering scores, and the Top-20 are selected as the final return results.

[0045] The multi-scale reordering strategy utilizes sub-encoders of different lengths from the original encoding for multi-way reordering.

[0046] Optionally, in the document preprocessing step, the structured information includes the document's title, body, abstract, author, source, publication time, document length, and language type;

[0047] The process of removing duplicate documents involves identifying and deduplicating approximately duplicate documents using the SimHash or MinHash algorithm.

[0048] The preprocessed document is stored in JSON format and contains document ID, text content, and metadata fields.

[0049] Optionally, in the result fusion step, the step-weighted fusion strategy dynamically adjusts the weights according to the document type dimension and confidence level; wherein, the weights adopt a geometric sequence form, and the weight values ​​increase step by step according to the document type dimension;

[0050] In calculating the fusion score using a tiered weighted fusion strategy, the formula for calculating the fusion score is as follows:

[0051]

[0052] Where d is the candidate document, T is the set of predicted document types, and w t It is the basic weight of document type, confidence. t It is the confidence score given by the type predictor. It is the similarity score retrieved from the index of type t.

[0053] Secondly, a cross-document type retrieval system based on differentiated nested encoding is provided, the system comprising:

[0054] The document preprocessing module is used to read the original document collection and perform standardized preprocessing on the documents, including removing special characters, unifying the encoding format, word segmentation, removing duplicate documents, and extracting the structured information of the documents;

[0055] The document type recognition module is used to identify the type of a document using a trained BERT classifier, analyze the multidimensional features of the document, output the probability distribution of the document's type, and select the type with the highest probability as the document's category label.

[0056] The differential encoding module is used to select the corresponding encoder to vectorize the document based on the document type, generating a 4096-dimensional original encoded vector;

[0057] The index building and optimization module is used to extract the corresponding prefix of the original encoding vector as a nested encoding according to the document type, insert it into the corresponding type of FAISS index, and store the complete 4096-dimensional encoding of the document in an independent vector storage system to optimize the index.

[0058] The online retrieval module is used to receive user queries, preprocess and extract features from the queries, use the BERT classifier to predict the most likely document type related to the query, use the corresponding encoder to generate query vectors for the predicted document types, and perform approximate nearest neighbor retrieval in parallel in the corresponding index, and return candidate documents and their similarity scores.

[0059] The results fusion module is used to merge candidate documents returned by different indexes into the results pool. For duplicate documents, the record with the highest similarity score is retained. Based on the confidence score predicted by the document type and the original similarity score retrieved, a step-weighted fusion strategy is used to calculate the fusion score, sort the candidate documents, and return the sorted search results.

[0060] Thirdly, an electronic device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the cross-document type retrieval method based on differential nested encoding as described in any of the first aspects above.

[0061] Fourthly, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the cross-document type retrieval method based on differential nested encoding as described in any of the first aspects above.

[0062] The beneficial effects of the technical solutions provided in this application include at least the following:

[0063] (1) Through differentiated nested encoding, the system can dynamically allocate the optimal vector dimension according to the document type. Simple documents (such as news and web pages) use low-dimensional encoding, while complex documents (such as academic papers and patent documents) use high-dimensional encoding, avoiding the waste of resources and information loss caused by uniform dimension encoding. This adaptive dimension allocation method significantly improves retrieval efficiency and enhances retrieval quality, especially in complex query and long document retrieval scenarios, where it can more accurately capture semantic information and improve recall and precision.

[0064] (2) Compared with traditional fixed high-dimensional encoding, the differentiated nested encoding strategy significantly reduces storage space requirements. For example, the average storage space is reduced by about 67% compared with uniform 4096-dimensional encoding. In addition, the query speed of low-dimensional indexes is faster, reducing unnecessary high-dimensional computational overhead and further reducing the computational cost of the system. This cost optimization enables large-scale document retrieval systems to significantly reduce the consumption of storage and computing resources while maintaining efficient retrieval, resulting in significant economic benefits.

[0065] (3) By establishing a mapping relationship between document types and encoding dimensions, this invention can flexibly adapt to different types of documents. The system can dynamically adjust the encoding dimensions according to the complexity, information density, and semantic features of the document, supporting multiple document types (such as news, web pages, technical documents, academic papers, patent documents, etc.). In addition, the system also has online learning and adaptive capabilities, and can dynamically adjust according to changes in the document library and the evolution of user query patterns, ensuring the stability and maintainability of the system.

[0066] (4) Through a multi-path recall and tiered weight fusion strategy, the system can comprehensively consider the relevance of different document types, the matching degree of the search, and the expressive power of dimensions to achieve scientific integration of multi-path results. In addition, the multi-scale re-ranking strategy uses sub-encodings of different lengths for precise calculation, further improving the search accuracy. This multi-dimensional fusion and re-ranking mechanism not only improves the accuracy of search results but also increases the diversity of results, avoiding excessive similarity among top results, thereby better meeting the diverse search needs of users. Attached Figure Description

[0067] To more clearly illustrate the embodiments of this application or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other embodiments based on the provided drawings without creative effort.

[0068] Figure 1 A flowchart illustrating the steps of a cross-document type retrieval method based on differential nested encoding provided in this application embodiment;

[0069] Figure 2 The system architecture diagram provided for the embodiments of this application;

[0070] Figure 3 A flowchart for document type identification provided in this application embodiment;

[0071] Figure 4 A flowchart of multi-channel recall and fusion provided for embodiments of this application;

[0072] Figure 5 A flowchart of multi-scale reordering provided for embodiments of this application;

[0073] Figure 6 A comparison chart of retrieval performance provided for embodiments of this application. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0075] In the description of this application, the terms "comprising," "having," and any variations thereof are intended to cover non-exclusive inclusion, such as a process, method, system, product, or apparatus that includes a series of steps or units, not necessarily limited to those steps or units that are expressly listed, but may also include other steps or units that are not expressly listed but are inherent to these processes, methods, products, or apparatuses, or steps or units added based on further optimizations conceived in this application.

[0076] In this application, the vector retrieval system with adaptive dimension allocation faces the following technical challenges:

[0077] Challenge 1: Accurate document type identification

[0078] A scientific document classification system needs to be established to accurately identify the complexity, information density, and semantic features of documents. This requires comprehensive consideration of multiple dimensions, including text length, lexical diversity, syntactic complexity, domain characteristics, and citation relationships. Documents from different domains have different feature distributions, making accurate classification difficult with a single feature.

[0079] Challenge 2: Precise quantification of dimensional requirements

[0080] It is necessary to establish a scientific mapping relationship between document features and vector dimensionality requirements to determine the optimal dimensions required for different document types. Too low a dimension leads to information loss and decreased retrieval quality; too high a dimension results in wasted resources and increased computational overhead. A key challenge is how to quantitatively assess the balance between "sufficient representation" and "avoiding redundancy."

[0081] Challenge 3: Unified retrieval of multi-dimensional vectors

[0082] Within the same search space, it's necessary to handle the similarity calculation and ranking of vectors with different dimensions. Vectors of different dimensions cannot be directly compared, requiring the design of reasonable normalization and fusion strategies to ensure the fairness and accuracy of search results. Achieving collaborative retrieval across multiple indexes while maintaining efficiency is a key technical challenge.

[0083] Challenge 4: System Dynamic Adaptability

[0084] A mechanism needs to be designed to dynamically adjust based on changes in the document library and query characteristics. The emergence of new document types, changes in user query patterns, and shifts in data distribution all require the system to have online learning and adaptive capabilities. At the same time, the system's stability and maintainability must be guaranteed.

[0085] Challenge 5: Complexity of Engineering Implementation

[0086] Managing multiple encoders, multiple vector indices, complex routing logic, and fusion strategies significantly increases the difficulty of system deployment, monitoring, debugging, and maintenance. Simplifying engineering implementation while maintaining performance is crucial for practical applications.

[0087] The purpose of this invention is to provide a cross-document type retrieval system and method based on differentiated nested encoding. By intelligently identifying document types and dynamically allocating optimal vector dimensions, it solves the problems of wasteful storage resources, low retrieval efficiency, and insufficient expressive power in the prior art, and achieves the optimal balance between retrieval quality, storage cost, and computational efficiency.

[0088] The core idea of ​​this invention is that different types of documents require vector representations of different dimensions. By identifying document types and adaptively allocating dimensions, the most suitable encoding dimension is selected for each type of document, constructing a differentiated nested encoding index system to achieve the best results with minimal resources.

[0089] Please refer to Figure 1 The document illustrates a flowchart of a cross-document type retrieval method based on differential nested encoding provided in this application embodiment, comprising six steps: document preprocessing, document type identification, differential encoding, index construction and optimization, online retrieval and result fusion.

[0090] Document preprocessing: Read the original document collection and perform standardized preprocessing on the documents, including removing special characters, unifying the encoding format, word segmentation, removing duplicate documents, and extracting the structured information of the documents.

[0091] Document type identification: The trained BERT classifier is used to identify the type of the document, analyze the multidimensional features of the document, output the probability distribution of the document's type, and select the type with the highest probability as the document's category label.

[0092] Differential encoding: Based on the document type, select the corresponding encoder to vectorize the document and generate a 4096-dimensional original encoded vector.

[0093] Index building and optimization: Based on the document type, the corresponding prefix of the original encoding vector is extracted as a nested encoding, which is then inserted into the corresponding type of FAISS index. The complete 4096-dimensional encoding of the document is stored in a separate vector storage system to optimize the index.

[0094] Online retrieval: Receives user queries, preprocesses and extracts features from the queries, uses a BERT classifier to predict the most likely document type related to the query, generates query vectors for the predicted document types using the corresponding encoder, performs approximate nearest neighbor retrieval in parallel in the corresponding index, and returns candidate documents and their similarity scores.

[0095] Result fusion: The candidate documents returned by different indexes are merged into the result pool. For duplicate documents, the record with the highest similarity score is retained. Based on the confidence score predicted by the document type and the original similarity score retrieved, a step-weighted fusion strategy is used to calculate the fusion score, the candidate documents are sorted, and the sorted retrieval results are returned.

[0096] Based on the above methods, such as Figure 2 This application provides a system architecture diagram for implementing the above method, which may include:

[0097] The system is divided into two main parts: offline index building and online retrieval services. The offline part is responsible for document classification, encoding, and index building; the online part is responsible for query processing, multi-index retrieval, and result fusion.

[0098] The following are the key technical points of this application:

[0099] Innovation Point 1: Differentiated Nested Encoding and Storage Strategy

[0100] Core idea: Store nested codes of different lengths in the recall index according to different document types.

[0101] like Figure 3A flowchart for document type identification is provided. The process begins with the input document and firstly, text preprocessing is performed, which may include steps such as noise removal, format standardization, and word segmentation. Next, the system extracts various features from the preprocessed text using a feature extractor. These features include basic features (such as document length and word count), lexical features (such as diversity and term density), structural features (such as paragraphs and citations), and domain features (obtained through matching with a specialized dictionary).

[0102] The extracted features are integrated into a feature vector, which is then fed into a BERT classifier for 5-class classification. The BERT classifier is a pre-trained deep learning model capable of understanding the context and semantics of text. After processing by the BERT classifier, the output is converted into a probability distribution through a Softmax layer, with each class corresponding to a probability value representing the likelihood that a document belongs to that class.

[0103] Based on these probability values, the system determines the document type, classifying it into the category with the highest probability: news, webpage, technical, academic paper, or patent. Finally, the system outputs the classification result and its corresponding confidence score, with the confidence score reflecting the reliability of the classification result.

[0104] Technical Implementation:

[0105] The first step is to establish a mapping relationship between document types and coding dimensions:

[0106] News documents (news_embedding): use 512-dimensional encoding, index name is news_index_512, specifically:

[0107] Features: concise content, low information density, and high timeliness;

[0108] Typical documents: news flashes, social media posts, short messages;

[0109] Reason for setting the dimensions: 512 dimensions are sufficient to express simple semantics, and higher dimensions will cause overfitting.

[0110] Web page documents (web_embedding): use 768-dimensional encoding, index name is web_index_768, specifically:

[0111] Features: Semi-standard structure, diverse content, medium length;

[0112] Typical documents: blog posts, product descriptions, FAQ pages;

[0113] Reason for setting the dimensions: 768 dimensions is the standard dimension for BERT-base, and the balance effect is good.

[0114] Technical documentation (tech_embedding): Encoded in 1024 dimensions, indexed as tech_index_1024, specifically:

[0115] Features: Rich in technical terms, highly logical, and clearly structured;

[0116] Typical documents: API documentation, technical manuals, user guides;

[0117] Reason for setting the dimensions: 1024 dimensions can capture fine-grained semantics in a professional field.

[0118] Academic papers (paper_embedding): use 2048-dimensional encoding, index name is paper_index_2048, specifically:

[0119] Characteristics: Long, information-dense, and complex in structure;

[0120] Typical documents: journal articles, conference papers, dissertations;

[0121] Rationale for setting the dimensions: 2048 dimensions can express multi-level academic concepts and logical relationships.

[0122] Patent documents (patent_embedding): use 2560-dimensional encoding, index name is patent_index_2560, specifically:

[0123] Features: Highly professional, rigorously structured, and densely packed with legal terminology;

[0124] Typical documents: Invention patents, utility model patents, and design patents;

[0125] Reason for setting the dimensions: 2560 dimensions meet the high-precision retrieval needs of patent documents.

[0126] The nested encoding design ensures that all vectors generated by the encoders adopt a nested structure, meaning that the higher-dimensional vector contains all the information of the lower-dimensional vector. Specifically, the nested encoding in this application includes:

[0127] First, the original coding is used to generate patent-level coding, the highest-dimensional coding containing the first 2560 dimensions of information, specifically designed for patent documents to meet their high-precision retrieval needs. Next, paper-level coding, containing the first 2048 dimensions of information, can be derived from the patent-level coding, suitable for information-intensive documents such as academic papers.

[0128] Furthermore, technical-level coding can be extracted from paper-level coding, containing the first 1024 dimensions of information, suitable for technical documents that contain technical terminology and logical structures. Similarly, webpage-level coding extracts the first 768 dimensions from technical-level coding, suitable for webpage documents with semi-standard structures and diverse content.

[0129] Finally, news-level coding extracts the first 512 dimensions from webpage-level coding, specifically designed for news documents with concise content and low information density. This nested structure allows high-dimensional coding to naturally contain all the information from low-dimensional coding, thereby enabling efficient utilization and comparison of vectors with different dimensions in the retrieval system.

[0130] Mathematical expression:

[0131] Let the original code be ,but:

[0132] News-level coding:

[0133] Webpage-level encoding:

[0134] Technical coding:

[0135] Paper-level coding:

[0136] Patent-level code:

[0137] in This indicates taking the first d dimensions of the vector.

[0138] Storage strategy:

[0139] Based on the document type, only the corresponding length of encoding is stored in the index:

[0140] Documents identified as news: Store 512 dimensions in news_index_512

[0141] Documents identified as web pages: Store 768 dimensions in web_index_768

[0142] Documents identified as technical: Store 1024 dimensions in tech_index_1024

[0143] Documents identified as academic papers: Store 2048 dimensions in paper_index_2048

[0144] Documents identified as patents: Store 2560 dimensions in patent_index_2560

[0145] Advantages analysis:

[0146] Significant savings in storage space: Compared to using a uniform 4096 dimensions, the average storage space is reduced by approximately 67%;

[0147] Improved retrieval efficiency: Low-dimensional indexes enable faster query speeds and reduce unnecessary high-dimensional calculations;

[0148] Lossless or improved quality: Use the most suitable dimension for each type of document to avoid overfitting or underfitting;

[0149] Highly flexible: The dimensional configuration of various documents can be adjusted according to actual needs.

[0150] Innovation Point Two: Adaptive Multi-Granularity Text Embedding Representation Learning Based on Gradient Optimization

[0151] Core idea: By using a sequential training strategy, the model can automatically learn the optimal representation of different dimensions during the training process, ensuring the quality of nested encoding.

[0152] Detailed implementation steps:

[0153] Step 1: Initialization Phase

[0154] The pre-trained large language model Qwen3-8b is used as the base encoder. A learnable linear projection layer is added on top of the encoder to project the hidden state dimension (typically 1024) onto the target dimension of 4096.

[0155] Initialization method: Use Xavier uniform initialization or normal distribution initialization to ensure that the initial values ​​of each dimension are reasonably distributed.

[0156] Step 2: Phase 1 Training (512-Dimensional Optimization)

[0157] In the first phase, the focus is on optimizing the representation quality of the first 512 dimensions.

[0158] Loss function design:

[0159]

[0160] in: It uses contrastive learning loss to ensure that positive sample pairs are similar and negative sample pairs are dissimilar. It is a regularization loss, used to prevent overfitting. It is the regularization coefficient, with a typical value of 0.01.

[0161] Contrastive learning loss is expressed in the form of InfoNCE:

[0162]

[0163] in: It is a query code. It is a positive sample encoding. It is negative sample encoding. It is the cosine similarity function. This is the temperature parameter, with a typical value of 0.05. N is the number of negative samples; it is recommended to use a large batch size (such as 256 or 512).

[0164] Training data: News documents and their related query pairs are used as the main training data, supplemented by other types of documents to maintain the model's generalization ability.

[0165] Optimizer configuration: Using AdamW optimizer, learning rate 2e-5, warmup steps 10%, weight decay 0.01.

[0166] Training epochs: Stop after the validation set performance stabilizes, usually requiring 3-5 epochs.

[0167] Step 3: Phase 2 Training (768-Dimensional Optimization)

[0168] In the second phase, the optimization scope is expanded to the first 768 dimensions while maintaining the performance of the first 512 dimensions.

[0169] Loss function design:

[0170]

[0171] in: and These are weighting coefficients, typically 0.3 and 0.7, which ensure that 512-dimensional performance is maintained while optimizing 768-dimensional performance, and retain the 512-dimensional loss term to prevent catastrophic forgetting.

[0172] Training data: Increase the proportion of web-based documents while retaining news-based documents. A suggested ratio is 30% news, 50% web-based documents, and 20% other types.

[0173] Learning rate adjustment: Appropriately reduce the learning rate to 1.5e-5 to avoid causing excessive perturbation to the already optimized 512 dimensions.

[0174] Step 4: Phase 3 Training (1024-Dimensional Optimization)

[0175] In the third stage, the optimization scope is expanded to the first 1024 dimensions.

[0176] Loss function design:

[0177] Weight configuration: = 0.2, = 0.3, = 0.5, gradually shifting the focus of optimization to higher dimensions.

[0178] Training data: mainly technical documents (50%), with a certain proportion of news and web documents (25% each).

[0179] Step 5: Phase 4 Training (2048-Dimensional Optimization)

[0180] In the fourth phase, the optimization scope is expanded to the first 2048 dimensions, with a focus on academic papers.

[0181] Loss function design:

[0182]

[0183] Weight configuration: .

[0184] Training data: mainly academic papers (60%), with technical documents (20%) and other types (20%) retained.

[0185] Special handling: For long documents, a strategy of segmented encoding followed by aggregation is adopted to ensure full utilization of the expressive power of 2048 dimensions.

[0186] Step 6: Phase 5 Training (2560-dimensional optimization)

[0187] In the final stage, the complete 2560-dimensional representation is optimized, primarily for patent documents.

[0188] Loss function design:

[0189] Weight configuration: .

[0190] Training data: mainly patent literature (70%), with retained papers (15%) and technical documents (15%).

[0191] Fine-tuning: Using a smaller learning rate (5e-6) and more training rounds ensures the stability of the high-dimensional representation.

[0192] Gradient optimization mechanism:

[0193] In each training phase, gradients are backpropagated through all 4096 dimensions of parameters, but the loss function only focuses on the dimension range currently being optimized. This design ensures that:

[0194] 1. Global optimization: All parameters are updated at every stage, avoiding the disconnect between dimensions.

[0195] 2. Local Focus: The loss function is optimized in a specific dimension range to improve training efficiency.

[0196] 3. Gradual Improvement: Optimize step by step from low dimension to high dimension, which conforms to the law of increasing difficulty in learning.

[0197] 4. Knowledge Retention: Maintain the performance of optimized dimensions through multi-task loss to prevent forgetting.

[0198] Regularization strategy:

[0199] In addition to the contrastive learning loss, the following regularization term is also introduced:

[0200] Dimensional smoothing regularization:

[0201]

[0202] Ensure that the values ​​of adjacent dimensions change smoothly to avoid abrupt changes.

[0203] Training monitoring metrics:

[0204] During training, continuously monitor the following metrics to evaluate model quality:

[0205] 1. Search performance across various dimensions: Recall@10, MRR@10.

[0206] 2. Performance consistency across dimensions: Ensure that performance in low dimensions does not degrade during training in high dimensions.

[0207] 3. Convergence trend of loss value: The loss should decrease steadily in each stage.

[0208] 4. Validation set performance: to prevent overfitting.

[0209] Innovation Point 3: Multi-channel recall and tiered weighting fusion strategy

[0210] The core idea is to predict the most relevant document type based on query features during the retrieval phase, recall the document in the corresponding index, and integrate the results from multiple sources through a scientific weight fusion strategy.

[0211] like Figure 4 A query processing flowchart (i.e., a multi-path recall and fusion flowchart) is provided, specifically including the entire process from user query input to the final return of Top-100 search results. The process begins with the user initiating a query, followed by query preprocessing, including text cleaning and word segmentation. Next, the system determines the query type based on specific strategies (such as tags, domains, and intents) and performs document type identification to determine the most relevant document categories.

[0212] After identifying the document type, the system predicts the confidence scores for the top-3 types. Based on these predictions, the system generates query vectors of corresponding dimensions for each type; for example, a 1024-dimensional vector for technical documents, a 2048-dimensional vector for academic papers, a 768-dimensional vector for web pages, a 2560-dimensional vector for patents, and a 512-dimensional vector for news articles.

[0213] Then, the system performs parallel retrieval in the corresponding indexes, using the generated query vectors to retrieve the Top-100 candidate documents from their respective indexes (such as tech_index_1024, paper_index_2048, etc.). After the retrieval is complete, the system merges these candidate documents into a result pool and performs deduplication, retaining the document with the highest similarity score.

[0214] Finally, the system performs a tiered weighted fusion of the documents in the results pool, calculates a fusion score, and comprehensively considers the confidence level of document type prediction and the similarity score obtained from the retrieval. The documents are then sorted according to their fusion scores, and the top 100 documents with the highest scores are returned to the user as the final search results.

[0215] Detailed implementation steps:

[0216] Step 1: Query Preprocessing

[0217] Standardize the query text entered by the user:

[0218] Text cleaning: Remove special characters, HTML tags, extra spaces, etc.

[0219] Case normalization: convert to lowercase (for English) or keep as is (for Chinese);

[0220] Word segmentation: Use jieba for Chinese or space for English.

[0221] Stop word filtering: Optionally removes high-frequency meaningless words;

[0222] Query expansion: Optionally add synonyms or related terms.

[0223] Step 2: Query Feature Extraction

[0224] Extracting multidimensional features from the query for subsequent type prediction:

[0225] 1. Length characteristics: number of characters, number of words, average word length.

[0226] 2. Lexical characteristics: frequency of professional terms, distribution of named entity types, and importance of keywords.

[0227] 3. Intent characteristics: query type (factual, analytical, exploratory), time sensitivity, and accuracy requirements.

[0228] 4. Domain characteristics: academic domain (matched through domain dictionary), technical domain (e.g., artificial intelligence, biomedicine), and degree of generality.

[0229] Step 3: Document Type Prediction

[0230] Use a trained BERT classifier to predict the most likely document type related to the query:

[0231] Model structure:

[0232] Input: Query text;

[0233] Encoder: BERT-based or similar pre-trained model;

[0234] Classification head: Fully connected layer + Softmax;

[0235] Output: Probability distributions of 5 types.

[0236] Training data: using historical query-click logs, where the clicked document type is used as a label.

[0237] Prediction strategy: Select the top-3 types with the highest confidence for multi-path recall. Threshold setting: If the highest confidence exceeds 0.8, only the top-1 types can be retrieved to improve efficiency.

[0238] Confidence calibration: The predicted probability is calibrated using the temperature scaling method to ensure that the confidence level reflects the true accuracy.

[0239] Step 4: Multipath Vector Encoding

[0240] For each predicted document type, generate a query vector corresponding to the specified dimension:

[0241] For the prediction "technical document": use the technical document encoder to generate a 1024-dimensional query vector;

[0242] For the prediction "paper": use the paper encoder to generate a 2048-dimensional query vector;

[0243] For the prediction "web page": use a web page encoder to generate a 768-dimensional query vector;

[0244] For the prediction "patent": use a web encoder to generate a 2560-dimensional query vector;

[0245] For the prediction "news": use a web encoder to generate a 512-dimensional query vector.

[0246] Encoding process:

[0247] 1. Input the query text into the corresponding encoder;

[0248] 2. Extract the representation of the [CLS] tag or perform average pooling;

[0249] 3. Map to the target dimension through a projection layer;

[0250] 4. Perform L2 normalization to ensure that the vector magnitude is 1.

[0251] Step 5: Parallel Index Retrieval

[0252] Perform approximate nearest neighbor (ANN) retrieval in parallel within the index corresponding to the predicted document type:

[0253] Using a 1024-dimensional query vector in tech_index_1024, the candidate and its similarity score are retrieved.

[0254] Using a 2048-dimensional query vector retrieval in paper_index_2048, the candidate and its similarity score are returned.

[0255] Retrieve candidates and their similarity scores using a 768-dimensional query vector in web_index_768;

[0256] Using a 2560-dimensional query vector search in patent_index_768, the candidate and its similarity score are returned.

[0257] Retrieve candidates and their similarity scores using a 512-dimensional query vector in news_index_768.

[0258] Parallel strategy: Use thread pools or asynchronous I / O to implement parallel retrieval, reducing overall latency. Each index retrieval can be completed independently, and the results are aggregated at the end.

[0259] ANN Algorithm: Efficient retrieval is achieved using FAISS's IndexHNSW. Parameter Configuration:

[0260] nprobe: Set to 5-10% of the number of indexed clusters;

[0261] efSearch: Set to 150.

[0262] Step 6: Step-by-step weight fusion

[0263] This is the key innovation of this invention. The fusion strategy considers two factors: the predicted confidence of document type and the original similarity score of the retrieval.

[0264] Fusion formula:

[0265] For each candidate document d, its fusion score is calculated as follows:

[0266]

[0267] in:

[0268] T is the set of predicted document types (e.g., {technical, academic, web page}).

[0269] w_t is the base weight of document type t;

[0270] It is the confidence score given by the type predictor;

[0271] It is the similarity score retrieved from the index of type t.

[0272] Tiered weighting design:

[0273] The base weight w_t is set according to the importance and dimensions of the document type, and is in the form of a geometric sequence:

[0274]

[0275] in:

[0276] is the base weight, typically 1.0; r is the common ratio, recommended 1.2; level(t) is the document type level.

[0277] Specific configuration:

[0278] News (512 dimensions):

[0279] Webpage (768 dimensions):

[0280] Technical documentation (1024 dimensions):

[0281] Paper (2048 dimensions):

[0282] Patent (2560 dimensions):

[0283] Reasons for weight design:

[0284] 1. Higher dimensionality means greater weight: Higher-dimensional encoding contains richer information and should therefore receive higher weight;

[0285] 2. Proportional growth: This avoids excessively large or small weight differences caused by linear growth;

[0286] 3. Adjustability: The common ratio r can be adjusted according to the actual effect, with a recommended range of 1.1-1.3.

[0287] Normalization process:

[0288] To ensure comparability of similarity scores across different dimensions, normalization is performed before fusion:

[0289]

[0290] in:

[0291] It is the mean of the similarity scores in the type t index. That is the standard deviation.

[0292] These statistics are calculated and stored in advance during index building.

[0293] Example calculation:

[0294] Suppose the type prediction result for the query "deep learning model optimization method" is:

[0295] Technical documents: 0.65 confidence level; Papers: 0.25 confidence level; Web pages: 0.10 confidence level.

[0296] The retrieval scores of a candidate document across the three indexes are as follows:

[0297] tech_index score: 0.85 (normalized); paper_index score: 0.78 (normalized); web_index score: 0.62 (normalized).

[0298] The fusion score is:

[0299]

[0300] Step 7: Sorting and Truncation of Results

[0301] All candidate documents are sorted in descending order based on their fusion scores, and the Top-100 are selected as the initial recall results.

[0302] Deduplication: If the same document is retrieved in multiple indexes, the one with the highest fusion score is retained.

[0303] Innovation Point 4: Multi-scale Reordering Strategy

[0304] Core idea: For the initially recalled candidate results, multi-way reordering is performed using sub-encoders of different lengths from the original encoding to further improve accuracy. For example... Figure 5 A flowchart of the reordering process is provided.

[0305] Detailed implementation steps:

[0306] Step 1: Encoding Retrieval

[0307] For the 100 candidate documents initially recalled, their original 4096-dimensional encoded vectors are retrieved from the storage system.

[0308] Storage strategy:

[0309] The vector index stores the truncated encoding (e.g., 1024 dimensions) for fast retrieval.

[0310] The complete 4096-dimensional encoding is stored in a separate vector store (such as HBase or Milvus) for fine sorting.

[0311] Search optimization: Batch retrieval of encoded vectors reduces the number of I / O operations.

[0312] Step 2: Generate multi-scale query vectors

[0313] The 4096-dimensional code of the query is truncated into sub-codes of different lengths:

[0314]

[0315] Step 3: Calculate multi-scale similarity

[0316] For each candidate document d, calculate its similarity to the query across different dimensions:

[0317]

[0318] Calculate the four similarity scores:

[0319]

[0320] Step 4: Step-by-step weight fusion

[0321] A tiered weighting strategy similar to that used in the recall phase is employed, but the weight configuration differs:

[0322]

[0323] Weighting (geometric sequence, common ratio 1.5):

[0324]

[0325] Normalize the weights (to make the sum equal to 1):

[0326]

[0327] Reasons for weight design:

[0328] 1. High dimensionality dominates: The weight of 4096 dimensions is the largest (41.5%), ensuring accuracy.

[0329] 2. Low dimensionality provides diversity: Low dimensions still account for a certain proportion, avoiding overfitting.

[0330] 3. Gradual increase: This conforms to the principle of increasing information content.

[0331] Step 5: Final Sorting

[0332] Sort the results in descending order based on the reordering scores, and select the Top-20 as the final results.

[0333] Sorting strategy:

[0334] Main basis: Reordering score;

[0335] Secondary criterion: Document freshness (time decay factor);

[0336] Diversity: The MMR (Maximum Marginal Relevance) algorithm can be optionally applied to increase the diversity of results.

[0337] Example calculation:

[0338] The candidate document scored as follows in the multi-scale similarity calculation:

[0339]

[0340] Reordering score:

[0341] Performance optimization:

[0342] 1. Parallel computing: Multi-scale similarity calculations can be parallelized, using GPUs to accelerate batch dot product operations.

[0343] 2. Early stopping strategy: If the 512-dimensional similarity is already very low (e.g., <0.5), subsequent calculations can be skipped.

[0344] 3. Caching mechanism: Memory caching of the 4096-dimensional encoding of popular documents.

[0345] Performance analysis shows that the reordering stage typically results in a 5-10% improvement in accuracy (measured by NDCG@20). The reason is:

[0346] 1. Use full-dimensional encoding for accurate calculations.

[0347] 2. Multi-scale fusion captures semantic matching at different granularities.

[0348] 3. Computing on a smaller candidate set allows for higher computational costs.

[0349] The detailed process of the technical solution in the embodiments of this application is as follows:

[0350] (1) Offline index building process

[0351] Process description (text):

[0352] Step 1: Document Preprocessing

[0353] The system first reads the original document collection and performs standardized preprocessing. Preprocessing includes: removing HTML tags and special characters, unifying the encoding format to UTF-8, performing word segmentation and standardization, removing duplicate documents, and extracting structured information such as document titles, body text, and abstracts.

[0354] Step 2: Document Type Recognition

[0355] Each document is categorized using a pre-trained BERT classifier. The classifier analyzes multidimensional features of the document, including text length, vocabulary distribution, syntactic structure, and terminology density, and outputs probability distributions for five categories (news, webpage, technology, paper, and patent). The category with the highest probability is selected as the category label for that document.

[0356] Step 3: Differential Coding

[0357] Based on the identified document type, select the corresponding encoder for vectorization:

[0358] News documents use a news encoder to generate 4096-dimensional vectors.

[0359] Web page documents use a web page encoder to generate 4096-dimensional vectors.

[0360] Technical documents use a technical encoder to generate 4096-dimensional vectors.

[0361] Thesis documents use a thesis encoder to generate 4096-dimensional vectors.

[0362] Patent documents use a patent encoder to generate 4096-dimensional vectors.

[0363] All encoders share the same infrastructure (such as RoBERTa-large), but are fine-tuned for their respective data types to ensure encoding quality.

[0364] Step 4: Vector Truncation

[0365] Extract the appropriate length of nested encoding based on document type:

[0366] News category: Extract the first 512 dimensions and store them in news_index_512

[0367] For web pages: Extract the first 768 dimensions and store them in web_index_768.

[0368] Technical categories: Extract the first 1024 dimensions and store them in tech_index_1024.

[0369] For academic papers: Extract the first 2048 dimensions and store them in paper_index_2048.

[0370] Patents: Extract the first 2560 dimensions and store them in patent_index_2560.

[0371] The truncation operation retains only the first N dimensions of the vector and discards the subsequent dimensions, thus saving storage space.

[0372] Step 5: Index Insertion

[0373] Insert the truncated vector into the corresponding FAISS index. Each index maintains:

[0374] Vector data: an N×D floating-point matrix, where N is the number of documents and D is the dimension.

[0375] Document ID Mapping: A mapping table from index position to document ID.

[0376] Inverted index structure (for IVF indexes): Clustering information accelerates retrieval.

[0377] The index supports incremental addition, and new documents can be inserted in real time or in batches.

[0378] Step 6: Complete Encoding and Storage

[0379] Store the complete 4096-dimensional encoding of the document in a separate vector storage system (such as HBase, Milvus, or a distributed file system). The storage structure is as follows:

[0380] Key: Document ID

[0381] Value: 4096-dimensional floating-point vector (16KB) + metadata (type, timestamp, etc.)

[0382] These complete codes are used for precise calculations during the reordering phase.

[0383] Step 7: Metadata Management

[0384] Save metadata information for each document to a relational database or NoSQL store:

[0385] Document ID (primary key)

[0386] Document types (news / web pages / technology / papers / patents)

[0387] Original text (optional, for display purposes)

[0388] Structured fields such as title, author, and source

[0389] Timestamp (publication time, indexing time)

[0390] Access statistics (page views, click-through rate, etc.)

[0391] Metadata and vector indexes are stored separately to improve query flexibility.

[0392] Step 8: Index Optimization and Persistence

[0393] After all documents have been processed, the index is optimized:

[0394] Train the cluster centers of the IVF index (e.g., using k-means clustering).

[0395] Build an inverted list to speed up retrieval

[0396] Compress index data (optional, using methods such as PQ).

[0397] Persist the index to disk to ensure it can be recovered after a system restart.

[0398] The optimized index provides the best retrieval performance and storage efficiency.

[0399] (2) Process description in text:

[0400] Step 1: Query Preprocessing

[0401] After a user enters a query, the system first performs text preprocessing. Preprocessing operations include: removing invalid characters and punctuation marks from the query, standardizing capitalization (depending on language characteristics), word segmentation (using jieba for Chinese and spaces for English), removing stop words (optional, depending on query length), and query expansion (optional, adding synonyms or related words). The preprocessed query text is typically between 5 and 50 words long.

[0402] Step 2: Feature Extraction

[0403] The system extracts multidimensional features from queries for type prediction. The feature extraction module analyzes: the number of words and character length in the query to determine if it's a short or long query; the density of technical terms in the query to identify whether it's a domain-specific query; the type and number of named entities, such as names of people, places, and organizations; the language characteristics of the query, such as Chinese, English, or a mixture of languages; the time sensitivity of the query, such as whether it contains time-related words; and the type of query intent, such as information retrieval, question answering, or exploratory search. These features form a feature vector, which is then input into the type predictor.

[0404] Step 3: Document Type Prediction

[0405] A lightweight BERT classifier is used to predict the most likely document types related to the query. The classifier takes the query text and its feature vector as input and outputs a probability distribution of five document types. The system selects the top-3 types with the highest probabilities for subsequent retrieval and records the confidence score for each type. For example, type 1 "technical document" has a confidence score of 0.65, type 2 "paper" has a confidence score of 0.25, and type 3 "web page" has a confidence score of 0.10.

[0406] Confidence calibration ensures the reliability of predicted probabilities. If the highest confidence level exceeds 0.8, the system can retrieve only the Top-1 type to improve efficiency; if all confidence levels are low (maximum < 0.3), the system will retrieve all types or return suggestions to improve the query.

[0407] Step 4: Multipath Vector Encoding

[0408] For each document type predicted, the system uses the corresponding encoder to generate a query vector. The specific process is as follows:

[0409] For type 1 (technical document, 1024 dimensions): The query is input into the technical document encoder. The encoder extracts the semantic representation of the query, maps it to a 1024-dimensional space through a projection layer, and performs L2 normalization to obtain the query vector q1.

[0410] For type 2 (paper, 2048 dimensions): Input the query into the paper encoder, and similarly generate a 2048-dimensional query vector q2.

[0411] For type 3 (web page, 768 dimensions): Input the query into the web page encoder to generate a 768-dimensional query vector q3.

[0412] Different encoders share the same underlying parameters, but differ in projection layers and fine-tuning parameters, ensuring the quality of each type of encoding.

[0413] Step 5: Parallel Index Retrieval

[0414] The system performs approximate nearest neighbor retrieval in parallel within the index corresponding to the predicted document type. Specifically:

[0415] In tech_index_1024, using query vector q1 for retrieval, FAISS performs vector similarity calculation and returns the Top-100 most similar candidate documents and their similarity scores.

[0416] In paper_index_2048, using query vector q2 for retrieval also returns Top-100 candidates.

[0417] In web_index_768, using query vector q3, the Top-100 candidates are returned.

[0418] Three retrieval tasks are executed in parallel, leveraging multithreading or asynchronous I / O to improve efficiency. The retrieval latency for each index is typically between 10 and 30 milliseconds.

[0419] Step 6: Merging and Deduplicating Results

[0420] The candidate documents returned by the three indexes are merged into the result pool. The result pool contains a maximum of 300 candidates (3 × 100), but the actual number may be less because:

[0421] The same document may be retrieved in multiple indexes (e.g., a technical paper appears in both tech_index and paper_index). For duplicate documents, the system retains the record with the highest similarity score.

[0422] The deduplicated result pool typically contains 150-250 unique documents.

[0423] Step 7: Step-by-step weight fusion

[0424] For each candidate document in the results pool, the system calculates its fusion score. The fusion formula is:

[0425] Fusion score = (weight 1 × confidence 1 × similarity 1) + (weight 2 × confidence 2 × similarity 2) + (weight 3 × confidence 3 × similarity 3)

[0426] The weights are designed in a tiered manner, set according to document dimensions and types. For example:

[0427] The base weight for the technical documentation (1024 dimensions) is 1.44.

[0428] The basic weight of the paper (2048 dimensions) is 1.728.

[0429] The basic weight of the webpage (768 dimensions) is 1.2.

[0430] The confidence score comes from the output of the type predictor (e.g., 0.65, 0.25, 0.10), and the similarity score comes from the raw score of the index retrieval (after normalization).

[0431] The fusion score comprehensively considers the relevance of document types, the matching degree of retrieval, and the expressive power of dimensions, achieving a scientific integration of multiple results.

[0432] Step 8: Preliminary Sorting

[0433] All documents in the result pool are sorted in descending order based on the fusion score, and the Top-100 are selected as the initial recall results. These 100 documents are the most likely candidates to be relevant to the query and will enter the re-ranking stage for further refinement.

[0434] Step 9: Obtain the complete encoding

[0435] For the Top-100 candidate documents, the system retrieves their complete 4096-dimensional encoded vectors in batches from the vector storage system. The batch retrieval operation is optimized in the following ways:

[0436] Using the batch read interface, the encoding of multiple documents can be retrieved in a single request, reducing the number of network round trips.

[0437] Cache the code for popular documents in memory; the hit rate can typically reach 30-50%.

[0438] Asynchronous I / O operations perform other calculations while retrieving the encoding.

[0439] The retrieval latency for a fully encoded file is typically between 20 and 50 milliseconds.

[0440] Step 10: Multi-scale reordering

[0441] The system uses complete 4096-dimensional encoding for multi-scale fine sorting. The specific process is as follows:

[0442] First, the 4096-dimensional code of the query is truncated into four sub-codes of different lengths: 512-dimensional, 1024-dimensional, 2048-dimensional, and 4096-dimensional.

[0443] Then, for each candidate document, its cosine similarity to the query is calculated in four dimensions, resulting in four similarity scores.

[0444] Next, the four similarity scores were weighted and summed using a step-wise weighting method, with the following weighting configurations: 512 dimensions accounted for 12.3%, 1024 dimensions accounted for 18.5%, 2048 dimensions accounted for 27.7%, and 4096 dimensions accounted for 41.5%.

[0445] Finally, the candidate rankings are sorted according to the re-ranking scores to obtain a refined ranking.

[0446] The reordering stage makes full use of the expressive power of high-dimensional encoding and improves retrieval accuracy through precise calculation.

[0447] Step 11: Final Result Return

[0448] The system selects the reordered Top-20 documents as the final result and returns it to the user. The returned information includes: document ID and title, relevance score (normalized to 0-1), document summary or excerpt, document type tag, and metadata (author, time, source, etc.). The system can also apply diversity optimization (such as the MMR algorithm) to increase the diversity of results while maintaining relevance, avoiding excessive similarity among the Top results.

[0449] The total latency of the entire retrieval process, from query input to result return, is typically between 60 and 100 milliseconds, meeting the real-time requirements of online services.

[0450] (1) Experimental setup

[0451] Dataset Size: This invention has been fully tested on a large-scale mixed document library, and the dataset includes Table 1:

[0452] Table 1

[0453]

[0454] The dataset spans from 2018 to 2024 and covers both Chinese and English languages ​​to ensure the representativeness and generalizability of the experiments.

[0455] Evaluation indicators:

[0456] System performance is evaluated from three dimensions:

[0457] 1. Search quality indicators:

[0458] Recall@10: Recall rate of relevant documents in the Top-10 results;

[0459] Recall@100: The recall rate of relevant documents in the Top-100 results;

[0460] MRR@10: Mean Reciprocal Rank;

[0461] NDCG@10: Normalized loss cumulative gain, a quality metric considering sorting position;

[0462] MAP: Mean Average Precision.

[0463] 2. Cost efficiency indicators:

[0464] Total storage space (GB): Total storage requirements for vector indexing and complete encoding;

[0465] Average storage per document (KB): The average storage cost per document;

[0466] Memory usage (GB): The memory requirements of the system during runtime;

[0467] Index build time (hours): Total time spent building the offline index.

[0468] 3. Service performance indicators:

[0469] Average query latency (ms): The average time from query input to result return;

[0470] P50 / P95 / P99 delay (ms): percentile of the delay distribution;

[0471] QPS (queries per second): The system's concurrent query processing capability;

[0472] Cache hit rate (%): The hit rate of multi-level cache.

[0473] Baseline method:

[0474] Compare with the following four baseline methods:

[0475] 1. Uniform-768: All documents use 768-dimensional BERT encoding, representing the standard scheme.

[0476] 2. Uniform-2048: All documents use 2048-dimensional high-quality encoding, representing a high-precision scheme.

[0477] 3. Uniform-4096: All documents use 4096-dimensional ultra-high-dimensional encoding, representing a resource-sufficient solution.

[0478] 4. PQ-Compressed: This compression scheme uses 768-dimensional encoding and product quantization.

[0479] Experimental environment:

[0480] Hardware configuration:

[0481] CPU: Intel Xeon Gold 6248R (24 cores, 48 ​​threads) × 2

[0482] GPU: NVIDIA A800 80GB × 8

[0483] Memory: 1024GB DDR4

[0484] Storage: 10TB NVMe SSD

[0485] Software environment:

[0486] Operating System: Ubuntu 20.04 LTS

[0487] Python version: 3.9.12

[0488] PyTorch version: 2.0.1

[0489] FAISS version: 1.7.4

[0490] Transformers version: 4.30.2

[0491] (2) Comparison of search quality

[0492] The main results are shown in Table 2:

[0493] Table 2

[0494]

[0495] Results analysis:

[0496] The method of this invention achieves optimal or near-optimal levels across all key indicators:

[0497] Compared to Uniform-768: Recall@10 improved by 16.0% (from 0.512 to 0.594), and NDCG@10 improved by 17.1% (from 0.445 to 0.521), significantly outperforming the standard solution.

[0498] Compared to Uniform-2048: the retrieval quality is comparable (Recall@10 is only 1.2% lower), but the storage cost is reduced by 44% (see subsequent chapters), achieving an optimal balance between quality and cost.

[0499] Compared to Uniform-4096, Recall@10 is only 1.2% lower (0.594 vs 0.601), but storage costs are reduced by 73%, proving that similar results can be achieved with fewer resources through differentiated allocation.

[0500] Compared to PQ-Compressed: Recall@10 is 22.5% higher (0.594 vs 0.485), proving that this method maintains high quality while saving storage, and is superior to simple compression schemes.

[0501] Performance analysis by type:

[0502] The retrieval performance of different document types is shown in Table 3:

[0503] Table 3

[0504]

[0505] Key findings:

[0506] 1. Performance improvement for simple documents: Simple documents such as news articles and web pages actually perform better when using low-dimensional encoding (512-768 dimensions), with a 2-8% improvement compared to 768 dimensions. This is because it avoids overfitting in high-dimensional spaces, and low-dimensional spaces are more suitable for simple semantic expressions.

[0507] 2. Complex documents benefit significantly: Academic papers and patent documents benefit the most from high-dimensional encoding, with an improvement of 18-21% compared to 768 dimensions. Complex documents contain rich, multi-layered semantics, which require higher dimensions to fully express them.

[0508] 3. Significant advantages of adaptive allocation: By assigning the most suitable dimensions to different types of documents, the overall performance of the system is better than any fixed-dimensional solution, achieving the goal of "allocation on demand, each getting what they need".

[0509] (3) Comparison of storage costs

[0510] The total storage space comparison (25 million documents) is shown in Table 4:

[0511] Table 4

[0512]

[0513] illustrate:

[0514] The Uniform scheme only stores truncated encodings and does not save complete encodings;

[0515] The PQ scheme uses a 96-byte product quantization representation (768 dimensions → 96 bytes), which minimizes storage but results in significant quality loss.

[0516] The method of this invention stores differentiated nested codes (average 1331 dimensions) for recall, and stores the complete 4096 dimensions for reordering;

[0517] It saves 22% of storage compared to Uniform-768, 68% compared to Uniform-2048, and 84% compared to Uniform-4096.

[0518] Storage space distribution details (method of this invention):

[0519] Table 5 shows the storage details for each document type:

[0520] Table 5

[0521]

[0522] Note: The total storage for the complete encoding is 100GB (instead of 125GB) because it uses mixed precision storage (FP16) and compression technology.

[0523] Average storage per document:

[0524] The method of this invention: 170GB / 25 million = 6.8 KB / document;

[0525] Uniform-768: 217GB / 25 million = 8.7 KB / document (28% high);

[0526] Uniform-2048: 537GB / 25 million = 21.5 KB / document (up 216%);

[0527] PQ solution: 49GB / 25 million = 2.0 KB / document (70% lower, but poorer quality).

[0528] Cost-benefit analysis:

[0529] Assuming cloud storage costs $0.02 per GB per month and index memory costs $1 per GB per month:

[0530] The method of this invention saves $4,792 (76%) in annualized cost compared to Uniform-2048, demonstrating significant economic benefits.

[0531] (4) Comparison of retrieval delay

[0532] Analysis conclusion:

[0533] Slightly higher than Uniform-768: This method has a latency of 63ms, which is 31% higher than Uniform-768's 48ms, mainly due to the additional overhead of multi-way retrieval and reordering. However, the retrieval quality is improved by 16%, making this performance-latency trade-off worthwhile.

[0534] Significantly faster than Uniform-2048: latency reduced by 53% (from 135ms to 63ms) because the use of low-dimensional indexes (average 1331 dimensions vs 2048 dimensions) reduces vector computation.

[0535] Significantly improved compared to Uniform-4096: latency reduced by 78% (from 287ms to 63ms), demonstrating the importance of differentiated dimension allocation for efficiency improvement.

[0536] Acceptable online latency: The average latency of 63ms and the P95 latency of 102ms fully meet the real-time requirements of online retrieval services (typically <200ms).

[0537] Concurrency performance test:

[0538] Key observations:

[0539] The system maintains good performance even with 100 concurrent users (QPS=980, P95 latency=198ms).

[0540] 200 concurrent requests is the optimal operating point for the system (QPS=1250, CPU utilization 95%).

[0541] Performance degrades significantly with 500 concurrent users; it is recommended to distribute the load across multiple servers using load balancing.

[0542] Delay optimization effect:

[0543] The system latency has been reduced by 45% compared to the initial version through the following optimization techniques:

[0544] Multi-level caching: Query result cache hit rate of 30%, vector cache hit rate of 50%, avoiding duplicate calculations.

[0545] Parallel retrieval: Parallelization of multi-index retrieval, latency changes from cumulative to maximum.

[0546] Batch processing: Retrieve complete codes in batches to reduce I / O operations.

[0547] SIMD acceleration: Accelerating vector computation using the AVX512 instruction set.

[0548] GPU inference: Encoder inference is accelerated using GPU, resulting in a 3x increase in throughput.

[0549] (5) Ablation test

[0550] To verify the contribution of each technical component, a systematic ablation experiment was conducted.

[0551] Key findings:

[0552] Document type identification is crucial: removing type classification resulted in a 3.0% performance decrease, demonstrating the importance of accurate document type identification for retrieval quality.

[0553] Reordering provides improved accuracy: The reordering stage contributes a 2.0% performance improvement, although it increases storage costs (+25GB), but this is worthwhile for high-precision scenarios.

[0554] Fine-grained classification is superior: using 5 types improves performance by 1.2% compared to 3 types, and finer-grained classification can more accurately match document characteristics.

[0555] The tiered weighting is effective: Compared with uniform weighting, the tiered weighting improves the efficiency by 0.8%, proving that the fusion strategy that takes into account dimensional differences is more scientific.

[0556] Differentiated encoders are essential: type-specific encoders outperform single encoders by 4.4%, and dedicated encoders are better able to capture the characteristics of various document types.

[0557] Comparison of Dimension Allocation Strategies:

[0558] Conclusion: The clustering + information entropy method of this invention is the optimal dimension allocation strategy, which scientifically quantifies the expression requirements of various types of documents.

[0559] (6) Scalability testing

[0560] Document size expansion:

[0561] Extensibility analysis:

[0562] The index building time is approximately linearly related to the number of documents, the index size grows strictly linearly (approximately 6.8GB per million documents), the retrieval latency grows logarithmically, remaining within 100ms even with 100 million documents, and memory usage grows linearly. Larger scales can be supported through sharding deployment.

[0563] Distributed deployment effect:

[0564] The deployment was carried out using 4 servers (6.25 million documents per server). The sharding strategy effectively reduced the load on a single machine and improved the overall throughput of the system.

[0565] (7) Practical application cases

[0566] Case Study 1: Enterprise Document Search System

[0567] A large technology company deployed the method of this invention for internal document retrieval:

[0568] Document size: 12 million (40% technical documents, 30% emails, 20% meeting minutes, and 10% others).

[0569] Number of users: 5000+ employees.

[0570] Queries: Average QPS = 50, Peak QPS = 200.

[0571] Deployment results:

[0572] Compared to the original Elasticsearch full-text search, the recall rate of relevant documents has been improved by 32%.

[0573] The query response time was reduced from 200ms to 65ms.

[0574] Storage costs are reduced by 60% (compared to the original plan of a unified 2048-dimensional solution).

[0575] User satisfaction rating improved from 3.2 / 5 to 4.5 / 5.

[0576] Case Study 2: Academic Paper Retrieval Platform

[0577] An academic search engine integrates the method of this invention:

[0578] Document size: 8 million academic papers.

[0579] Query characteristics: Highly specialized, with a high proportion of long queries.

[0580] Users served: Researchers worldwide.

[0581] Deployment results:

[0582] For complex academic queries, the recall rate of relevant papers increased by 25%.

[0583] The P95 query latency was reduced from 180ms to 95ms.

[0584] Monthly cloud service cost savings of $3,500.

[0585] User retention rate increased by 18%.

[0586] Case Study 3: Patent Search System

[0587] An intellectual property agency used this method to build a patent search system:

[0588] Document size: 6 million patent documents (multilingual).

[0589] Search characteristics: Contains a high concentration of legal terminology and requires extremely high precision.

[0590] Application scenarios: patent search, infringement analysis.

[0591] Deployment results:

[0592] For patent searches, the accuracy of the top 20 results improved by 41%.

[0593] It supports multilingual mixed search, improving cross-language recall rate by 35%.

[0594] The search speed is increased by 3 times, meeting the needs of real-time retrieval.

[0595] It helps improve review efficiency by 40%.

[0596] Service Responsibilities Description:

[0597] Query service: Responsible for receiving user requests, coordinating various sub-services, and returning the final result. Supports horizontal scaling by distributing requests through a load balancer.

[0598] Type prediction service: Dedicated to document type prediction, it uses GPUs for BERT inference and supports batch prediction to improve throughput.

[0599] Encoding service: Responsible for vector encoding of queries and documents, using GPU acceleration, and supporting parallel inference of multiple encoders.

[0600] Search service: Manages FAISS vector indexes, performs ANN searches, and supports index sharding and distributed deployment.

[0601] Reordering service: Responsible for multi-scale reordering, obtaining complete encoding from vector storage, and calculating accurate similarity.

[0602] Vector storage: Stores the complete 4096-dimensional encoding of documents, supports efficient batch queries, and can use Milvus, HBase, or self-built storage.

[0603] Metadata service: Stores metadata information for documents, including title, author, timestamp, etc., and supports filtering and sorting.

[0604] Monitoring system: Monitors the performance metrics of each service in real time, including QPS, latency, error rate, resource usage, etc., and supports alerts and visualization.

[0605] The index building process specifically includes:

[0606] Detailed steps for building an offline index:

[0607] Step 1: Data Preprocessing

[0608] Read raw documents from data sources (databases, file systems, APIs, etc.). Clean each document: remove HTML tags, special characters, and control characters; encode uniformly to UTF-8; extract structured information (title, body, summary, etc.); perform deduplication, identifying approximately duplicate documents using SimHash or MinHash algorithms.

[0609] The preprocessed document is stored in JSON format and contains fields such as document ID, text content, and metadata.

[0610] Step 2: Batch document categorization

[0611] The trained BERT classifier is used to identify document types. To improve efficiency, a batch processing strategy is adopted: 1000 documents are processed per batch, and GPU is used to accelerate inference, with an inference throughput of approximately 5000 documents / second.

[0612] The classifier outputs probability distributions for five categories, selecting the one with the highest probability as the document type. Confidence scores are also recorded for subsequent quality monitoring.

[0613] For documents with low confidence levels (e.g., highest probability < 0.5), mark them as "requires manual review" and submit them for manual review or process them using the default type.

[0614] Step 3: Differential Coding

[0615] Based on the document type, select the appropriate encoder for vectorization. The encoding process is as follows:

[0616] For news documents, a news encoder is used. The encoder reads the document text (up to 512 tokens), extracts semantic representations through a Transformer layer, extracts the hidden state of the [CLS] tag (1024 dimensions), maps it to 4096 dimensions through a linear projection layer, and performs L2 normalization.

[0617] For web-based documents, a web encoder is used, and the process is similar. For technical, academic, and patent documents, the corresponding encoders are used respectively.

[0618] The batch size is dynamically adjusted based on GPU memory, typically ranging from 128 to 256 documents per batch. The encoding throughput is approximately 3000 documents per second (using an A100 GPU).

[0619] Step 4: Vector Truncation and Index Insertion

[0620] Based on the document type, extract the corresponding prefix from the 4096-dimensional encoding:

[0621] News category: Take the first 512 dimensions and prepare to insert news_index_512.

[0622] Web page type: Take the first 768 dimensions, and prepare to insert web_index_768.

[0623] Technical category: Take the first 1024 dimensions and prepare to insert tech_index_1024.

[0624] Paper-based: Take the first 2048 dimensions, preparing to insert paper_index_2048.

[0625] Patents: Take the first 2560 dimensions, and prepare to insert patent_index_2560.

[0626] The truncated vectors are then batch-inserted into their corresponding FAISS indices. The insertion operation includes:

[0627] Add vectors to the index's vector matrix (using the add method); record the mapping relationship between document IDs and index positions; update the index's metadata information.

[0628] For large-scale indexes (such as those in the tens of millions), it is recommended to use the IndexIVFFlat or IndexIVFPQ structure, perform clustering training first, and then add vectors.

[0629] Step 5: Complete Encoding and Storage

[0630] The complete 4096-dimensional encoding of the document is stored in a vector storage system. Storage methods include:

[0631] Using Milvus: Milvus is a specialized vector database that supports efficient vector storage and retrieval. Create a collection, specifying the vector dimension as 4096, and insert the vectors and their metadata.

[0632] Using HBase: HBase is a distributed NoSQL database suitable for large-scale data storage. It uses document IDs as rowkeys and vector data as column values, supporting batch reads and writes.

[0633] Using self-built storage: You can use embedded databases such as LevelDB and RocksDB, or use a distributed file system (such as HDFS) to store serialized vector files.

[0634] The recommended storage format is FP16 (half-precision floating point), which saves 50% of space compared to FP32, and the precision loss is negligible.

[0635] Step 6: Metadata Management

[0636] Store document metadata in a relational database (such as MySQL or PostgreSQL) or a document database (such as MongoDB). Example metadata table structure:

[0637] doc_id (primary key, string): Unique identifier for the document

[0638] doc_type (string): Document type (news / web / tech / paper / patent)

[0639] title (text): Document title

[0640] author(string): author

[0641] source (string): source

[0642] publish_time (timestamp): Publication time

[0643] index_time (timestamp): Index time

[0644] text_length (integer): Length of the text

[0645] confidence (floating-point number): Classification confidence level

[0646] extra_fields (JSON): Other extended fields

[0647] Metadata is used for functions such as result display, filtering, and sorting.

[0648] Step 7: Index Optimization

[0649] After all documents have been processed, the FAISS index is optimized:

[0650] For indexes using the IVF structure, perform cluster training. Use the k-means algorithm to cluster the vectors into several clusters (the recommended number is sqrt(N), where N is the number of documents). Construct an inverted index, where each cluster contains a list of its member vectors.

[0651] To persist the index, the index data in memory is written to a disk file. FAISS provides the `write_index` function, which can save the index to a binary file.

[0652] Optionally, product quantization (PQ) can be used to further compress the index, reducing memory usage while maintaining retrieval quality.

[0653] Step 8: Quality Verification

[0654] After the index is built, perform quality verification:

[0655] Randomly select 1000 documents and use their text as a query to check if the documents themselves are in the Top-10 results (self-retrieval test). Calculate the average rank and recall rate to evaluate the index quality.

[0656] Search performance was evaluated using standard test sets (such as MS MARCO and Natural Questions), and metrics such as Recall@10 and MRR@10 were calculated. Figure 6 A comparison chart of retrieval performance is provided.

[0657] Check the physical integrity of the index to ensure all documents are correctly indexed without loss or corruption. If verification fails, troubleshoot the issue (such as encoder errors, index building errors, etc.), fix it, and rebuild the index.

[0658] In one embodiment, an electronic device, which may be a computer, is provided. The electronic device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device is used for cross-document type data retrieval based on differential nested encoding. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a cross-document type retrieval method based on differential nested encoding.

[0659] In one embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the cross-document type retrieval method based on differential nested encoding described above.

[0660] In one embodiment of this application, a computer program product is provided, including a computer program / instructions, which, when executed by a processor, implements the steps of the cross-document type retrieval method based on differential nested encoding described above.

[0661] The computer-readable storage medium and computer program product provided in this embodiment are similar in implementation principle and technical effect to the above method embodiments, and will not be repeated here.

[0662] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0663] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0664] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A cross-document type retrieval method based on differentiated nested encoding, characterized in that, The method comprises six steps: document preprocessing, document type identification, differential encoding, index construction and optimization, online retrieval, and result fusion. Specifically, it includes: Document preprocessing: Read the original document collection and perform standardized preprocessing on the documents, including removing special characters, unifying the encoding format, word segmentation, removing duplicate documents, and extracting the structured information of the documents; Document type identification: The trained BERT classifier is used to identify the type of the document, analyze the multidimensional features of the document, output the probability distribution of the document's type, and select the type with the highest probability as the document's category label; Differential encoding: Based on the document type, select the corresponding encoder to vectorize the document and generate a 4096-dimensional original encoded vector; Index building and optimization: Based on the document type, the corresponding prefix of the original encoding vector is extracted as a nested encoding, which is then inserted into the corresponding type of FAISS index. The complete 4096-dimensional encoding of the document is stored in an independent vector storage system to optimize the index. Online retrieval: Receives user queries, preprocesses and extracts features from the queries, uses the BERT classifier to predict the most likely document type related to the query, generates query vectors for the predicted document types using the corresponding encoder, performs approximate nearest neighbor retrieval in parallel in the corresponding index, and returns candidate documents and their similarity scores; Result fusion: The candidate documents returned by different indexes are merged into the result pool. For duplicate documents, the record with the highest similarity score is retained. Based on the confidence score predicted by the document type and the original similarity score retrieved, a step-weighted fusion strategy is used to calculate the fusion score, the candidate documents are sorted, and the sorted retrieval results are returned. The differential coding step specifically includes: Establish a mapping relationship between document types and encoding dimensions, and assign nested encodings of different lengths to different types of documents; A nested encoding structure is designed to ensure that all encoder-generated vectors adopt a nested structure, wherein the high-dimensional vector contains all the information of the low-dimensional vector. The encoders include a news encoder, a web page encoder, a technical document encoder, a thesis encoder, and a patent encoder, each of which is optimized for its corresponding document type. The 4096-dimensional original encoded vector is mapped through a linear projection layer and then subjected to L2 normalization. Based on the document type, only the corresponding length of encoding is stored in the index; specifically, documents identified as news are encoded with 512 dimensions to news_index_512, documents identified as web pages are encoded with 768 dimensions to web_index_768, documents identified as technology documents are encoded with 1024 dimensions to tech_index_1024, documents identified as papers are encoded with 2048 dimensions to paper_index_2048, and documents identified as patents are encoded with 2560 dimensions to patent_index_2560.

2. The method according to claim 1, characterized in that, The differential encoding step further includes adaptive multi-granularity text embedding representation learning based on gradient optimization, specifically including: During the initialization phase, a pre-trained large language model, Qwen3-8b, is used as the base encoder. A learnable linear projection layer is added on top of the encoder to project the hidden state dimension onto the target dimension of 4096. Xavier uniform initialization or normal distribution initialization is used. Training is then performed, specifically including: In the first phase, the focus is on optimizing the representation quality of the first 512 dimensions; In the second phase, the optimization scope is expanded to the first 768 dimensions while maintaining the performance of the first 512 dimensions. In the third stage, the optimization scope is expanded to the first 1024 dimensions; In the fourth stage, the optimization scope is expanded to the first 2048 dimensions, with a focus on academic paper documents. In the final stage, the complete 2560-dimensional representation is optimized, mainly for patent documents; In each training phase, the gradient is backpropagated through all 4096 dimensions of parameters, and the loss function is set to focus only on the dimension range of the current optimization.

3. The method according to claim 1, characterized in that, The online retrieval and result fusion steps include: Query preprocessing involves standardizing the query text input by the user, including text cleaning, case normalization, word segmentation, stop word filtering, and query expansion. Query feature extraction: Extract multidimensional features from the query for subsequent type prediction. Document type prediction: Use a trained BERT classifier to predict the document type most likely to be relevant to the query. Multi-path vector encoding generates a query vector of the corresponding dimension for each document type being predicted; Parallel index retrieval performs approximate nearest neighbor retrieval in parallel within the index corresponding to the predicted document type; A tiered weighted fusion strategy is employed, which considers the predicted confidence of document type and the original similarity score of the retrieval to scientifically integrate results from multiple sources. All candidate documents are sorted in descending order based on the fusion score, and the Top-100 are selected as the initial recall results. The query preprocessing includes removing invalid characters, unifying case, word segmentation, removing stop words, and query expansion. The word segmentation process selects different word segmentation tools based on the language type of the document; jieba is used for Chinese and space is used for English. The feature extraction includes extracting the number of words in the query, character length, frequency of professional terms, distribution of named entity types, query intent type, and domain features; The query intent types include information search, question answering, and exploratory search.

4. The method according to claim 1, characterized in that, The result fusion step further includes a multi-scale re-ranking strategy, specifically including the following steps: Encoding retrieval involves retrieving the original 4096-dimensional encoding vectors from the storage system for the 100 initially recalled candidate documents. The vector index stores the truncated encoding for fast recall, while a separate vector store holds the complete 4096-dimensional encoding for fine ranking. Retrieval optimization includes batch retrieval of encoding vectors to reduce I / O operations. Generate a multi-scale query vector, and truncate the 4096-dimensional code of the query into sub-codes of different lengths; Calculate multi-scale similarity: for each candidate document, calculate its similarity to the query across different dimensions; The tiered weight fusion adopts a tiered weight strategy similar to that of the recall phase, but with different weight configurations. The final sorting is performed in descending order based on the reordering scores, and the Top-20 are selected as the final return results. The multi-scale reordering strategy utilizes sub-encoders of different lengths from the original encoding for multi-way reordering.

5. The method according to claim 1, characterized in that, In the document preprocessing step, the structured information includes the document's title, body, abstract, author, source, publication time, document length, and language type. The process of removing duplicate documents involves identifying and deduplicating approximately duplicate documents using the SimHash or MinHash algorithm. The preprocessed document is stored in JSON format and contains document ID, text content, and metadata fields.

6. The method according to claim 1, characterized in that, In the result fusion step, the step-weighted fusion strategy dynamically adjusts the weights according to the document type dimension and confidence level; wherein, the weights adopt the form of a geometric sequence, and the weight values ​​increase step by step according to the document type dimension. In calculating the fusion score using a tiered weighted fusion strategy, the formula for calculating the fusion score is as follows: ， Where d is the candidate document, T is the set of predicted document types, and w t It is the basic weight of document type, confidence. t It is the confidence score given by the type predictor. It is the similarity score retrieved from the index of type t.

7. A cross-document type retrieval system based on differentiated nested encoding, characterized in that, The system includes: The document preprocessing module is used to read the original document collection and perform standardized preprocessing on the documents, including removing special characters, unifying the encoding format, word segmentation, removing duplicate documents, and extracting the structured information of the documents; The document type recognition module is used to identify the type of a document using a trained BERT classifier, analyze the multidimensional features of the document, output the probability distribution of the document's type, and select the type with the highest probability as the document's category label. The differential encoding module is used to select the corresponding encoder to vectorize the document based on the document type, generating a 4096-dimensional original encoded vector; The index building and optimization module is used to extract the corresponding prefix of the original encoding vector as a nested encoding according to the document type, insert it into the corresponding type of FAISS index, and store the complete 4096-dimensional encoding of the document in an independent vector storage system to optimize the index. The online retrieval module is used to receive user queries, preprocess and extract features from the queries, use the BERT classifier to predict the most likely document type related to the query, use the corresponding encoder to generate query vectors for the predicted document types, and perform approximate nearest neighbor retrieval in parallel in the corresponding index, and return candidate documents and their similarity scores. The result fusion module is used to merge candidate documents returned by different indexes into the result pool, retain the record with the highest similarity score for duplicate documents, calculate the fusion score based on the confidence score predicted by the document type and the original similarity score retrieved, sort the candidate documents, and return the sorted search results. The differential coding module specifically includes: Establish a mapping relationship between document types and encoding dimensions, and assign nested encodings of different lengths to different types of documents; A nested encoding structure is designed to ensure that all encoder-generated vectors adopt a nested structure, wherein the high-dimensional vector contains all the information of the low-dimensional vector. The encoders include a news encoder, a web page encoder, a technical document encoder, a thesis encoder, and a patent encoder, each of which is optimized for its corresponding document type. The 4096-dimensional original encoded vector is mapped through a linear projection layer and then subjected to L2 normalization. Based on the document type, only the corresponding length of encoding is stored in the index; specifically, documents identified as news are encoded with 512 dimensions to news_index_512, documents identified as web pages are encoded with 768 dimensions to web_index_768, documents identified as technology documents are encoded with 1024 dimensions to tech_index_1024, documents identified as papers are encoded with 2048 dimensions to paper_index_2048, and documents identified as patents are encoded with 2560 dimensions to patent_index_2560.

8. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program that, when executed by the processor, implements the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 6.

Images