Knowledge document search engine based on word vectors and context summary

Abstract

This application relates to the field of document search, specifically disclosing a knowledge document search engine based on word vectors and contextual summarization. First, it performs multimodal parsing on the original heterogeneous documents, constructing a keyword weight index by calculating the multidimensional features of the documents, and then using contextual summarization generation technology to construct a summary vector index. In the retrieval stage, the core lies in introducing a query understanding and dynamic weight generation mechanism. By extracting the linguistic features of user queries to deeply analyze query intent (such as precise search or concept exploration), it predicts and generates dynamic fusion weights in real time. This mechanism can adaptively adjust the weight of semantic recall and keyword recall in the final ranking according to the query type, effectively solving the one-size-fits-all defect of traditional static fusion strategies. Finally, it combines a local large language model to generate accurate question answers, significantly improving the recall and precision of searches in complex scenarios.

Description

Technical Field

[0001] This application relates to the field of document search, and more specifically, to a knowledge document search engine based on word vectors and contextual summarization. Background Technology

[0002] With the rapid development of information technology, enterprises and organizations have accumulated massive amounts of heterogeneous knowledge documents. These documents not only include regular Word, PDF, and spreadsheet files, but also involve a large amount of unstructured data such as images and presentations. How to quickly and accurately retrieve the required information from this diverse and complex data, and to conduct effective knowledge management and question-and-answer generation, has become a significant challenge facing the field of information technology.

[0003] Existing search engine technologies are mainly divided into traditional retrieval based on keyword matching and vector retrieval based on semantic understanding. Traditional algorithms such as TF-IDF calculate relevance by statistically analyzing word frequencies, but they rely on literal matching and cannot deeply understand the semantics of words. For example, when searching for AI assistants, they struggle to recall documents that only contain the word "AI Agent." While semantic retrieval based on word vectors can capture semantic connections, when processing long documents, key information in specific areas is easily diluted by a large amount of irrelevant content, resulting in low overall vector similarity and affecting ranking performance. To assist retrieval, existing technologies often introduce algorithms such as TextRank to extract keywords. However, traditional TextRank relies solely on the topological structure of the word co-occurrence graph and does not fully consider multi-dimensional features such as average information entropy, part-of-speech, and position in the document. This leads to inaccurate keyword weight calculations and an inability to reflect the core of the document. More importantly, to balance the accuracy of keyword matching with the generalization ability of semantic retrieval, existing technologies typically employ hybrid retrieval strategies. However, this fusion mechanism often uses static, fixed weight coefficients to combine the scores of both, lacking dynamic perception of the user's query intent. In reality, users' query intentions are diverse: for precise queries targeting specific terms or models, keyword matching should take precedence; while for conceptual queries about open-ended questions, semantic retrieval is more important. Because existing technologies ignore the mapping relationship between query features such as query length, entity word ratio, and syntactic structure, and fusion strategies, this one-size-fits-all static weight allocation cannot dynamically switch according to query intention. This leads to irrelevant results being introduced due to excessively high semantic weights in precise queries, or semantically relevant documents being missed due to excessively high keyword weights in conceptual queries, making it difficult to simultaneously meet the requirements of high precision and high recall.

[0004] Therefore, building a search engine that can combine multi-dimensional keyword features with dynamic query intent perception is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] To address the aforementioned technical challenges, this application is proposed. According to this application, a knowledge document search engine based on word vectors and contextual summarization includes: an original knowledge document set acquisition module, used to acquire the original knowledge document set.

[0006] The multimodal parsing and index building module is used to perform multimodal document parsing on the original knowledge document set to obtain the summary vector index and keyword weight index.

[0007] The query understanding module is used to analyze the query intent and generate dynamic weights for the acquired user queries to obtain query vectors, query terms, and dynamic fusion weights.

[0008] The dual-path recall module is used to perform parallel recall of query vectors and query terms based on the summary vector index and the keyword weight index to obtain semantic recall candidate sets and keyword recall candidate sets.

[0009] The fusion and re-ranking module is used to dynamically fuse and re-rank the semantic recall candidate set and the keyword recall candidate set based on dynamic fusion weights to obtain the final ranked document list.

[0010] The local question-and-answer generation module is used to process the final sorted document list and user query through a local question-and-answer large language model to obtain the final search results, which include direct answers and cited sources.

[0011] Compared with existing technologies, this application provides a knowledge document search engine based on word vectors and contextual summarization, addressing the technical problems of rigid weight allocation in existing hybrid retrieval technologies, which cannot adapt to changing query intentions, and the dilution of key information caused by vectorization of long documents. First, multimodal parsing is performed on the original heterogeneous documents. A keyword weight index is constructed by calculating the multidimensional features of the documents, and a summary vector index is built using contextual summarization technology, thus balancing the detailed accuracy and core semantics of the documents at the index layer. In the retrieval stage, the core lies in introducing a query understanding and dynamic weight generation mechanism. By extracting the linguistic features of user queries to deeply analyze query intentions (such as precise search or concept exploration), dynamic fusion weights are predicted and generated in real time. This mechanism can adaptively adjust the proportion of semantic recall and keyword recall in the final ranking according to the query type, effectively solving the one-size-fits-all defect of traditional static fusion strategies. Finally, it combines with a local large language model to generate accurate question answers, significantly improving the recall and precision of searches in complex scenarios. Attached Figure Description

[0012] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0013] Figure 1 This is a block diagram of a knowledge document search engine based on word vectors and contextual summarization according to an embodiment of this application.

[0014] Figure 2 This is a schematic diagram of the data flow of a knowledge document search engine based on word vectors and contextual summaries according to an embodiment of this application.

[0015] Figure 3 This is a schematic diagram of the data flow in a multimodal parsing and indexing construction module of a knowledge document search engine based on word vectors and contextual summarization, according to an embodiment of this application. Detailed Implementation

[0016] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0017] This application addresses the issues of process rigidity and operational inaccuracy in existing technologies when handling complex business interactions with dynamic and multi-intents. Figure 1 This is a block diagram of a knowledge document search engine based on word vectors and contextual summarization according to an embodiment of this application. Figure 2 This is a schematic diagram illustrating the data flow of a knowledge document search engine based on word vectors and contextual summarization according to embodiments of this application. Specifically, as... Figure 1 and Figure 2As shown, the knowledge document search engine 100 based on word vectors and contextual summarization according to an embodiment of this application includes: an original knowledge document set acquisition module 110, used to acquire an original knowledge document set; a multimodal parsing and index construction module 120, used to perform multimodal document parsing on the original knowledge document set to obtain a summary vector index and a keyword weight index; a query understanding module 130, used to perform query intent analysis and dynamic weight generation on the acquired user query to obtain a query vector, query terms, and dynamic fusion weight; a dual-path recall module 140, used to perform dual-path parallel recall on the query vector and query terms based on the summary vector index and the keyword weight index to obtain a semantic recall candidate set and a keyword recall candidate set; a fusion and re-ranking module 150, used to perform dynamic fusion and re-ranking on the semantic recall candidate set and the keyword recall candidate set based on the dynamic fusion weight to obtain a final ranked document list; and a local question-answering generation module 160, used to pass the final ranked document list and the user query through a local question-answering large language model to obtain a final search result, wherein the final search result includes a direct answer and a cited source.

[0018] Specifically, the original knowledge document set acquisition module 110 is used to acquire the original knowledge document set. It is understood that in the macro-environment of digital office and enterprise information management, various organizations often have massive amounts of business data and knowledge assets stored in a dispersed manner. This data typically resides on different physical servers, cloud storage spaces, office automation platforms, and even employees' personal terminals, forming isolated data silos. At the same time, the presentation of this data is highly heterogeneous, including both structured database records and unstructured text descriptions, image materials, and presentations. To achieve efficient utilization of this complex knowledge, this application first establishes a data access mechanism capable of transcending storage media and encompassing multiple formats, aggregating the dispersed original files into a logical set that can be uniformly read and processed by computer programs. Therefore, for subsequent multimodal parsing, index construction, and semantic retrieval, this application acquires the original knowledge document set, ensuring that the search engine can cover every corner of the enterprise knowledge base, thereby breaking down information barriers and providing the necessary source of information for subsequent deep semantic mining and accurate question answering.

[0019] In a feasible technical solution, the processing of the original knowledge document set acquisition module 110 is as follows: As the data entry point for the entire search engine, the core function of this module is to retrieve files from a specified source address and form an original knowledge document set through a preset data connection interface and collection strategy. The specific implementation process of this module is not a simple file copying, but a complete processing flow including connection establishment, traversal scanning, format filtering, metadata extraction, and incremental identification. The original knowledge document set refers to the collection of all target files waiting to enter the parsing process after initial screening. This set contains not only the binary content stream of the files but also the original metadata information of the files. In specific implementation, this module first initializes the data connector based on preset configuration information. This connector supports multiple standard network protocols and file system interfaces, such as the SMB protocol for internal enterprise shared disks, the NFS protocol for Linux servers, and the S3 compatible protocol for object storage services, to adapt to the complex infrastructure environment of enterprises.

[0020] After the connection is established, the module initiates a traversal scanning program to recursively search all files under the target storage path. During this process, the module strictly filters file extensions according to a preset whitelist rule to ensure that only valid documents relevant to knowledge retrieval are obtained. This original collection of knowledge documents covers a variety of common office document formats, specifically including Word documents (such as .doc and .docx formats) for detailed text reports, PDF documents for publishing formal specifications or papers, spreadsheet files (such as .csv and .xlsx formats) for displaying statistical reports, presentation slides (such as .ppt and .pptx formats) for meeting reports and solution presentations, and image files containing scanned copies, drawings, or photographs (such as .jpg and .png formats). For each format, the module is configured with a corresponding file stream read handle.

[0021] During file acquisition, the module implements an incremental update strategy by calculating the file's hash fingerprint (e.g., using MD5 or SHA-256 algorithms). For each scanned file, the module calculates its hash value and compares it with the fingerprint in the local historical database. If the hash value does not exist, it is determined to be a new document; if the hash value already exists but the corresponding file's last modification time has changed, it is determined to be an updated document; if both match, the file is skipped, thus avoiding duplicate collection of unchanged data and reducing system resource consumption. For files determined to be new or updated, the module reads their complete binary data stream into the search engine's temporary storage area.

[0022] Simultaneously, the module also extracts metadata for each file, including but not limited to filename, file path (as the basis for generating citation sources later), file size, creation time, last modification time, and author information. This metadata is encapsulated into a unified JSON object, which, together with the corresponding binary file stream, constitutes a single document object in the original knowledge document collection. For example, for a file named "2023 Annual R&D Report.pdf", the module reads its PDF content as a byte stream and generates a metadata data package containing its storage path / server / docs / 2023 / and modification time. Once all target paths have been scanned and file transfers are complete, the module outputs this original knowledge document collection containing files of various formats and their metadata as input data for the multimodal parsing and indexing construction module, thus initiating the subsequent document standardization and deep parsing process.

[0023] Specifically, the multimodal parsing and indexing module 120 is used to perform multimodal document parsing on the original knowledge document set to obtain a summary vector index and a keyword weight index. Correspondingly, after acquiring a massive set of original knowledge documents, although these data are centrally stored, they are essentially still a jumbled mess of binary data streams with varying formats. For a computer, a PDF technical document, a JPG architecture diagram, and a Word meeting record have completely different underlying encoding methods, making direct, unified content comparison and retrieval impossible. Furthermore, original documents are often lengthy and contain a large amount of noise information such as formatting symbols, headers, footers, and polite phrases that are useless for expressing the core semantics. Directly vectorizing the entire text would not only be computationally intensive but would also dilute the core semantics, severely impacting the recall accuracy of subsequent searches. Therefore, the step of performing multimodal document parsing on the original knowledge document set to obtain the summary vector index and keyword weight index is to transform heterogeneous physical files into standard semantic representations that can be understood and computed by computers. By removing noise, extracting summaries, and constructing a high-dimensional vector space, a standardized data foundation is laid for subsequent high-precision semantic matching and keyword retrieval.

[0024] Figure 3 This is a schematic diagram illustrating the data flow in a multimodal parsing and indexing construction module of a knowledge document search engine based on word vectors and contextual summarization, according to an embodiment of this application. For example... Figure 3As shown, in a feasible technical solution, the multimodal parsing and indexing construction module 120 includes: a document parsing and standardization unit 121, used to perform heterogeneous document parsing and content standardization on the original knowledge document set to obtain a plain text database; a summary generation and semantic vectorization unit 122, used to perform contextual summary generation and semantic vectorization on the plain text database to obtain a summary-vector pair set; a keyword weight calculation unit 123, used to calculate keyword weights on the plain text database to obtain a keyword weight database; and an index generation unit 124, used to generate a summary vector index and a keyword weight index based on the keyword weight database and the summary-vector pair set.

[0025] Specifically, the multimodal parsing and index building module 120 processes the data as follows: First, the document parsing and standardization unit 121 receives the original knowledge document set output from the original knowledge document set acquisition module. This set contains text documents such as "2023 Annual R&D Center Work Report.pdf" and image documents such as "Server Cluster Topology.jpg". The document parsing and standardization unit 121 first initiates a distribution parsing task, traversing each document in the set and reading its file extension metadata to determine the processing logic. For text documents (such as .doc, .docx, .pdf, .xlsx, .txt), this unit calls a pre-built Python standard API library for direct content extraction. Taking "2023 Annual R&D Center Work Report.pdf" as an example, this unit calls the PyPDF2 or pdfplumber library to parse the PDF's file structure tree, locate the text stream object, and decode the characters encoded in it, such as UTF-8, into a string sequence. For table files (.xlsx), the pandas library is called to read the data frame and convert the table content into a structured text description (such as the content of row 1, column A being...). For image or image-text hybrid documents (such as .jpg, .png, or images embedded in PPT), this unit calls the Optical Character Recognition (OCR) engine interface for processing. Taking "Server Cluster Topology.jpg" as an example, this document contains a network architecture diagram and labeled text. The OCR engine first preprocesses the image, including grayscale conversion to reduce computational dimensionality and binarization processing, such as using the Otsu algorithm to separate background and foreground text. The formula is: if the pixel grayscale value is greater than the threshold, it is set to 255 (white); otherwise, it is set to 0 (black). Subsequently, a deep learning-based text detection network, such as CTPN or DBNet, is used to locate the text regions in the image and generate a series of bounding box coordinates. In this step, if the CTPN model is used, its core architecture combines CNN and RNN. Specifically, CTPN first extracts spatial feature maps of the image using the first five convolutional layers of the VGG16 network. Next, it performs convolution operations on the feature maps using a 3×3 sliding window to generate a fixed-width sequence of anchor points. These feature sequences are then input into a bidirectional long short-term memory network to capture the contextual sequence features of text lines in the horizontal direction, which is particularly effective for detecting long text lines. Finally, a fully connected layer outputs the vertical coordinates (y-coordinate) and scores of each anchor point, and connects and merges consecutive thin rectangular boxes with high scores to accurately locate the bounding boxes of the text lines. If DBNet is used, a fully convolutional network predicts probability maps and threshold maps, and a differentiable binarization module is introduced. The learned threshold map adaptively binarizes the probability map to generate a binarized map, and then a contour extraction algorithm is used to directly obtain the boundaries of text instances. This architecture is particularly adept at handling curved or irregularly shaped text regions.Specifically, the parameters in the above model are all obtained through training. Then, the cropped text region is input into a sequence recognition network such as the CRNN model. The CRNN model extracts image features through a convolutional layer (CNN), predicts the label distribution of the feature sequence through a recurrent layer (RNN / LSTM), and finally converts the predicted label sequence into the final text string through a connectionist temporal classification (CTC) layer, such as extracting text content like load balancer, master node IP: 192.168.1.10, etc. After the content extraction is completed, the document parsing and normalization unit integrates and cleans the extracted original text. This unit binds the extracted text with the unique identifier of the document, i.e., the document ID. Then, preprocessing operations are performed: invisible control characters (such as \x00), redundant line breaks and tab characters are removed using regular expressions; in order to ensure the quality of the text data, character filtering rules are defined in this embodiment (such as judging the character encoding range through code logic), restricting to only retain the valid Unicode encoding range (for example, from \u4e00 to \u9fff) to filter out garbled or invalid symbols of non-Chinese characters; at the same time, a predefined stop word list containing high-frequency function words with low semantic contribution such as "the", "is", "has", "in" is loaded, and the stop words in the text are filtered. For example, the original sentence "The goal of this project is to improve efficiency" is normalized to "project goal improve efficiency". To take into account subsequent semantic understanding tasks, the embodiment of this application adopts a dual-path storage strategy, storing the cleaned documents in two copies, one retaining the sentence structure, paragraph structure, and discourse structure, and merging multiple documents into one database. This database stores text streams with complete context logic such as DeepSeek research reports for subsequent in-depth reading and summary generation by large language models. Finally, this unit persists and stores the document ID, such as DOC_2024_001, and its corresponding normalized pure text content as a record in the pure text database.

[0026] Subsequently, the semantic vectorization unit 122 reads data from the plain text database and begins contextual summarization and semantic vectorization processing. This unit aims to address the semantic dilution problem caused by direct vectorization of long documents. During the summarization stage, this unit traverses the plain text database and calls the API interface of a Large Language Model (LLM) for the plain text content of each document. In this embodiment, the LLM model used, such as the DeepSeek model or a generative model based on the Transformer architecture, has been pre-trained on a massive corpus. The specific architecture of this model is based on the Decoder-only or Encoder-Decoder structure of Transformer. Taking the Transformer structure as an example, its core components include a multi-head self-attention mechanism. The model receives the token sequence embeddings of the input text and superimposes positional encodings to preserve sequence information. In the self-attention layer, the input vector is mapped to a query matrix, a key matrix, and a value matrix. Through multi-layered stacked attention blocks and feedforward neural networks, the model is able to capture long-distance dependencies in the text. In its implementation, this unit constructs specific prompts, such as: "Please read the following technical document and generate a concise summary of no more than 200 words, summarizing its core technical points and conclusions: [Input document content]". The standardized text "2023 R&D Center Work Report" is input into the model, which generates a summary text based on probability prediction, such as: "This year, the R&D team completed the architecture reconstruction of the distributed storage system, increasing the throughput of core modules by 40%, and successfully launched the intelligent operation and maintenance platform, reducing the average fault recovery time to less than 5 minutes." This generated text is the document summary. In the semantic vectorization stage, this unit maps the generated document summary to a high-dimensional vector space. To construct the vector model for mapping, the segmented training data X_train is first imported, and a shallow neural network model (Word2Vec) is instantiated. The vector dimension vector_size is set to 300 to ensure semantic richness, and the minimum word frequency min_count is set to 10 to filter noise. Then, the build_vocab and train functions are called to iteratively train on the corpus, thereby obtaining the basic word embedding vectors. To visually verify the trained word vector model's ability to capture business semantics, in an interactive testing environment, the word search function (word_search) was invoked, with the target word "code" (w='code') input and the recall count set to 20 (k=20). After calculating the distance in the vector space, the model returned a list of semantically strongly related words, including "warehouse," "product," "trading day," and "product specifications," along with their corresponding distance scores (e.g., 0.140869, 0.143331, etc.).These data demonstrate that the basic word vector model has successfully learned the implicit entity association logic in business documents (e.g., "code" is often associated with "product" in "warehouse"), providing a high-quality vector space foundation for subsequent semantic processing. Building upon this, to further capture complex syntactic features, this embodiment employs a pre-trained word vector model, specifically the BERT model. The BERT model mainly consists of multiple layers of Transformer Encoders. Unlike generative models, BERT is pre-trained using Masked Language Modeling (Masked LM) and Next Sentence Prediction (NSP) tasks, possessing bidirectional contextual understanding capabilities. This unit segments the document summary and adds a special [CLS] marker at the beginning of the sequence and a [SEP] marker at the end. The input tokens are converted into the sum of word embedding vectors, segment embedding vectors, and positional embedding vectors. After processing by the 12-layer (Base version) or 24-layer (Large version) Transformer encoder within the BERT model, the parameters (weight matrix and bias term) of each layer are fixed values ​​obtained through iterative updates based on a large-scale general corpus using the backpropagation algorithm during the pre-training phase. The model outputs a context vector for each token. To obtain a document vector representing the semantics of the entire summary, this unit extracts the [CLS] marker from the output vector of the last layer, or calculates the average pooling value of all token output vectors. If the model output dimension is 768, the generated document vector is an array containing 768 floating-point numbers, for example, [-0.024, 0.156, ..., 0.882]. Mathematically, this vector represents the coordinate position of the document summary in the semantic space, and the cosine similarity between vectors represents the degree of semantic similarity. Finally, data aggregation is performed. This unit combines the unique identifier of the document, such as the readable text summary generated from DOC_2024_001, and the calculated high-dimensional document vector into a tuple (unique identifier, text summary, document vector). This tuple is stored in a summary-vector pair set.

[0027] In this embodiment, the keyword weight calculation unit 123, as a core component of the multimodal parsing and indexing construction module, is responsible for extracting the most representative keywords from the plain text database and calculating their weights. In a feasible technical solution, the keyword weight calculation unit 123 includes: a multidimensional feature calculation subunit 1231, used to perform multidimensional feature calculations on each candidate word in the plain text database based on the corpus to obtain a candidate word feature database, wherein the multidimensional features include average information entropy, part-of-speech features, and positional features; a comprehensive initial weight calculation subunit 1232, used to calculate the comprehensive initial weight of each candidate word in the candidate word feature database to obtain a word initial weight database; a weighted word graph iteration subunit 1233, used to construct a weighted word graph structure based on the word initial weight database and the plain text database, and to perform iterative calculations on the weighted word graph structure to obtain a word final weight database; and a keyword extraction subunit 1234, used to extract and store Top-K keywords from the word final weight database to obtain a keyword weight database.

[0028] The multi-dimensional feature calculation subunit 1231 executes first: Here, the corpus refers to the complete text set aggregated from all document content in the plain text database output by the document parsing and standardization unit in the previous stage. This corpus not only contains the currently processed documents but also provides statistical background, offering information on the global distribution of words. During implementation, this subunit first performs preprocessing operations. In the preprocessing stage, this unit traverses each document in the plain text database. Using document ID DOC_2024_001, this unit calls the embedded Chinese Academy of Sciences word segmentation system ICTCLAS 2015 for processing. ICTCLAS, based on a cascaded hidden Markov model, can efficiently handle out-of-vocabulary words and resolve ambiguous segmentation problems. During processing, the system segments the text sequence into independent lexical units, i.e., candidate words. It also labels each word with its part of speech, such as noun (n), verb (v), adjective (a), etc., and records the word's position in the document, such as title, first paragraph, or middle of the body text. Specifically, it completes the task by calculating the part-of-speech sequence and word boundary sequence with the highest probability of occurrence given a word sequence. After processing, the original text is converted into a labeled word sequence, for example: "This year / t R&D / vn team / n completed / v of / u distributed / n storage / vn system / n architecture / n refactoring / vn ...". At the same time, this unit marks the position of words according to their position in the document structure. Then, this sub-unit targets each candidate word in each document. Parallel computation of feature values ​​in three dimensions: average information entropy. Part-of-speech features and location features The first feature is the average information entropy. This feature measures the evenness of word distribution and information content across the entire document set. Keywords representing the core theme of a document should appear frequently in that specific document but be relatively scarce in the entire corpus, i.e., unevenly distributed. This sub-unit calculates words according to the following formula. Average information entropy:

[0029]

[0030] In the formula, This is the calculated average information entropy, whose value is directly used as the feature value. .symbol This represents the total number of documents in the corpus, for example, there are 10,000 documents in the corpus. Words In the Frequency (number of times) of appearance in the document. Words The total frequency of occurrences across the entire document set, i.e., the corpus. Using the term "distributed" as an example, the specific calculation logic is as follows: First, count the total number of occurrences of "distributed" across all documents. If it is 50 times, then iterate through these 1000 documents. If it is the 50th document... If it appears twice in the document, then =2. Item This represents the probability distribution of the word in a specific document. This is the self-information of that probability. If a word is uniformly distributed across all documents, this entropy value... The entropy value will approach 1; if a word appears only in a very small number of documents (i.e., it has strong document distinguishability), the entropy value will approach 0. In this embodiment, to conform to the logic that the larger the weight, the more important it is, the value may be 1- in practical applications. Alternatively, the entropy value can be used directly as an indicator of its generalization ability; the calculated value here will be... Value as For example, the calculated result is 0.35. The second dimension feature is the part-of-speech feature. Different parts of speech contribute differently to expressing the document's topic. This unit uses ICTCLAS-annotated parts of speech, such as "distributed" being labeled as a noun (n) or a gerund (vn), and consults a pre-defined keyword part-of-speech distribution table. This distribution table is based on prior probabilities derived from a large-scale, high-quality keyword dataset; for example, the probability of a noun (n) being a keyword is 56.2%, a verb (v) is 22.7%, and an adjective (a) is 10.5%. This unit maps parts of speech to specific weight values. For example, for the noun "distributed," its... It is assigned a value of 0.562; while for the auxiliary word "de", its weight may be only 0.01. The third-dimensional feature is the position feature . This unit assigns different weights according to the position where the word appears. This strategy is based on an empirical rule: words that appear in the title or abstract are usually more important than those in the body text. For example, the empirical weights are set as follows: Title = 0.5, Summary = 0.4, FirstGraph = 0.3, Body = 0.2. For the candidate word "distributed", since it appears in the abstract text "This year, the R & D team completed the architecture reconstruction of the distributed storage system..." generated by the abstract generation unit, this unit assigns it a value of 0.4. If the word also appears in the title, the maximum value or the cumulative value is taken. In this example, the maximum value of 0.5 is taken. After the above calculations are completed, the multi-dimensional feature calculation sub-unit 1231 encapsulates the data, and the generated output format is {"DOC_2024_001": [("distributed", 0.35, 0.562, 0.5), ("storage", 0.32, 0.562, 0.5), ("reconstruction", 0.41, 0.227, 0.4),...]} and stores it in the candidate word feature database

[0031] . Immediately afterwards, the comprehensive initial weight calculation sub-unit 1232 fuses the above three independent feature dimensions into a unified initial weight value to provide a high-quality node initial state for subsequent graph model iterations. In a feasible technical solution, the comprehensive initial weight calculation sub-unit 1232 is used to: calculate the comprehensive initial weight of each candidate word in the candidate word feature database according to the following formula, and the formula is

[0032]

[0033] where , and are the average information entropy, the词性特征 (should be "positional feature" in the original text, assuming it's a typo here), and the positional feature of the candidate word respectively, , and are the weight coefficients of the average information entropy, the positional feature, and the positional feature respectively, is the comprehensive initial weight of the candidate word . This sub-unit first traverses each document record in the candidate word feature database. For each candidate word w and its feature tuple ( , , ) in the record, it applies the linear weighted formula to calculate its comprehensive initial weight . It is worth elaborating that these three weight coefficients , and It was obtained through a pre-trained lightweight neural network model and satisfies The neural network model is structured as a single-layer fully connected network or a linear regression model. During system initialization, it is trained using a standard dataset containing manually labeled keywords. The input layer receives three feature values ​​( , , The output layer represents the probability score (between 0 and 1) of whether a word is a keyword. Through backpropagation and a gradient descent optimizer (such as Adam), the model continuously adjusts the weight parameters corresponding to the input features. After training converges, the extracted feature weights are... , and If, after training, the parameters obtained are: =0.4 (This indicates that the distribution characteristics are the most important). =0.3 (part of speech is secondary), =0.3 (position is also important). Based on the above parameters, this unit performs specific calculations for candidate words, such as "distributed": 0.4 × 0.35 + 0.3 × 0.562 + 0.3 × 0.5 = 0.4586. Similarly, for the word "reconstruction", =0.41, =0.227, =0.4: =0.4×0.41+0.3×0.227+0.3×0.4=0.3521. A comparison shows that although the reconstructed version has a higher information entropy, its final initial weight is also lower because its combined score for part-of-speech (gerund) and positional weights is lower than that of the distributed version. This calculation method effectively corrects the bias of a single feature. Finally, the integrated initial weight calculation subunit formats and stores the calculation results. For document DOC_2024_001, the generated record is {"DOC_2024_001":[("Distributed",0.4586),("Storage",0.4421),("Reconstruction",0.3521),...]}. This data is written to the word initial weight database.

[0034] The first step of the weighted word graph iterative subunit 1233 is to construct the weighted word graph structure. This subunit traverses each document in the plain text database. Taking document DOC_2024_001 as an example, this document contains words such as "distributed," "storage," and "reconstruction" that have undergone word segmentation. This subunit uses a sliding window mechanism to determine the co-occurrence relationship between words. The window size is preset to an integer of 5 based on the principle of local relevance in natural language text. When the distance between the words "distributed" and "storage" in the text sequence is less than the window size, the algorithm determines that there is a co-occurrence relationship between them and places them in the graph structure. Establish a connecting edge between them. This represents a set of nodes, i.e., candidate words in the document; This represents the set of edges. Unlike the traditional TextRank algorithm, which simply sets edge weights to 1, this scheme constructs a directed weighted graph where edge weights are influenced by the node's own attributes. To transform the abstract graph structure into an algebraic form that can be efficiently processed by a computer, this sub-unit instantiates the weighted word graph as a modified word segmentation relevance matrix. In this matrix (a 125×125 dimension local view is shown in this example), rows and columns represent candidate word nodes in the document (such as owner, personal information, project, etc.). The element values ​​in the matrix represent the one-way or two-way influence weight between two word nodes. For example, the graph shows that the association weight between owner and personal information is as high as 2.533885, indicating that based on the comprehensive initial weights calculated in the preceding steps (average information entropy, part-of-speech, position) and the co-occurrence statistics within the sliding window, these two words have a very strong semantic binding relationship in the current document context. Compared to the sparse 0 / 1 adjacency matrix in traditional algorithms, this dense floating-point matrix can more finely characterize the potential logical distance between words, providing a more accurate gradient direction for subsequent iterative convergence. After the graph construction is completed, the initialization and iterative calculation phase begins. This sub-unit reads the pre-calculated comprehensive initial weights from the word initial weight database. In the preceding steps, the overall initial weight of the candidate word "distributed" has been calculated. =0.4586, the overall initial weight of "Storage" =0.4421, the initial weight of the "reconstruction" =0.3521. This sub-unit first performs the initialization operation: a. Set the overall initial weight of each word. b. As the attraction factor for in-degree traffic to the corresponding node in the graph. Iterative score Initialize to 1. This means that before the iteration begins, it is assumed that all words are equally important; differences will gradually emerge during subsequent graph walks guided by topology and initial weights. c. Set the damping coefficient. This parameter simulates the probability of continuing browsing in a random walk model. Based on the classic configuration of the PageRank algorithm, it is set to 0.85, meaning there is a 15% probability of jumping to any node in the graph, thus avoiding getting trapped in a local loop. Subsequently, this sub-unit uses the improved TextRank formula for iterative calculation. This formula corrects the problem of node homogenization in the traditional algorithm by introducing a comprehensive initial weight for the nodes. The iteration formula is as follows:

[0035]

[0036] In this formula, Representative node For example, the weight score calculated for "distributed" in this iteration. It points to a node The set of all neighboring nodes. If both "Storage" and "System" co-occur with "Distributed", then they belong to this set. Neighboring nodes For example, "storage" refers to the collection of all nodes. Neighboring nodes Score in the previous iteration. Key Items Represents the node Transfer to node The probability of edge weights. In this technical solution, edge weights... It is the target node Comprehensive initial weights Decision. That is. = This means that if the initial weights are "distributed" =0.4586, which is higher than the initial weight of "System". Therefore, when the algorithm distributes weights outward from the "Storage" node, it will allocate more shares to "Distributed". This mechanism ensures that words with high scores in multi-dimensional feature calculation can further aggregate weights in graph iteration. Specifically, let's describe one iteration process: calculate the weights of the "Distributed" nodes. Value. If the node connected to it has "Storage". "Storage" represents the current value. The value is 1. "Storage" connects to two nodes: "Distributed" and "System". The initial weight of the target node is known: =0.4586, =0.25. Therefore, the probability of migrating from "storage" to "distributed" is: =0.4586 / (0.4586+0.25)≈0.647. Similarly, the probability of transferring to "System" is approximately 0.353. If "Storage" is the only node pointing to "Distributed", then the new score for "Distributed" is: =(1-0.85)+0.85×(0.647×1)=0.69995. In this way, originally important words "vote" for each other through co-occurrence relationships, causing the weight of the core keyword to continuously accumulate and amplify. This sub-unit continues to perform the above calculation, and after each iteration, it calculates the weight of all current nodes. Value compared to the previous iteration The absolute value of the difference between the values. This sub-unit has a preset convergence threshold, for example, 0.000001. This threshold is stored in the system configuration file and is used to balance computational accuracy and time cost. When the difference between all nodes is less than this threshold, the decision graph structure has reached a steady state, and the iteration process ends. At this point, the final weights of the "distributed" nodes are... It may converge to 1.85, while common words such as "conduct" may converge to 0.2. This sub-unit stores all words and their final weights in the word final weight database.

[0037] Finally, the keyword extraction subunit 1234 processes the above results. This subunit first iterates through each document record in the final word weight database. For document DOC_2024_001, this subunit reads all words... The scores are calculated and sorted in descending order. After sorting, this sub-unit performs a filtering operation. This involves a configurable parameter K, the K value in Top-K. This parameter K represents the upper limit of the number of keywords retained for each document, dynamically calculated based on the system's indexing strategy configuration or the average document length. For example, for a technical document with an average length of 3000 words, the K value is typically set to 10 or 15 to cover the document's main technical points. In this embodiment, K=5. This sub-unit extracts the top 5 words from the sorted list. For example, the top five words in the sorted result might be: 1. Distributed ( =1.85); 2. Storage ( =1.76); 3. Reconstruct ( =1.42); 4. Architecture ( =1.10); 5. Throughput ( =0.95). These words and their corresponding weights are extracted, forming the core feature representation of the document. Finally, this sub-unit associates and formats the document identifier (document ID) with this set of (keyword, weight score) pairs, generating a record in the form of {"DOC_2024_001":[("Distributed",1.85),("Storage",1.76),...]}, and persists it in the keyword weight database.

[0038] The index generation unit 124 is implemented in two parallel or sequential processing flows: constructing a vector index and constructing a keyword weight index. In constructing the summary vector index, this unit first extracts all (document ID, summary vector) tuples from the summary-vector pair set. Taking the document DOC_2024_001 generated in the previous step as an example, its corresponding 768-dimensional summary vector is [-0.024, 0.156, ..., 0.882]. To efficiently retrieve massive amounts of high-dimensional vectors, this unit introduces a specialized vector index library; in this embodiment, the FAISS library is used. Instead of using a brute-force flat index (IndexFlatL2), this unit constructs an optimized inverted file product quantization index structure (IVF-PQ). The construction process of this structure specifically includes two core steps: coarse quantization and fine quantization. First, this unit sets the number of cluster centers based on the total number of vectors. The settings are determined based on the size of the dataset, such as 100 or 4096. The K-Means algorithm is then used to partition the high-dimensional vector space into... The vector of document DOC_2024_001 is assigned to an inverted list represented by the nearest cluster center. This process, called IVF coarse quantization, significantly narrows the search scope by scanning only a few clusters close to the query vector during retrieval. Secondly, to reduce memory usage and accelerate distance calculation, this unit performs PQ product quantization on the vector. The 768-dimensional vector is divided into M subvectors, determined by the vector dimension and the desired compression ratio (e.g., M=8). Each subvector is clustered in its own subspace and represented by a short code (e.g., an 8-bit integer). After training and addition operations, the original floating-point vector of DOC_2024_001 is compressed and indexed. The final generated summary vector index file is persistently stored. This index supports quickly returning the list of most similar Top-K document IDs using Euclidean distance or inner product distance by inputting a query vector. As a functional verification example after the vector index was built, the system was tested in an actual program interaction environment by calling a custom context search function, `context_search`. When the input query term list ['IoT'] was set to a recall threshold k=20, the system quickly recalled a series of highly semantically related documents from the massive dataset based on the Euclidean distance calculated by the built IVF-PQ index (such as the value 0.038877 shown in the list). The smaller these distance values, the closer the document vector is to the query intent in the semantic space, thus intuitively verifying the effective capture and accurate sorting capability of the summary vector index for core semantics (such as IoT control, management system, etc.) when processing heterogeneous documents such as CSV. Not limited to single-dimensional queries, this index structure also supports joint retrieval of complex intents. In advanced testing, the input variable `w` was set to a keyword list containing multiple discrete semantics ['dropdown menu', 'main axis', 'personnel'], and the recall threshold `k` was set to 20. After calling the search function `context_search`, the system performed aggregation operations on multi-word semantics in the vector space. The output results show that the system successfully retrieved composite technical documents containing the aforementioned multidimensional features in the graph. This result demonstrates that the summary vector index constructed in this application can effectively handle the semantic intersection and fusion of multiple keywords, and can find potentially highly relevant documents through the geometric proximity of the vector space even when the document title does not completely cover all query terms. In constructing the keyword weight index, this unit traverses each record in the keyword weight database. Continuing from the preceding data, document DOC_2024_001 contains a set of keywords [("distributed", 1.85), ("storage", 1.76), ("reconstruction", 1.42),...]. This unit reorganizes the data through inverted index logic to construct a mapping structure, where the keys are independent keywords and the values ​​are inverted lists.For the keyword "distributed," the system creates an entry in the index that points to a list containing information about all documents where the word appears. Each element node in the list contains not only the document ID but also the final weight score of the word within the document. For example, the inverted list for the keyword "distributed" might be [...] (DOC_2024_001, 1.85) -> (DOC_2023_089, 0.92) -> ... . Here, the weight 1.85 is the high-precision semantic weight calculated through iterative preorder weighted word graph analysis, rather than a traditional word frequency statistic. For the keyword "storage," its inverted list also contains the node (DOC_2024_001, 1.76). This unit performs this operation on all keywords across all documents, resulting in a weighted inverted index structure. Unlike traditional Boolean indexes, this structure allows for the use of ranking algorithms such as BM25 during the retrieval phase. It directly utilizes the weight values ​​stored in the index to assign weighted scores to documents, ensuring that documents containing core keywords (with high weight) are ranked higher in the recall list.

[0039] Specifically, the query understanding module 130 is used to perform query intent analysis and dynamic weight generation on the acquired user query to obtain query vectors, query terms, and dynamic fusion weights. It should be understood that in traditional search engine architectures, the query string entered by the user is often directly used to match the index, and the system does not care about the true search intent behind this string. However, user query behavior is extremely diverse and complex: sometimes it is to find specific technical standards or error codes, requiring extremely high precision matching; sometimes it is to explore solutions to a concept or industry trends, relying more on semantic generalization understanding. Most existing hybrid retrieval technologies adopt static weight allocation strategies (e.g., fixed semantic retrieval weight of 0.7 and keyword weight of 0.3). This one-size-fits-all approach cannot perceive changes in user intent, leading to noise introduced due to over-generalization of semantics in precise query scenarios, or missing relevant documents due to overly strict keyword matching in fuzzy query scenarios. To break this deadlock, this application implements a step of performing query intent analysis and dynamic weight generation on the acquired user queries to obtain query vectors, query terms, and dynamic fusion weights. This is to enable the search engine to have an intelligent brain that can understand user intent and predict the optimal semantic and keyword weight ratio in real time based on the specific characteristics of the query, thereby achieving adaptive and accurate recall for different types of query tasks.

[0040] In a feasible technical solution, the query understanding module 130 includes: a query preprocessing unit 131, used to perform lexicalization and vectorization processing on the acquired user query to obtain query lexical units and query vectors; a query feature extraction unit 132, used to extract linguistic features from the query lexical units and user query to obtain query feature vectors; and a dynamic fusion weight prediction unit 133, used to input the query feature vectors into a pre-trained lightweight regression model to obtain dynamic fusion weights.

[0041] Specifically, the query understanding module 130 processes queries as follows: The query preprocessing unit 131 is the entry point for the entire module, responsible for receiving user queries from the front-end interactive interface. A user query refers to the raw text string entered by the user in the search box, which may be a phrase, a complete question, or a set of keywords. The query is obtained by passing it to the backend service via an HTTP request or RPC call. Assuming the user enters a specific query: "How to optimize the read / write throughput of a distributed storage system?", the query preprocessing unit first performs lexicalization. This unit calls a pre-trained word vector model (such as BERT or RoBERTa) and its associated tokenizer. This model is based on the Transformer architecture and has undergone self-supervised learning from massive amounts of text, possessing strong language understanding capabilities. The tokenizer segments the input string into the smallest semantic units that the model can recognize. For the above query, the tokenizer segments it into ['how', 'optimize', 'distributed', 'storage', 'system', 'of', 'read / write', 'throughput', '?']. This list of words or subwords is the query terminology, which will be directly used for subsequent keyword retrieval and feature extraction. Next, this unit performs vectorization processing. The original user query string is input into the encoder of the same pre-trained model. The encoder consists of a multi-layer self-attention mechanism and a feedforward neural network. During the model's forward propagation, each input token is first mapped to an initial embedding vector, and then, through layers of encoding, contextual information is incorporated. Finally, a specific position vector from the model output is extracted, such as BERT's [CLS] token vector, or all token output vectors are averaged and pooled to obtain a high-dimensional, dense floating-point vector that represents the semantics of the entire query statement—the query vector. If the model output dimension is 768, this vector is an array containing 768 real values, for example, [0.12, -0.56, ..., 0.89]. This vector will be used in the subsequent semantic retrieval stage to search for semantically similar documents in the summary vector index.

[0042] Subsequently, the query feature extraction unit 132 takes over the processing, aiming to quantify the intent tendency of the query from a linguistic perspective. This unit uses query lexical units and the user's query string to extract four key scalar features, forming a query feature vector. Specifically, the query feature vector includes query length, average inverse document frequency (IVF), entity word ratio, and whether it is an interrogative sentence structure. First, the query length is calculated. This unit directly counts the number of elements in the query lexical list. For the list ['how', 'optimize', 'distributed', 'storage', 'system', 'of', 'read / write', 'throughput', '?'], it contains 9 lexical units, therefore the query length feature value is 9. This feature is based on a naive assumption: longer queries tend to be natural language descriptions, leaning more towards semantic understanding; while very short queries, such as error code 502, tend to be exact searches, leaning more towards keyword matching. Second, the average inverse document frequency (IVF) is calculated. This unit relies on a global IDF lookup table, which is generated statistically based on the entire corpus during the offline phase. The table records the rarity of each word in the entire document set. Common terms, such as "how" and "how," have lower IDF values, while specialized terms, such as "distributed" and "throughput," have higher IDF values. This unit iterates through the query term list, finds the IDF value of each word, sums them, and finally divides by the query length. For example, the IDF of "how" is 1.2, "distributed" is 8.5, "storage" is 6.3, "throughput" is 9.1, and the average of other words is 2.0. The total is approximately 1.2 + 5.0 + 8.5 + 6.3 + 4.0 + 1.0 + 7.5 + 9.1 + 1.0 = 43.6. The average IDF = 43.6 / 9 ≈ 4.84. A higher average IDF value suggests that the query contains more unique information and may focus more on the exact hit of keywords. Next, the entity word ratio is calculated. This unit calls a Named Entity Recognition (NER) model. This NER model is based on a BiLSTM-CRF or BERT-CRF architecture. The BERT layer extracts contextual features, and the CRF (Conditional Random Field) layer uses state transition probabilities to constrain the rationality of the output label sequence (e.g., B-LOC cannot be followed by I-PER). When the user query "How to optimize the read / write throughput of a distributed storage system?" is input into the model, the model identifies that "distributed storage system" might be labeled as a technical term, and "read / write throughput" might be labeled as a metric. If the total number of identified entity words is 5 (distributed, storage, system, read / write, throughput), then the entity word ratio = 5 / 9 ≈ 0.56. A high entity word ratio indicates that the query intent is highly focused on a specific object, and the importance of keyword retrieval increases accordingly. Finally, it determines whether it is an interrogative sentence structure. This unit performs rule matching: checking whether the user query string begins with interrogative words such as "how," "why," or "what," or ends with a question mark "?". In this example, the query begins with "how" and ends with "?", which obviously meets the condition. Therefore, this feature bit is set to 1.The interrogative structure strongly suggests that the user is seeking an explanation or solution, rather than simply looking for documents containing a certain word. This requires the semantic retrieval engine to demonstrate stronger generalization capabilities. After completing the above calculations, the unit assembles these four scalar values ​​[9, 4.84, 0.56, 1] into a one-dimensional vector, namely the query feature vector.

[0043] Finally, the dynamic fusion weight prediction unit 133 uses this feature vector to decide on the retrieval strategy. At its core is a pre-trained lightweight regression model, such as a multilayer perceptron (MLP). The MLP architecture consists of an input layer (4 neurons), one or more hidden layers (e.g., 16 neurons using the ReLU activation function), and an output layer (1 neuron). The model's parameters (weight matrix and bias terms) are obtained during the system development phase through supervised learning using a large number of query-document pairs labeled with optimal retrieval weights. Labeled data may come from manual annotation (expert judgment on the proportion of semantic retrieval for a certain type of query) or backfitting based on user click logs. The specific processing involves two steps. The first is semantic tendency prediction. The query feature vector [9, 4.84, 0.56, 1] is input into the MLP model. The model performs forward propagation calculations: ,in It is a pre-trained weight matrix. It is a bias vector. This is the query feature vector. This linear transformation is essentially a weighted sum of the input features, with each layer of neurons extracting latent patterns from different feature combinations. For example, the model might learn that when the query is an interrogative sentence (feature 4=1) and the query is long (feature 1>8), the output value should be significantly increased. The final output layer produces a real scalar. That is, the logit value of semantic tendency, such as the model's calculated logit value in this example. The value is 1.2. The second step is weight normalization generation. To convert the logit value into a usable probability or weight, the unit will... Input sigmoid activation function for =1.2, calculated to be approximately 0.77, which is the final dynamic fusion weight. This value is not just a number; it represents the system's deep understanding of the current query intent: for this long and complex sentence "How to optimize...", the system judges that semantic understanding accounts for 77% of the importance, while keyword matching accounts for 23%.

[0044] Specifically, the dual-path recall module 140 is used to perform parallel dual-path recall on query vectors and query terms based on the summary vector index and keyword weight index to obtain semantic recall candidate sets and keyword recall candidate sets. Correspondingly, in practical applications of enterprise-level knowledge retrieval, a single retrieval mode often struggles to simultaneously achieve both recall and precision—two core metrics. Traditional keyword-based retrieval (such as BM25) exhibits extremely high accuracy when handling specific terms, error codes, or product models, but it is prone to missing a large number of relevant documents due to semantic ambiguity, diverse expressions, or the need for conceptual reasoning. Conversely, while vector-based semantic retrieval can overcome literal differences and capture the deeper meaning of documents, it often introduces a large number of low-relevance or even erroneous noise results due to semantic drift in the vector space when facing scenarios requiring precise matching. To fully leverage the advantages of these two retrieval paradigms and compensate for their respective shortcomings, a parallel dual-path retrieval architecture needs to be constructed, utilizing semantic features and lexical features for independent candidate set recall. Therefore, this application uses abstract vector index and keyword weight index to perform parallel dual-path recall on query vectors and query terms to obtain semantic recall candidate sets and keyword recall candidate sets, so as to comprehensively and multi-dimensionally screen potentially relevant documents from massive document libraries, providing a high-quality initial screening pool that includes both semantic relevance and precise matching for subsequent refined re-ranking.

[0045] In a feasible technical solution, the dual-path recall module 140 includes: a semantic recall unit 141, used to perform a K-nearest neighbor search on the query vector in the summary vector index to obtain a semantic recall candidate set; and a keyword recall unit 142, used to search on the query terms in the keyword weight index to obtain a keyword recall candidate set.

[0046] Specifically, the dual-path recall module 140 processes the following: The semantic recall unit 141 first receives the query vector generated by the query understanding module. The task of this unit is to use this vector to find the document that is closest to it in the semantic space within the pre-built summary vector index. The summary vector index invoked by this unit is an IVF-PQ structure built based on the FAISS library. At the start of the retrieval, the system first performs the same preprocessing on the query vector, such as normalization. Then, the unit performs a coarse search: calculating the distance between the query vector and the preset cluster centers in the index, and locking in the closest... Clusters, such as =5. This step significantly narrows the search scope. Subsequently, within these selected clusters, the unit performs a refined search. The similarity between the query vector and the summary vector of each document within the cluster is calculated. The similarity calculation uses the cosine similarity formula. Specifically, the dot product of the summary vector and query vector for document DOC_2024_001 is 0.85, while the dot product for another document, DOC_2023_089, is 0.72. The unit sorts all candidate documents in descending order based on these similarity scores and selects the top K documents as the result. Here, K is a configurable system parameter, set to 50 or 100, designed to ensure sufficient candidate documents advance to the next round of selection. Finally, the unit outputs a semantic recall candidate set, whose data structure is a list containing document IDs and their semantic relevance scores, for example, [(DOC_2024_001, 0.85), (DOC_2023_089, 0.72),...]. The documents in this collection are highly semantically aligned with the user's query intent. Even if the document does not directly contain the term "read / write throughput," it can still be successfully recalled as long as its summary includes descriptions such as I / O performance or data transfer rate.

[0047] Simultaneously, keyword recall unit 142 is launched in parallel. This unit receives a list of query terms generated by the query understanding module and retrieves them using a keyword weight index. This index is an inverted index structure that stores the precise weight of each keyword in each document. The retrieval algorithm of this unit is based on a variant of the BM25 algorithm, combined with keyword weights specific to this scheme. BM25 is a classic probabilistic retrieval model used to evaluate the literal matching degree between a query and a document. For each term contained in query Q... Its relationship with the document The formula for calculating the relevance score is as follows:

[0048]

[0049] In this formula, It is the keyword matching score. It is a word element Inverse document frequency (IDF) reflects the rarity of a word. Specialized terms like "distributed" and "throughput" have high IDF values. It is a word element In the document Word frequency in. It is the length of document D. It is the average document length of the entire corpus. and It is a moderating factor, preset based on rules of thumb in the field of information retrieval, such as setting it to... =1.5, =0.75. Specifically, this scheme introduces... This is the word weight directly obtained from the keyword weight index, calculated based on multidimensional features and graph iteration, such as the weight of "distributed" calculated in the previous step as 1.85. This term, acting as a multiplicative factor or additive bias, greatly enhances the influence of core keywords on the final score. This unit traverses each word in the query term list, searching the inverted index for a list of documents containing that word. For example, for the term "distributed," documents DOC_2024_001 with a weight of 1.85 and DOC_2023_089 with a weight of 0.92 are found; for the term "throughput," DOC_2024_001 with a weight of 0.95 is found. This unit sums up the scores of all matching terms. For document DOC_2024_001: Hitting "Distributed": BM25 base score × 1.85 = 5.2; Hitting "Storage": BM25 base score × 1.76 = 4.8; Hitting "Throughput": BM25 base score × 0.95 = 3.1, Keyword Matching Score =5.2 + 4.8 + 3.1 = 13.1. Similarly, this unit calculates the score of all documents that match this set of query terms, sorts them in descending order, and selects the top K, for example, K=50. Finally, this unit outputs a keyword recall candidate set, for example, [(DOC_2024_001, 13.1), (DOC_2019_003, 11.5),...]. The documents in this set are highly matched to the user's query in word order, ensuring that documents containing specific terms are not missed.

[0050] In particular, while existing dual-path recall architectures balance generalization and accuracy to some extent by executing semantic and keyword retrieval in parallel, these two processes typically operate independently, lacking information interaction. This separation prevents simple keyword recall from bridging the semantic gap. For example, when a user searches for the read / write throughput of a distributed storage system, the system struggles to recall highly relevant technical documents that only contain near-synonymous technical terms like I / O transfer rates or data bandwidth. Furthermore, a fixed Top-K recall strategy (e.g., recalling 50 documents from each path) lacks adaptability to query intent. For highly conceptual queries like "how to improve team collaboration efficiency," the system should obtain more results from the semantic path, while for precise error code queries, it should focus on the keyword path. Maintaining a fixed ratio leads to a large number of low-relevance, noisy documents being introduced into the non-dominant path, wasting computational resources and reducing the accuracy of subsequent ranking. Therefore, the step of implementing joint recall optimization of query expansion and resource adaptation is to introduce semantic understanding in the recall stage to enhance the breadth of keyword matching, and to use dynamic fusion weights to intelligently allocate the document quantity quota of the two recall paths, thereby achieving complementary advantages and discriminative combination in the two recall processes, fundamentally improving the recall and precision of the candidate set.

[0051] Based on this, in a feasible preferred technical solution, the dual-path recall module 140 is used for:

[0052] Based on the query vector, semantic probing-based query expansion is performed in the summary vector index to obtain expanded query terms. In specific implementation, this process first performs semantic probing-based query expansion to leverage the strong semantic understanding capabilities of vector retrieval, automatically mining potentially relevant synonyms or related words to the user's query, thus overcoming the limitation of keyword retrieval which can only perform literal matching. Specifically, the system uses the query vector generated by the query understanding module to perform a small-scale, high-precision semantic probe in the summary vector index. For example, the system sets the probe count to 3, retrieving the 3 document IDs most relevant to the query on distributed storage performance optimization, denoted as the probe document set. Next, the system... For each document ID, the keyword weight index is retrieved, and the top-N keywords with the highest weight in each document are extracted (e.g., N=2 keywords). If a detected document contains high-weight keywords such as "fragmentation" and "copy," these words, although not directly appearing in the user's query, are highly relevant to the topic. The system merges and deduplicates these extracted expanded terms with the original query terms to obtain expanded query terms. Through this step, the system can recall more documents using different technical terms or wording but with the same theme, significantly improving the recall rate of the keyword pathway.

[0053] Based on the dynamic fusion weights and the total recall base, the number of documents to be recalled for the semantic path and the keyword path are calculated separately to obtain the semantically recalled document count and the keyword recalled document count. Subsequently, the document count is calculated based on the dynamic fusion weights and the total recall base. This step aims to break the rigidity of a fixed Top-K, dynamically allocating computing resources according to the user's intent and reducing noise introduced by unnecessary paths. Specifically, the system reads the pre-set total recall base, such as 100 documents, and obtains the dynamic fusion weight generated for the current query, denoted as... For example, a value of 0.77 calculated in the previous steps indicates a strong semantic bias in the query. The system calculates the recall count for both paths using the following formulas: number of semantically recalled documents. Number of documents retrieved by keywords This process ensures that for concept-based queries, the system allocates the majority of the recall quota to the semantic pathway, while for exact match queries, it automatically favors the keyword pathway.

[0054] Based on the number of semantically recalled documents, the query vector is searched for similarity in the summary vector index to obtain a semantic recall candidate set. Based on the calculated quota, the system then performs a semantic similarity search based on the number of semantically recalled documents. This step aims to maximize the capture of relevant documents at a macro-semantic level by utilizing the allocated optimized quota. The system performs a K-nearest neighbor search on the query vector in the summary vector index, where the K value is dynamically calculated to be 77. The system returns the 77 documents with the highest similarity and their scores, constituting the semantic recall candidate set. Due to the increased quota, some potentially relevant long-tail documents that might otherwise have been truncated to a fixed Top-50 list are retained.

[0055] Based on the number of documents retrieved by keywords, the expanded query terms are searched for similarity in the keyword weight index to obtain a keyword recall candidate set. Parallel or immediately following, the system performs a keyword similarity search based on the number of documents retrieved by keywords. This step is implemented to achieve precise capture of detailed information by utilizing the expanded vocabulary and precise quotas. The system retrieves the expanded query terms generated in the first step, containing words such as "fragmentation" and "copy," in the keyword weight index. The retrieval truncation threshold is set to a dynamically calculated 23. The system returns the top 23 documents based on keyword matching scores (e.g., weighted BM25), forming the keyword recall candidate set. Due to the introduction of expanded terms, this set not only includes literal matching documents but also semantically related documents; and due to the limited number, it effectively filters out low-quality documents containing only general terms. Through these steps, the entire recall system demonstrates strong robustness to different query intentions, and the high-quality candidate set output lays a solid foundation for subsequent fusion and re-ranking.

[0056] Specifically, the fusion and re-ranking module 150 is used to dynamically fuse and re-rank the semantic recall candidate set and the keyword recall candidate set based on dynamic fusion weights to obtain the final ranked document list. It should be understood that after obtaining the two sets of candidate documents—semantic recall and keyword recall—although these two sets of documents represent the system's best filtering results for user query intent and literal matching, respectively, there is an inherent heterogeneity between them. The document score for semantic recall is based on vector cosine similarity, with a value range between [-1, 1], and often clustered in the high score range; while the document score for keyword recall (such as BM25) is based on a probabilistic statistical model, with a value range of non-negative real numbers and no fixed upper bound, and is greatly affected by document length and word frequency. If these two sets of scores are directly added, the keyword score, with its larger numerical range, will completely dominate the ranking result, rendering semantic information ineffective. Furthermore, duplicate documents inevitably exist in both candidate sets, and direct merging will lead to redundant results. More importantly, the preceding steps have already calculated the dynamic fusion weights for the current query intent (e.g., semantics accounts for 77%), and this intelligent decision needs to be accurately applied to the final ranking. Therefore, this application uses dynamic fusion weights to dynamically fuse and re-rank the semantic recall candidate set and the keyword recall candidate set. Through standardized data cleaning and mathematical normalization, the dimensional differences between different scoring systems are eliminated, and dynamic weights are used to comprehensively evaluate the value of each candidate document, thereby generating a high-quality ranked list that accurately hits the user intent while also considering detailed matching.

[0057] In a feasible technical solution, the fusion and re-ranking module 150 includes: a candidate set merging and deduplication unit 151, used to retrieve candidate sets and deduplicate them from the semantic recall candidate set and the keyword recall candidate set to obtain an original score mapping table; a score normalization unit 152, used to normalize the scores within the candidate set of the original score mapping table to obtain a normalized score mapping table; a dynamic weighted fusion unit 153, used to calculate the final score of each record in the normalized score mapping table based on dynamic fusion weights to obtain a final score mapping table; and a ranking output unit 154, used to generate a final ranked document list based on the final score mapping table.

[0058] In a feasible technical solution, the processing of the fusion re-ranking module 150 is as follows: The first step of the candidate set merging and deduplication unit 151 is to extract document IDs and create a union. This unit traverses two lists: the semantic recall candidate set and the keyword recall candidate set. Recalling the previous example, the semantic recall candidate set might contain documents and scores like: [(DOC_2024_001, 0.85), (DOC_2023_089, 0.72), (DOC_2024_005, 0.68)]. These scores are calculated based on cosine similarity. The keyword recall candidate set might contain: [(DOC_2024_001, 13.1), (DOC_2019_003, 11.5), (DOC_2023_089, 8.4)]. These scores are calculated based on the weighted BM25 algorithm. All document IDs are extracted. For example, if the semantic set ID is {DOC_2024_001, DOC_2023_089, DOC_2024_005}, and the keyword set ID is {DOC_2024_001, DOC_2019_003, DOC_2023_089}, a unified candidate set {DOC_2024_001, DOC_2023_089, DOC_2024_005, DOC_2019_003} without duplicate elements is generated through a set union operation. This step ensures that documents have a chance to participate in the final competition regardless of the path through which they are recalled. Next, this unit constructs a score mapping table. A dictionary structure named the original score mapping table is initialized. This unit iterates through each document ID in the unified candidate set and backtracks to find its score in the two original candidate sets. For document DOC_2024_001, it appears in both sets, so its score is recorded as (0.85, 13.1). For document DOC_2023_089, it also appears in both sets, recorded as (0.72, 8.4). For document DOC_2024_005, it only appears in the semantic recall set, meaning its keyword match is extremely low (not in the Top-K). To avoid losing its semantic value, the system pads its keyword score to zero, recording it as (0.68, 0.0). For document DOC_2019_003, it only appears in the keyword recall set, meaning its overall semantics are far from the query vector. Similarly, its semantic score is padded to zero (or set to a very small threshold such as 0.0), recorded as (0.0, 11.5). The final generated original score mapping table is as follows: {"DOC_2024_001":(0.85,13.1),"DOC_2023_089":(0.72,8.4),"DOC_2024_005":(0.68,0.0),"DOC_2019_003":(0.0,11.5)}.At this point, the data structure has been unified, but the difference in dimensions (0.85 vs 13.1) still exists, which will seriously interfere with the subsequent weighted calculation.

[0059] Subsequently, the score normalization unit 152 takes over. This unit aims to map the two sets of scores to the same interval [0,1] to achieve fair weighted fusion. This unit first extracts the score lists. All semantic relevance scores from the original score mapping table are extracted to form a list [0.85, 0.72, 0.68, 0.0], and all keyword matching scores are extracted to form a list [13.1, 8.4, 0.0, 11.5]. Next, the extreme values ​​are calculated. For the semantic score list, the maximum value is 0.85, and the minimum value is 0.0. For the keyword score list, the maximum value is 13.1, and the minimum value is 0.0. It is worth noting that if the minimum value in the list is 0 (i.e., there are cases of no recall), then 0 is directly used as the minimum value; if all documents are recalled using both methods, then the actual minimum score is used. Finally, normalization calculation is performed. This unit iterates through each record in the original score mapping table, applying the Min-Max Scaling formula: For document DOC_2024_001: Normalized semantic score = (0.85-0.0) / (0.85-0.0) = 1.0. Normalized keyword score = (13.1-0.0) / (13.1-0.0) = 1.0. For document DOC_2023_089: Normalized semantic score = (0.72-0.0) / (0.85-0.0) = 0.847. Normalized keyword score = (8.4-0.0) / (13.1-0.0) = 0.641. For document DOC_2024_005: Normalized semantic score = 0.8. Normalized keyword score = 0.0. For document DOC_2019_003: Normalized semantic score = 0.0. Normalized keyword score = 0.878. After the calculation is completed, the unit stores these dimensionless scores into a normalized score mapping table. The table structure is as follows: {"DOC_2024_001":(1.0,1.0),"DOC_2023_089":(0.847,0.641),"DOC_2024_005":(0.8,0.0),"DOC_2019_003":(0.0,0.878)}.

[0060] The dynamic weighted fusion unit 153 first obtains the normalized score mapping table from the previous-level score normalization unit. Simultaneously, this unit retrieves the dynamic fusion weights generated for the current query, "How to optimize the read / write throughput of a distributed storage system?", from the query understanding module. In the previous steps, this weight was predicted to be 0.77, meaning the system considers the query more semantically oriented; therefore, the semantic score should account for 77% of the weight, while the keyword score should account for 23% (i.e., 1-0.77=0.23). Next, this unit iterates through each record in the normalized score mapping table and applies the weighted summation formula to calculate the final comprehensive score for each document. The formula is as follows:

[0061]

[0062] In this formula: This is the final overall score for document d, used to determine the ranking. This is the dynamic fusion weight, which is 0.77 in this example. and These are the normalized semantic relevance score and keyword matching score, respectively. The specific calculation process is as follows: For document DOC_2024_001: =0.77×1.0+0.23×1.0=1.0 This document is the strongest in both semantics and keywords, and should rightfully receive full marks. Regarding document DOC_2023_089: =0.77×0.847+0.23×0.641=0.7996. For document DOC_2024_005, the semantics are strong but the keywords are weak. =0.77×0.8+0.23×0.0=0.616. Although it didn't match the keyword (possibly because it used I / O rate instead of throughput), it still achieved a good score because the current query was judged to have heavy semantics (with a weight as high as 0.77). For document DOC_2019_003, the keyword is strong but the semantics are weak: =0.77×0.0+0.23×0.878=0.2019. Although it contains many keywords (which may be simple keyword stuffing or contain irrelevant context), its final score is significantly suppressed because its semantic vector is far removed from the query and the current query strategy emphasizes semantics. This accurately reflects the adjustment role of dynamic weights: in the context of concept queries, it suppresses documents that are only hit by keywords but are semantically irrelevant. After the calculation is completed, the unit stores the final score of each document in the final score mapping table, with the structure {"DOC_2024_001":1.0,"DOC_2023_089":0.7996,"DOC_2024_005":0.616,"DOC_2019_003":0.2019}.

[0063] Finally, sorting output unit 154 takes over. Its main task is to transform this unordered mapping table into an ordered list for front-end display or subsequent use by the question-and-answer module. This unit performs the sorting operation, based on... The values ​​are used to sort all entries in descending order. The sorting results are as follows: 1. DOC_2024_001 (score 1.0), 2. DOC_2023_089 (score 0.7996), 3. DOC_2024_005 (score 0.616), 4. DOC_2019_003 (score 0.2019). Subsequently, this unit extracts the document ID list, generating ['DOC_2024_001', 'DOC_2023_089', 'DOC_2024_005', 'DOC_2019_003']. This ordered list is the final sorted document list. This list not only accurately places the most relevant documents first but also cleverly balances the relationship between semantics and keywords. For example, although DOC_2024_005 did not have a keyword match, it ranked higher than DOC_2019_003, which had a higher keyword match, due to its strong semantic relevance. This perfectly meets the expectations of users who are looking for exploratory queries such as "how to optimize...".

[0064] Specifically, the local question-and-answer generation module 160 is used to obtain the final search results by processing the final sorted document list and user query through a local question-and-answer large language model. The final search results include direct answers and cited sources. In other words, in traditional enterprise knowledge retrieval scenarios, after obtaining a list of search results, users often face a heavy workload of information filtering and integration. Faced with hundreds or thousands of technical documents, complex tables, and lengthy meeting minutes, users have to click on links one by one, read the full text, and piece together the answer to their specific question in their minds. This document-centric interaction mode is not only inefficient but also prone to overlooking key details due to information overload. Especially when users ask open-ended questions requiring comprehensive reasoning, such as how to solve... or what the reasons are, a simple document list cannot directly provide a definite conclusion and lacks clear traceability of the answer's source, making it difficult to meet the stringent requirements of enterprise-level applications for information accuracy and traceability. Therefore, this application uses a local question-answering large language model to obtain the final search results by sorting the final document list and user queries. This is to leverage the powerful natural language understanding and summarization capabilities of generative artificial intelligence to transform the retrieved fragmented, highly relevant documents into coherent, accurate, and verifiable direct answers. This achieves a paradigm shift from searching for documents to searching for answers, greatly improving the efficiency and credibility of knowledge acquisition.

[0065] In a feasible technical solution, the local question-answering generation module 160 processes the following: This module receives the final sorted document list from the fusion and reordering module and the user query from the query understanding module, and generates the answer through reasoning using a locally deployed large language model. The implementation process will be described in detail below.

[0066] The local question-answering generation module 160 first performs a context construction operation. In the preceding steps, the fusion and reordering module has already output a dynamically weighted and optimized ordered document list. For the query "How to optimize the read and write throughput of a distributed storage system?", this module first extracts the top three documents by ID from the list based on a preset truncation parameter, such as 3. The top three documents are: DOC_2024_001 (overall score 1.0, content related to the 2023 R&D Center Work Report), DOC_2023_089 (overall score 0.7996, content related to distributed storage architecture specifications), and DOC_2024_005 (overall score 0.616, content related to I / O performance tuning guidelines). Subsequently, this module uses these IDs to access the plain text database to obtain the corresponding full text or highly relevant summary fragments of the documents. To adapt to the input context window limitations of large language models, such as 4096 or 8192 tokens, this module may slice or summarize excessively long documents, retaining only the paragraphs most relevant to the query terms, such as optimization and throughput semantics. Next, the module assembles these text fragments according to a specific prompt template. The template structure is designed as follows: "Please answer the user's question based on the following reference documents. Please explicitly cite the document number in your answer. If there is no answer in the reference document, please state that you cannot answer.\n\nReference Document 1 (DOC_2024_001): ...[Document Content]...\nReference Document 2 (DOC_2023_089): ...[Document Content]...\n\nUser Question: How to optimize the read and write throughput of a distributed storage system?". In this way, unstructured document content is transformed into a structured model input context.

[0067] Next, the answer generation phase begins. This module inputs the constructed complete prompts, including context and the user's query, into a locally deployed question-answering large language model. The emphasis on "local" is because enterprise knowledge documents often contain sensitive and confidential information that cannot be uploaded to public cloud APIs for processing. This local LLM is based on a Transformer architecture decoder-only model (such as private deployment versions of LLaMA, Baichuan, or Qwen). Its specific architecture consists of dozens of stacked Transformer decoder blocks, each containing a multi-head self-attention mechanism and a feedforward neural network. The model's parameters (weight matrix and bias terms) are obtained during the pre-training phase through self-supervised learning on a massive general corpus of trillions of tokens, minimizing prediction errors using backpropagation. Furthermore, to enhance the model's ability to follow instructions and answer based on context, it also undergoes supervised fine-tuning and reinforcement learning with human feedback.

[0068] During processing, the model performs an autoregressive generation task. The model receives an input sequence of prompt tokens and uses an attention mechanism to calculate the correlation strength between the query terms and content at different locations in the reference document, focusing on key information such as caching strategies, asynchronous I / O, and network bandwidth. Then, the model predicts the output tokens one by one, generating coherent natural language text. For the above query, the model might generate the following direct answer: According to the reference document, measures to optimize the read / write throughput of a distributed storage system mainly include: 1. Implementing asynchronous I / O scheduling strategies to reduce disk wait time [DOC_2024_005]; 2. Upgrading the network transmission protocol to RDMA to reduce transmission latency [DOC_2023_089]; 3. Enabling metadata caching on the client side to reduce the pressure on the master node [DOC_2024_001]. This answer not only directly addresses the user's optimization intent but also uses square brackets to indicate the source of evidence for each suggestion, thus putting the facts into practice. To verify the feasibility of the above question-answering generation logic at the practical technical level, in the code implementation of this embodiment, the system uses the full-text database (i.e., the `full_text` variable in the code) as context input and calls a locally deployed pre-trained question-answering model. When a specific question, "What role does DeepSeek play in investment research?" is posed to the local knowledge base, the model infers based on the local database as its knowledge source, returning not only the accurate question-answer content "significantly improves the efficiency of daily investment research work," but also the confidence score of the answer (score: 0.151...) and the contextual position index of the answer in the original document (start: 18460, end: 18474). This structured output data demonstrates that the model has the ability to accurately locate knowledge points and provide quantifiable evaluation answers, providing a solid data foundation for the system's subsequent answer quality filtering and citation tracing.

[0069] After generation, this module integrates the results. It parses the text stream output by the LLM, separating the direct answer and citation markers. The system verifies whether citation markers, such as [DOC_2024_001], truly exist in the input context to prevent the model from creating illusionary citations. The verified direct answer, along with the original citation sources (document titles, authors, links, and other metadata), is encapsulated into a final search result object. This object contains two core fields: the generated text and a list of cited source documents.

[0070] Finally, this module handles page rendering. The final search results are formatted into a front-end renderable data structure (such as JSON). On the final results page returned to the user, the generated direct answer is prominently displayed at the top as an AI summary or intelligent response, allowing users to grasp the core conclusions without having to consult documents. Below the direct answer, the system still displays the complete final sorted document list, allowing users to view the original text when further research is needed. For example, after seeing the "Enable metadata caching" suggestion, a user can click the DOC_2024_001 link below to directly jump to the specific section of that document to view detailed configuration parameters. This perfectly balances the convenience of information retrieval with the rigor of knowledge, providing users with an efficient and reliable search experience.

[0071] In summary, a knowledge document search engine 100 based on word vectors and context summarization, as described in this application, addresses the technical problems of rigid weight allocation in existing hybrid retrieval technologies, which cannot adapt to changing query intentions, and the dilution of key information caused by vectorization of long documents. First, multimodal parsing is performed on the original heterogeneous documents. A keyword weight index is constructed by calculating the multidimensional features of the documents, and a summary vector index is constructed using context summarization generation technology. This balances the accuracy of document details and core semantics at the index layer. During the retrieval stage, the core lies in introducing a query understanding and dynamic weight generation mechanism. By extracting the linguistic features of user queries to deeply analyze query intentions (such as precise search or concept exploration), dynamic fusion weights are predicted and generated in real time. This mechanism can adaptively adjust the proportion of semantic recall and keyword recall in the final ranking according to the query type, effectively solving the one-size-fits-all defect of traditional static fusion strategies. Finally, it combines with a local large language model to generate accurate question answers, significantly improving the recall and precision of searches in complex scenarios.

Claims

1. A knowledge document search engine based on word vectors and contextual summarization, characterized in that, include: The original knowledge document set acquisition module is used to acquire the original knowledge document set; The multimodal parsing and index building module is used to perform multimodal document parsing on the original knowledge document set to obtain the summary vector index and keyword weight index; The query understanding module is used to analyze the query intent and generate dynamic weights from the acquired user queries to obtain query vectors, query terms, and dynamic fusion weights. It includes: a query preprocessing unit, which performs lexicalization and vectorization on the acquired user queries to obtain query terms and query vectors; a query feature extraction unit, which extracts linguistic features from the query terms and user queries to obtain query feature vectors, where the query feature vectors include query length, average inverse document frequency, entity word ratio, and whether it is an interrogative sentence structure; and a dynamic fusion weight prediction unit, which inputs the query feature vectors into a pre-trained lightweight regression model to obtain dynamic fusion weights. The dual-path recall module is used to perform parallel dual-path recall of query vectors and query terms based on the summary vector index and the keyword weight index to obtain semantic recall candidate sets and keyword recall candidate sets. The fusion and re-ranking module is used to dynamically fuse and re-rank the semantic recall candidate set and the keyword recall candidate set based on dynamic fusion weights to obtain the final ranked document list. The local question-and-answer generation module is used to process the final sorted document list and user query through a local question-and-answer large language model to obtain the final search results, which include direct answers and cited sources.

2. The knowledge document search engine based on word vectors and contextual summarization according to claim 1, characterized in that, The multimodal parsing and indexing construction module includes: The document parsing and standardization unit is used to perform heterogeneous document parsing and content standardization on the original knowledge document set to obtain a plain text database. A semantic vectorization unit for abstract generation is used to perform contextual abstract generation and semantic vectorization on a plain text database to obtain a set of abstract-vector pairs. The keyword weight calculation unit is used to calculate keyword weights in a plain text database to obtain a keyword weight database. The index generation unit is used to generate summary vector indexes and keyword weight indexes based on the keyword weight database and the summary-vector pair set.

3. The knowledge document search engine based on word vectors and contextual summarization according to claim 2, characterized in that, The keyword weight calculation unit includes: The multidimensional feature calculation subunit is used to perform multidimensional feature calculation on each candidate word in the plain text database based on the corpus to obtain a candidate word feature database. The multidimensional features include average information entropy, part-of-speech features and positional features. The integrated initial weight calculation subunit is used to calculate the integrated initial weight of each candidate word in the candidate word feature database to obtain the word initial weight database; The weighted word graph iterative subunit is used to construct a weighted word graph structure based on the initial word weight database and the plain text database, and to iteratively calculate the weighted word graph structure to obtain the final word weight database. The keyword extraction subunit is used to extract and store Top-K keywords from the final word weight database to obtain a keyword weight database.

4. The knowledge document search engine based on word vectors and contextual summarization according to claim 3, characterized in that, The comprehensive initial weight calculation subunit is used to calculate the comprehensive initial weight of each candidate word in the candidate word feature database using the following formula: ;in, , and Candidate words Average information entropy, part-of-speech features, and positional features, and These are the weight coefficients for average information entropy, part-of-speech features, and positional features, respectively. Candidate words The overall initial weights.

5. The knowledge document search engine based on word vectors and contextual summarization according to claim 1, characterized in that, The dual-path recall module includes: The semantic recall unit is used to perform a K-nearest neighbor search on the query vector in the summary vector index to obtain a semantic recall candidate set. The keyword recall unit is used to search for query terms in the keyword weight index to obtain a keyword recall candidate set.

6. The knowledge document search engine based on word vectors and contextual summarization according to claim 1, characterized in that, The fusion reordering module includes: The candidate set is combined with the deduplication unit to retrieve candidate sets and deduplications from the semantic recall candidate set and the keyword recall candidate set to obtain the original score mapping table. The score normalization unit is used to normalize the scores within the candidate set of the original score mapping table to obtain a normalized score mapping table. The dynamic weighted fusion unit is used to calculate the final score of each record in the normalized score mapping table based on the dynamic fusion weights to obtain the final score mapping table. The sorting output unit is used to generate a final sorted list of documents based on the final score mapping table.

Images