Hybrid retrieval method, system and equipment based on multi-algorithm fusion and storage medium

Abstract

The invention provides a mixed retrieval method, system and device based on multi-algorithm fusion and a storage medium, and the method comprises the steps: obtaining various types of document data, preprocessing the document data, and obtaining plain text data; segmenting the plain text data into a plurality of pieces of block node data, and generating a plurality of derivative problems for each piece of block node data by using a large language model; carrying out vectorization coding on the block node data and the derivation problem, and storing the block node data and the derivation problem in a vector database; receiving a user query question, and generating a plurality of related preset questions for the query question by using the large language model; converting the query question and the preset question into a vector form, executing keyword retrieval and vector retrieval, and obtaining corresponding candidate results in a vector database; and carrying out weighted fusion on the candidate results through a weighted reciprocal rearrangement algorithm to obtain most relevant retrieval data. According to the method, the efficiency of information retrieval and the correlation of results can be improved, and the retrieval requirement of a user in a complex semantic scene is met.

Description

Technical Field

[0001] The present application relates to the field of hybrid retrieval technology, and in particular to a hybrid retrieval method, system, device and storage medium based on multi-algorithm fusion. Background Art

[0002] With the rapid development of information technology, information retrieval technology plays a vital role in various fields. It can help users quickly and accurately find the information they need from massive amounts of data, effectively saving time and energy. However, existing retrieval methods have some significant limitations: First, traditional search methods overly rely on word frequency statistics and precise word matching, resulting in significant deficiencies in semantic understanding. For example, when faced with semantically similar words like "car" and "vehicle," or complex semantic relationships like "apple" and "apple," traditional methods struggle to accurately identify and match them, leading to inaccurate search results. Furthermore, traditional search methods often fail to accurately perform well on long-tail queries (i.e., queries with low-frequency word combinations) and queries with complex intent (e.g., queries requiring multiple hops of reasoning to accurately answer).

[0003] Second, pre-trained models perform poorly in processing long texts and in domain transfer. When document length exceeds a certain limit, such as 512 tokens, the processing power of pre-trained models decreases significantly. Furthermore, when migrating from general corpus to specialized domains, the performance of pre-trained models also degrades significantly.

[0004] In view of this, the present invention proposes a hybrid retrieval method, system, device and storage medium based on multi-algorithm fusion, which can improve the efficiency of information retrieval and the relevance of results, and meet the retrieval needs of users in complex semantic scenarios. Summary of the Invention

[0005] In order to solve the problems of inaccurate retrieval results in the existing technology and poor performance of pre-trained models in long text processing and domain migration, the present invention provides a hybrid retrieval method, system, device and storage medium based on multi-algorithm fusion to solve the above technical defects.

[0006] In a first aspect, the present invention proposes a hybrid retrieval method based on multi-algorithm fusion, which comprises the following steps: S1. Obtain various types of document data, pre-process the document data, and obtain plain text data; S2. Split the plain text data into multiple block node data and use the large language model to generate multiple derivative questions for each block node data; S3, vectorize and encode the block node data and derived questions, and store them in the vector database; S4. Receive a user query and use the large language model to generate multiple related preset questions for the query; convert the query and its preset questions into vector form, perform keyword search and vector search, and obtain corresponding candidate results from the keyword library and vector database; S5. Perform weighted fusion on the candidate results through the weighted inverse reordering algorithm to obtain the most relevant search data.

[0007] Preferably, in step S4, a user query is received, a large language model is used to generate multiple related preset questions for the query, the query and the preset questions are converted into vector form, keyword search and vector search are performed, and corresponding candidate results are obtained from the vector database, which specifically includes the following sub-steps: S41. Receive a user query question and use the large language model to generate multiple relevant preset questions for the query question; S42, converting the query question and the preset question into vector form, and calculating the cosine similarity between the query question and the preset question and the data in the vector database, respectively, to obtain multiple sets of similarity results; S43. Sort each group of similarity results from high to low according to similarity, and select the first k data segments in each group as candidate results, where k is a preset natural number.

[0008] Preferably, in step S5, the candidate results are weighted and integrated by a weighted inverse rearrangement algorithm to obtain the most relevant search data, which specifically includes the following sub-steps: S51, determining whether there is an identical information segment in the candidate results; If there are identical information fragments, the inverse rearrangement results of the corresponding data fragments are added together to obtain a data result set; If there is no identical information fragment, the information fragments in all candidate results are taken as the data result set; S52. Extracting data segments that meet the threshold from the data result set according to a preset threshold; S53. Using the score calculation formula, calculate the final hybrid retrieval score for the extracted data segments according to the weight threshold. The score calculation formula is:

[0009] In the formula, score(d) represents the mixed retrieval score, is a constant that controls the score weight, is the weight of different retrieval methods k, The similarity results under different retrieval methods k are ranked from high to low; k represents different retrieval algorithms; is the ranking position of text d retrieved by the i-th retrieval algorithm; And determine the most relevant retrieval data based on the final hybrid retrieval score.

[0010] Preferably, in step S1, the document data includes HTML format data, and preprocessing the HTML format data to obtain plain text data specifically includes: Locate and extract specific tag content based on tag parameters for HTML format data, including paragraph tags, first-level title tags, and second-level title tags; After removing redundant tag contents irrelevant to specific tag contents from HTML format data, plain text data is obtained.

[0011] Preferably, in step S1, the document data includes data in JSON format, and preprocessing the JSON format data to obtain plain text data specifically includes:

[0012] Recursively split nested objects and array elements of JSON format data according to the hierarchical relationship, and parse each independent object or array element into a single semantic node; If the field value of an independent object is a complex type, the independent object will be split into independent JSON nodes; Based on single semantic nodes and JSON independent nodes, semantic information is extracted to generate plain text data.

[0013] Preferably, in step S1, each type of document data includes Markdown format data, and preprocessing the Markdown format data to obtain plain text data specifically includes: Divide Markdown format data into different levels of title nodes according to title symbols, associate the content corresponding to the title with the title node, and split list items into independent Markdown nodes; Integrate the title node and the Markdown independent node to generate plain text data.

[0014] Preferably, the hybrid retrieval method based on multi-algorithm fusion proposed by the present invention further includes: The most relevant search data obtained in step S5 and the user query question are input into the large language model to generate a preliminary answer; Use the predefined sensitive word library to perform keyword matching scans on the preliminary answers, filter out sensitive words, and obtain intermediate answers; Through regular expression matching and named entity recognition technology, private information in the intermediate answers is identified and shielded, desensitized, and the optimized final answer is obtained, which is then fed back to the user.

[0015] In a second aspect, the present invention proposes a hybrid retrieval system based on multi-algorithm fusion, the system comprising: An acquisition module is configured to acquire various types of document data, pre-process the document data, and obtain plain text data; The preprocessing module is configured to split the plain text data into multiple block node data and use the large language model to generate multiple derivative questions for each block node data; A vectorized encoding module configured to vectorize the block node data and derived questions and store them in a vector database; The retrieval module is configured to receive user queries and generate multiple related pre-set questions for the query using a large language model; convert the query and its pre-set questions into vector form, perform keyword search and vector search, and obtain corresponding candidate results from the keyword library and vector database; The output module is configured to perform weighted fusion on the candidate results through a weighted inverse rearrangement algorithm to obtain the most relevant retrieval data.

[0016] In a third aspect, the present invention proposes a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements any of the steps of the hybrid retrieval method based on multi-algorithm fusion described above.

[0017] In a fourth aspect, the present invention proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned hybrid retrieval methods based on multi-algorithm fusion.

[0018] Compared with the prior art, the present invention has the following advantages: (1) Significantly improve retrieval accuracy: By integrating multiple retrieval technologies such as keyword retrieval, vector retrieval, and weighted reciprocal reranking algorithm, and giving full play to the advantages of each algorithm, the present invention can more accurately locate data that is highly relevant to user queries, effectively solving the shortcomings of traditional retrieval methods in semantic understanding due to their reliance on word frequency statistics and precise word matching, and greatly improving the retrieval accuracy in complex semantic scenarios.

[0019] (2) Wide applicability: The hybrid retrieval method proposed in this invention is applicable to the retrieval of various types of document data (such as HTML, JSON, Markdown, etc.). It has wide applicability and practicality and can meet the information retrieval needs in different scenarios.

[0020] (3) Enhanced long text processing capabilities: For long text data, we use derivative question generation and summary generation technology based on a large language model to split it into multiple block node data for processing. This avoids the performance degradation of the pre-trained model when processing long texts with more than 512 tokens, ensuring the efficiency and accuracy of long text retrieval.

[0021] (4) Improve domain adaptability: By constructing targeted indexes and adopting a retrieval strategy that integrates multiple algorithms, the present invention can better adapt to the characteristics of corpora in different fields, effectively solve the problem of poor performance of pre-trained models in domain migration, and improve the system's retrieval performance in professional fields.

[0022] (5) Ensure information security and privacy: Introduce sensitive word filtering and privacy information desensitization processing mechanisms, use predefined sensitive word libraries and advanced information recognition technology to strictly screen and process the generated answers, ensure that the search results meet the company's information security requirements, and effectively protect user privacy and corporate secrets.

[0023] (6) Optimize user experience: Starting from the user's query, generate natural, smooth and contextual preliminary answers, and feed them back to the user after multiple layers of optimization and security processing. At the same time, provide intuitive display pages, enhance the user's interactive experience with the system, and enable users to obtain the required information more efficiently and conveniently. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Other features, objects and advantages of the present application will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 is a flow chart of a hybrid retrieval method based on multi-algorithm fusion according to the present invention; Figure 2 is a schematic diagram of a hybrid retrieval system based on multi-algorithm fusion according to the present invention; Figure 3 It is a schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] The present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the relevant invention and are not intended to limit the invention. It should also be noted that, for ease of description, only portions relevant to the relevant invention are shown in the accompanying drawings.

[0026] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0027] This invention proposes a hybrid retrieval method based on multi-algorithm fusion. Figure 1 The flow chart of the hybrid retrieval method based on multi-algorithm fusion of the present invention is shown as follows: Figure 1 As shown, the method includes the following steps: S1. Obtain various types of document data, pre-process the document data, and obtain plain text data.

[0028] Among them, various types of document data include enterprise knowledge base data in formats such as HTML, JSON, and Markdown.

[0029] In the index construction phase, the present invention develops specific splitting and extraction strategies for different data types to deeply mine data semantic information. Specifically: The preprocessing steps for HTML format data are as follows: Locate and extract specific tag content based on tag parameters, including paragraph tags, first-level title tags, and second-level title tags; After removing redundant tags that are not related to specific tags in HTML format data, the pure text data is obtained. This method can significantly improve the efficiency and accuracy of data splitting.

[0030] The preprocessing steps for JSON format data are as follows: Recursively split nested objects and array elements according to the hierarchical relationship, and parse each independent object or array element into a single semantic node; If the field value of an independent object is a complex type (such as an object or array), the independent object is split into independent JSON nodes so that each node has clear and independent semantic information; Based on single semantic nodes and JSON independent nodes, semantic information is extracted to generate plain text data.

[0031] The preprocessing steps for Markdown format data are as follows: The Markdown document is divided into different levels of heading nodes based on heading symbols (e.g., "#" for a first-level heading, "##" for a second-level heading, etc.), and the corresponding content of the heading is associated with the heading node. List items are also split into independent Markdown nodes. This approach preserves the structural information of the Markdown document and relatively complete independent semantic content.

[0032] Integrate the title node and the Markdown independent node to generate plain text data.

[0033] After the above processing, all data are converted into plain text data with complete structure and clear semantics.

[0034] Continue to refer Figure 1 The hybrid retrieval method based on multi-algorithm fusion provided by the present invention further includes the following steps: S2. Divide the plain text data into multiple block node data, and use the large language model to generate multiple derivative questions for each block node data.

[0035] Specifically, the plain text data is segmented using a block size of 1024 and a context sliding window of 24 to obtain multiple independent block node data. To improve retrieval accuracy, a large language model is used to automatically generate three related derivative questions for each block node data. Furthermore, summaries can be generated for long text within the block node data.

[0036] S3 vectorizes and encodes block node data and derived questions, and stores them in a vector database to provide support for efficient retrieval and question answering.

[0037] S4. Receive user query questions and use the large language model to generate multiple related preset questions for the query questions; convert the query questions and their preset questions into vector form, perform keyword search and vector search, and obtain corresponding candidate results in the keyword library and vector database.

[0038] Among them, keyword retrieval uses the BM25 algorithm to calculate the similarity of word frequency at a fine-grained level to ensure that matching data can still be found when semantic information is insufficient. It should be understood that when the amount of text is small, the retrieval efficiency is not high, the text content changes frequently and it is difficult to summarize keywords in advance, the original text can be directly matched and retrieved. For large-scale text data, building a keyword library and establishing an index in advance can significantly improve the retrieval speed. In addition, in professional fields such as medicine and law, due to the presence of a large number of professional terms and specific expressions, building a field-specific keyword library is crucial to enhancing the RAG system's understanding and processing capabilities for professional field queries.

[0039] Vector retrieval uses NV-Embed-v2 technology to convert input and data into high-dimensional vectors, accurately locating multiple pieces of data that highly match the input, and is suitable for complex semantic queries.

[0040] It should be understood that after receiving the user's query, the system may again use the large language model to generate relevant preset questions for the query. The main purpose of this approach is to more accurately understand the user's question intent, so as to more accurately match the user's question with the data in the database. By generating preset questions, the system can expand and refine user questions from different angles, thereby improving the ability to find more relevant answers in the vector database. However, according to the actual application scenario, it is also possible to no longer use the large language model to generate relevant preset questions for the query, and only generate derivative questions in the database construction stage, and use other methods to handle problems when the user queries.

[0041] In a specific embodiment, step S4 specifically includes the following sub-steps: S41. Receive a query question input by a user through a system interface, and use a large language model to generate multiple relevant preset questions for the query question; S42, converting the query question and the preset question into vector form, and calculating the cosine similarity between the query question and the preset question and the data in the vector database, respectively, to obtain multiple sets of similarity results; S43. Sort each group of similarity results from high to low according to similarity, and select the first k data segments in each group as candidate results, where k is a preset natural number.

[0042] Continue to refer Figure 1 The hybrid retrieval method based on multi-algorithm fusion provided by the present invention further includes the following steps: S5. Perform weighted fusion on the candidate results through the weighted inverse reordering algorithm to obtain the most relevant search data.

[0043] In a specific embodiment, step S5 specifically includes the following sub-steps: S51, judging whether there are identical information segments in the candidate results. First, the system checks whether there are identical information segments in the candidate results obtained from different retrieval methods.

[0044] If there are identical information fragments, the corresponding data fragments are reordered by adding up the inverse reordering results to obtain a data result set. This addition operation is intended to increase the weight of information fragments that perform well in multiple search methods, thereby increasing their priority in the final results.

[0045] If there is no identical information fragment, the information fragments in all candidate results are taken as the data result set.

[0046] S52: Extract data segments that meet the threshold from the data result set based on a preset threshold. This step is used to filter out information segments with low relevance or that do not meet quality requirements, thereby focusing on the most relevant and high-quality search results.

[0047] S53: Using the score calculation formula, calculate the final hybrid search score for the extracted data segments according to the weight threshold, and determine the most relevant search data based on the final hybrid search score. The score calculation formula is:

[0048] In the formula, score(d) represents the mixed retrieval score, is a constant that controls the score weight, is the weight of different retrieval methods k, The similarity results under different retrieval methods k are ranked from high to low; k represents different retrieval algorithms, and λ can be set to 0.5; is the ranking position of text d retrieved by the i-th retrieval algorithm.

[0049] In a specific embodiment, the present invention specifically considers the semantic fragmentation problem that may be caused by slicing long texts, and introduces a preset question similarity matching mechanism to solve this problem. The details are as follows: During the indexing phase, three derived preset questions {Q1, Q2, Q3} are generated for each block node data. These questions are generated by a large language model and aim to capture the semantic information of the block node data from different perspectives.

[0050] The cosine similarity between the preprocessed user query question Qp and its preset questions Q1, Q2, and Q3 and the data in the enterprise knowledge base is calculated, and the top k relevant data are selected respectively.

[0051] Qp: , Q1: , Q2: , Q3: ; The similarity results of each question are sorted from high to low and then rearranged in reverse order to further optimize the ranking of the results.

[0052] If there are identical information fragments in the first k related data of Qp, Q1, Q2, and Q3, the corresponding inverse rearrangement results are added together; if there are no identical fragments, a total of 4k information fragments are retained.

[0053] Relevant segments that meet a preset threshold are extracted and a final hybrid retrieval score is calculated with a weight of 0.5 to ensure the completeness and accuracy of the retrieved data. This series of operations not only improves retrieval accuracy but also effectively addresses the challenge of semantic coherence in processing long texts.

[0054] To integrate the results of multiple search methods, while simultaneously removing duplicate fragments and reducing the impact of fragments with low similarity, the system uses a weighted reciprocal re-ranking (PRF) method. First, based on the data indexing (including data preprocessing and other stages), a preliminary search is performed using a combination of keyword indexing and vector indexing. The similarity results from different search methods are converted into a ranking from high to low, thereby obtaining multiple ranked candidate data. The candidate data from the preliminary search is then weighted and integrated using the weighted reciprocal re-ranking (PRF) algorithm, calculating the scores of all indexed data under all methods to obtain the most relevant search data.

[0055] In a specific embodiment, the present invention proposes a hybrid search method based on multi-algorithm fusion, further comprising the following steps: The most relevant search data obtained in step S5 is combined with the user's query into a large language model (LLM) to generate a preliminary answer. The LLM plays a crucial role in this process, not only generating natural and fluent responses but also adapting tone and wording to the context to suit different conversational scenarios.

[0056] The preliminary answers are scanned for keywords using a predefined sensitive word library, filtering out sensitive terms (such as those that clearly violate corporate information security policies) to obtain intermediate answers. This step ensures that the answers meet corporate security standards and prevents the leakage of sensitive information.

[0057] To process user privacy data, regular expression matching and named entity recognition (NER) technology can be used to accurately identify and block private information in generated answers, such as personal identity information, financial data, and corporate secrets. Furthermore, private information is desensitized based on contextual information to ensure that answers remain complete and readable without leaking sensitive information.

[0058] The final answer, which has been rigorously screened and optimized, is then presented to the user. The system also provides an intuitive display page to showcase the user's question and the answer provided by the system, enhancing the user's understanding and interactive experience.

[0059] Further references Figure 2 As an implementation of the above method, the present invention provides a second embodiment of a structural diagram of a hybrid search system 200 based on multi-algorithm fusion. The system can be applied to various electronic devices. The hybrid search system 200 based on multi-algorithm fusion includes the following modules: The acquisition module 210 is configured to acquire various types of document data, pre-process the document data, and obtain plain text data; A pre-processing module 220 is configured to segment the plain text data into a plurality of block node data and generate a plurality of derivative questions for each block node data using a large language model; A vectorized encoding module 230 is configured to vectorize the block node data and the derived questions and store them in a vector database; The search module 240 is configured to receive a user query, generate multiple related pre-set questions for the query using a large language model, convert the query and its pre-set questions into vector form, perform keyword search and vector search, and obtain corresponding candidate results from the keyword library and vector database; The output module 250 is configured to perform weighted fusion on the candidate results through a weighted inverse rearrangement algorithm to obtain the most relevant search data.

[0060] In a third aspect, the present invention proposes a terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the processor implements any of the steps of the hybrid retrieval method based on multi-algorithm fusion described above.

[0061] In a fourth aspect, the present invention proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the steps of any of the above-mentioned hybrid retrieval methods based on multi-algorithm fusion.

[0062] Reference below Figure 3 , which shows a structural diagram of a computer system 300 suitable for implementing a terminal device or server of an embodiment of the present application. Figure 3 The terminal device or server shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0063] like Figure 3 As shown, computer system 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage unit 308 into a random access memory (RAM) 303. Various programs and data required for the operation of computer system 300 are also stored in RAM 303. CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0064] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, mouse, and the like; an output section 307 including a liquid crystal display (LCD), speakers, and the like; a storage section 308 including a hard disk and the like; and a communication section 309 including a network interface card such as a LAN card or a modem. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, is installed in the drive 310 as needed, so that computer programs read from the media can be installed in the storage section 308 as needed.

[0065] In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the method illustrated in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via the communication portion 309 and / or installed from removable media 311. When the computer program is executed by the central processing unit (CPU) 301, the functions defined in the methods of this application are performed. It should be noted that the computer-readable medium described herein can be a computer-readable signal medium or a computer-readable medium, or any combination thereof. The computer-readable medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable media include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium may be any tangible medium that contains or stores a program for use by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, embodying computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including, but not limited to, wireless, wire, optical cable, RF, or any suitable combination thereof.

[0066] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0067] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or operation, or can be implemented by a combination of dedicated hardware and computer instructions.

[0068] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention herein is not limited to technical solutions formed by a specific combination of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the aforementioned inventive concept. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A hybrid retrieval method based on multi-algorithm fusion, characterized in that: The following steps are involved: S1. Acquire various types of document data, pre-process the document data, and obtain plain text data; S2. Divide the plain text data into a plurality of block node data, and use a large language model to generate a plurality of derivative questions for each block node data; S3. Vectorize and encode the block node data and derived questions, and store them in a vector database; S4. Receive a user query question, generate multiple related preset questions for the query question using a large language model, convert the query question and the preset questions into vector form, perform keyword search and vector search, and obtain corresponding candidate results in the keyword library and the vector database; S5. Perform weighted fusion on the candidate results using a weighted inverse rearrangement algorithm to obtain the most relevant search data.

2. The hybrid retrieval method based on multi-algorithm fusion according to claim 1 is characterized in that: In step S4, a user query is received, and a large language model is used to generate multiple related preset questions for the query. The query and its preset questions are converted into vector form, and keyword search and vector search are performed to obtain corresponding candidate results in the vector database. The specific steps include the following: S41, receiving a user query question, and using a large language model to generate multiple relevant preset questions for the query question; S42, converting the query question and the preset question into vector form, and calculating the cosine similarity between the query question and the preset question and the data in the vector database, respectively, to obtain multiple sets of similarity results; S43. Sort each group of similarity results from high to low according to similarity, and select the first k data segments in each group as candidate results, where k is a preset natural number.

3. The hybrid retrieval method based on multi-algorithm fusion according to claim 1 is characterized in that: In step S5, the candidate results are weighted and integrated by a weighted reciprocal rearrangement algorithm to obtain the most relevant search data, which specifically includes the following sub-steps: S51, determining whether there is an identical information segment in the candidate results; If there are identical information fragments, the inverse rearrangement results of the corresponding data fragments are added together to obtain a data result set; If there is no identical information fragment, the information fragments in all the candidate results are taken as the data result set; S52. Extracting data segments that meet the threshold from the data result set according to a preset threshold; S53. Using a score calculation formula, and according to a weight threshold, a final hybrid search score is calculated for the extracted data segments. The score calculation formula is: In the formula, score(d) represents the mixed retrieval score, is a constant that controls the score weight, is the weight of different retrieval methods k, is the weight of different retrieval methods k, The similarity results under method k are ranked from high to low; k represents different retrieval algorithms; is the ranking position of text d retrieved by the i-th retrieval algorithm; and determining the most relevant search data according to the final hybrid search score.

4. The hybrid retrieval method based on multi-algorithm fusion according to claim 1 is characterized in that: In step S1, the document data includes HTML format data, and the HTML format data is preprocessed to obtain plain text data, specifically including: Locating and extracting specific tag content from the HTML format data based on tag parameters, wherein the specific tag content includes a paragraph tag, a first-level title tag, and a second-level title tag; After removing redundant tag contents irrelevant to the specific tag contents from the HTML format data, plain text data is obtained.

5. The hybrid retrieval method based on multi-algorithm fusion according to claim 1 is characterized in that: In step S1, the document data includes data in JSON format, and the JSON format data is preprocessed to obtain plain text data, specifically including: Recursively split the nested objects and array elements of the JSON format data according to the hierarchical relationship, and parse each independent object or array element into a single semantic node; If the field value of the independent object is a complex type, the independent object is split into JSON independent nodes; Based on the single semantic node and the JSON independent node, semantic information is extracted to generate plain text data.

6. The hybrid retrieval method based on multi-algorithm fusion according to claim 1 is characterized in that: In step S1, the various types of document data include Markdown format data, and the Markdown format data is preprocessed to obtain plain text data, specifically including: Dividing the Markdown format data into title nodes of different levels according to title symbols, associating the content corresponding to the title with the title node, and splitting the list items into independent Markdown nodes; The title node and the Markdown independent node are integrated to generate plain text data.

7. The hybrid retrieval method based on multi-algorithm fusion according to claim 1 is characterized in that: Also includes: The most relevant search data obtained in step S5 and the user query question are input into the large language model to generate a preliminary answer; Using a predefined sensitive word library to perform keyword matching scanning on the preliminary answer, filtering sensitive words, and obtaining an intermediate answer; Through regular expression matching and named entity recognition technology, the privacy information in the intermediate answers is identified and shielded, desensitization processing is performed, an optimized final answer is obtained, and the final answer is fed back to the user.

8. A hybrid retrieval system based on multi-algorithm fusion, characterized in that: The system comprises: an acquisition module configured to acquire various types of document data, pre-process the document data, and obtain plain text data; A preprocessing module configured to segment the plain text data into a plurality of block node data and generate a plurality of derivative questions for each block node data using a large language model; A vectorized encoding module configured to vectorize the block node data and the derived questions and store them in a vector database; A search module is configured to receive a user query question, generate multiple related preset questions for the query question using a large language model, convert the query question and the preset questions into vector form, perform keyword search and vector search, and obtain corresponding candidate results in the keyword library and the vector database; The output module is configured to perform weighted fusion on the candidate results through a weighted inverse rearrangement algorithm to obtain the most relevant search data.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the hybrid retrieval method based on multi-algorithm fusion as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the hybrid retrieval method based on multi-algorithm fusion as claimed in any one of claims 1 to 7 are implemented.

Images