Data retrieval method and electronic equipment

Abstract

The invention is suitable for the technical field of data retrieval, and provides a data retrieval method and electronic equipment. The method comprises the following steps: obtaining a query statement; a plurality of target data indexes matched with the query statement are determined from a plurality of data index sets, the data index sets correspond to data blocks, and each data index set comprises one or more data indexes; different data indexes of the same data index set are used for representing data blocks corresponding to the data index set from different semantic perspectives; on the basis of the multiple target data indexes, one or more candidate data blocks matched with the query statement are obtained through query from different semantic perspectives; and determining a target data block based on the one or more candidate data blocks. Through the method, the data retrieval quality can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data retrieval, and in particular to a data retrieval method and electronic equipment. Background Art

[0002] A Retrieval-augmented Generation (RAG) system generally involves a data retrieval process, which retrieves information related to a given query statement from a specified database (also called a knowledge base) and outputs the retrieved information to other modules of the RAG system for processing.

[0003] The existing retrieval process only matches the query statement with the data block itself in the knowledge base during retrieval, and the data retrieval quality is poor. Summary of the invention

[0004] The embodiments of the present application provide a data retrieval method and an electronic device, which can improve the quality of data retrieval.

[0005] In a first aspect, an embodiment of the present application provides a data retrieval method, comprising:

[0006] Get the query statement;

[0007] Determine, from a plurality of data index sets, a plurality of target data indexes matching the query statement, wherein the data index sets correspond to data blocks, each of the data index sets includes one or more data indexes, and different data indexes of the same data index set are used to represent the data blocks corresponding to the data index set from different semantic perspectives;

[0008] Based on the plurality of target data indexes, query from different semantic angles to obtain one or more candidate data blocks matching the query statement;

[0009] Based on one or more of the candidate data blocks, a target data block is determined.

[0010] In the above scheme, by obtaining a query statement, multiple data indexes matching the query statement can be determined from multiple data index sets. Since the data index set corresponds to a data block, each data index set includes one or more data indexes. Different data indexes of the same data index set are used to represent the data blocks corresponding to the data index set from different semantic perspectives. Therefore, based on the different data indexes in the data index set, one or more candidate data blocks matching the query statement can be queried from different semantic perspectives, thereby determining a target data block that is more closely matched with the query statement based on the one or more candidate data blocks, thereby improving the data retrieval quality.

[0011] In a possible implementation manner of the first aspect, the data retrieval method further includes:

[0012] Split the data to be processed into multiple data blocks;

[0013] Construct the data index set for each of the plurality of data blocks;

[0014] storing the data index set in a first database;

[0015] The plurality of data blocks and the correspondence information between the data blocks and the data indexes are stored in a second database.

[0016] In the above scheme, by constructing a data index set for each data block and storing the data index set in the first database, and storing multiple data blocks and the correspondence information between the data blocks and the data indexes in the second database, it is possible to facilitate higher quality data retrieval based on the first database and the second database during subsequent data retrieval.

[0017] In a possible implementation manner of the first aspect, before respectively constructing the data index sets for the plurality of data blocks, the data retrieval method further includes:

[0018] Determine one or more index categories to be constructed corresponding to the data to be processed;

[0019] The step of respectively constructing the data index set for the plurality of data blocks comprises:

[0020] Based on one or more of the to-be-constructed index categories and each of the data blocks, the data index set is constructed for the corresponding data block.

[0021] In the above scheme, based on one or more index categories to be constructed corresponding to the data to be processed and the data block itself, a data index set containing one or more data indexes can be constructed for the data block. In subsequent data retrieval, the data index can be used to match the query statement with the data block from multiple semantic perspectives, thereby improving the quality of data retrieval.

[0022] In a possible implementation manner of the first aspect, determining one or more to-be-constructed index categories corresponding to the to-be-processed data includes:

[0023] Determine the matching scores between the data to be processed and N candidate index categories, where N is greater than or equal to 2;

[0024] If there are one or more target index categories among the N candidate index categories, the target index category is determined to be the index category to be constructed, and the target index category is a candidate index category whose matching score is greater than a first preset score.

[0025] In the above scheme, based on the matching scores between the data to be processed and N candidate index categories, one or more target index categories that are more closely matched with the data to be processed can be screened out from the N candidate index categories, and the target index category can be used as the index category to be constructed. Based on the index category to be constructed, a data index that can more accurately represent the semantics of the data block can be constructed for the data block.

[0026] In a possible implementation manner of the first aspect, determining, from a plurality of data index sets, a plurality of target data indexes matching the query statement includes:

[0027] Matching the query statement with each of the data indexes in the plurality of data index sets to obtain a matching score for each of the data indexes;

[0028] Based on the matching score of each of the data indexes, multiple target data indexes are screened out from the multiple data index sets, and the matching scores of the multiple target data indexes are greater than a second preset score or are arranged in the first M positions, where M is a preset integer greater than 1.

[0029] In the above solution, based on the matching score of each data index in the multiple data index sets, multiple target data indexes that are more closely matched with the query statement can be screened out from the multiple data index sets.

[0030] In a possible implementation manner of the first aspect, before determining the target data block based on the one or more candidate data blocks, the method further includes:

[0031] Determining matching scores of one or more candidate data blocks based on matching scores of the plurality of target data indexes;

[0032] The determining of the target data block based on one or more candidate data blocks includes:

[0033] The target data block is determined based on the matching scores of one or more candidate data blocks.

[0034] In the above scheme, since the target data index is determined from the data indexes representing the candidate data blocks from different semantic angles, that is to say, the target data index can more accurately represent the candidate data blocks, the matching scores of the candidate data blocks can be more accurately determined based on the matching scores of the target data index, and the target data blocks that are more closely matched with the query statement can be determined based on the matching scores of the candidate data blocks.

[0035] In a possible implementation manner of the first aspect, determining the matching scores of one or more candidate data blocks based on the matching scores of the plurality of target data indexes includes:

[0036] Determine one or more target data indexes belonging to the same candidate data block;

[0037] The matching score of the candidate data block is determined based on the matching scores of one or more target data indexes belonging to the same candidate data block.

[0038] In the above scheme, since one or more target data indexes belonging to the same candidate data block can more accurately represent the candidate data block, the matching score of the candidate data block can be more accurately determined based on the matching score of the target data indexes belonging to the same candidate data block.

[0039] In a possible implementation manner of the first aspect, when there are target data indexes corresponding to different index categories among the multiple target data indexes, determining the matching scores of one or more candidate data blocks based on the matching scores of the multiple target data indexes includes:

[0040] Based on the matching scores of the plurality of target data indexes and the index categories to which the plurality of target data indexes belong, generating matching score rankings corresponding to different index categories;

[0041] The matching score rankings corresponding to the different index categories are merged to obtain the matching scores of one or more candidate data blocks.

[0042] In the above scheme, since different data indexes of the same data block represent the data block from different semantic perspectives, the matching score rankings corresponding to different index categories can all represent the matching degree between the corresponding data block and the query statement. By integrating the matching score rankings corresponding to different index categories, the influence of the matching scores corresponding to different index categories on the matching degree between the data block and the query statement can be considered, and the advantages of different rankings can be fully utilized to improve the data retrieval quality.

[0043] In a possible implementation manner of the first aspect, after determining the target data block, the method further includes:

[0044] The query statement and the target data block are input into a large language model to obtain a query result.

[0045] In the above scheme, since the target data block is relatively matched with the query sentence, a relatively accurate query result can be obtained by inputting the query sentence and the target data block into the large language model.

[0046] In a second aspect, an embodiment of the present application provides a data retrieval device, comprising:

[0047] A query acquisition module is used to obtain query statements;

[0048] An index determination module, used to determine multiple target data indexes matching the query statement from multiple data index sets, wherein the data index sets correspond to data blocks, each of the data index sets includes one or more data indexes, and different data indexes of the same data index set are used to represent the data blocks corresponding to the data index set from different semantic perspectives;

[0049] A data determination module, configured to query from different semantic angles based on the plurality of target data indexes to obtain one or more candidate data blocks matching the query statement;

[0050] The target determination module is used to determine a target data block based on one or more candidate data blocks.

[0051] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the electronic device implements a data retrieval method as described in any one of the above-mentioned first aspects.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the data retrieval method as described in any one of the above-mentioned first aspects is implemented.

[0053] In a fifth aspect, an embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the data retrieval method described in any one of the first aspects above.

[0054] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0056] Figure 1 It is a schematic diagram of the architecture of an application scenario of the data retrieval method provided in an embodiment of the present application;

[0057] Figure 2 It is a flowchart of a data retrieval method provided in an embodiment of the present application;

[0058] Figure 3 is another flow chart of the data retrieval method provided in an embodiment of the present application;

[0059] Figure 4 This is a sample diagram of a document containing multiple levels of headings;

[0060] Figure 5 is an example diagram of constructing a data index provided in an embodiment of the present application;

[0061] Figure 6 is a flowchart of a data retrieval method according to another embodiment of the present application;

[0062] Figure 7 is a flowchart of a data retrieval method according to another embodiment of the present application;

[0063] Figure 8 is a structural schematic diagram of a data retrieval device provided in an embodiment of the present application;

[0064] Fig. 9 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0066] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0067] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0068] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.

[0069] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0070] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0071] It should be noted that the present application does not limit the application scenario of the data retrieval solution. As an example but not a limitation, the data retrieval solution in the present application can be applied to RAG.

[0072] For ease of understanding, this embodiment describes the data retrieval solution in this application using the RAG system as an example, but this does not constitute a limitation on the application scenarios of the data retrieval solution.

[0073] Before explaining the present application scheme, in order to facilitate the reader's understanding, the terms involved in the present scheme are first explained.

[0074] RAG system: combines large language models (LLM) and information retrieval technology. Specifically, when LLM needs to generate text or answer questions, it first retrieves relevant data blocks from a huge external knowledge base, and then uses these retrieved data blocks to guide text generation, thereby improving the quality and accuracy of predictions.

[0075] Data block: It can also be called knowledge base fragment, text block, etc. It is a smaller fragment (Chunk) divided by text.

[0076] The RGB system consists of three main processes: retrieval, enhancement, and generation.

[0077] Retrieval: Obtain relevant data blocks from the knowledge base based on the query statement entered by the user. Specifically, the query statement entered by the user is converted into a vector through the embedded model so as to match it with the relevant data blocks stored in the knowledge base, and find the data block that best matches the query statement as the input for subsequent generation.

[0078] Embedded models: In the fields of machine learning and natural language processing, they are used to convert discrete data items into fixed-size vectors that capture the intrinsic characteristics and properties of the data items, thereby participating in subsequent reasoning calculations.

[0079] Enhancement: The query statement entered by the user and the retrieved related data blocks are embedded in a preset prompt word template (Prompt) to form richer context information. The prompt word template is used to prompt LLM to generate the corresponding output (i.e., query results).

[0080] Generate: Input the search-enhanced cue word content into the LLM to generate the desired output.

[0081] The RAG system can play a role in various natural language processing tasks, such as question-answering systems, document generation, intelligent assistance and virtual agents, information retrieval, and knowledge graph filling. At the same time, the RAG system has the advantages of timely updates, strong interpretability, high customization, security and privacy management, and reduced training costs. Compared with fine-tuning, the RAG system has better versatility, is suitable for a variety of tasks, and can achieve instant knowledge updates without retraining the model, thereby greatly reducing the dependence on expensive computing resources.

[0082] The BM25 algorithm is an information retrieval algorithm based on word frequency and document length. It is used to calculate the relevance score (i.e., matching score) between query terms and documents. It takes into account the frequency of query terms in the document, the length of the document, and the distribution of terms in the entire document collection.

[0083] The Reciprocal Rank Fusion (RRF) algorithm is an algorithm that combines the rankings of multiple search result lists to generate a single unified ranking.

[0084] Since the RAG system is equipped with a knowledge base for LLM, the embedded model retrieves relevant data blocks from the knowledge base first, and then enhances the generation capability of LLM based on the relevant data blocks, the accuracy of data retrieval in the knowledge base will directly affect the generation capability of LLM. Especially in vertical and proprietary fields, due to the lack of training of embedded models on the knowledge of these fields (such as internal non-public documents, special nouns, special terms, special description rules and other data in the proprietary field), the quality of knowledge base retrieval is generally poor, which directly affects the generation of LLM.

[0085] The embodiments of the present application provide a data retrieval method, a data retrieval device, and an electronic device. The embodiments of the present application can query data blocks from different semantic angles based on different data indexes in a plurality of data index sets. Therefore, in the data retrieval process, based on a plurality of target data indexes matching the query statement, one or more candidate data blocks matching the query statement are queried from different semantic angles, and based on the one or more candidate data blocks, a target data block that is relatively matching the query statement is determined, thereby improving the data retrieval quality.

[0086] The data retrieval method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, laptop computers, servers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0087] See also Figure 1 , Figure 1 The schematic diagram of the architecture of the application scenario of the data retrieval method provided in the embodiment of the present application is shown. The architecture of the application scenario includes a terminal 101 and a server 102. A communication connection can be established between the terminal 101 and the server 102 to realize data transmission between the terminal 101 and the server 102.

[0088] The terminal 101 can obtain the query statement input by the user and send the query statement to the server 102. After receiving the query statement, the server 102 can determine the target data block that is more matched with the query statement through the data retrieval solution provided by the present application, and input the query statement and the target data block into the LLM, thereby obtaining a more accurate query result.

[0089] See also Figure 2 , Figure 2 The following is a flow chart of a data retrieval method provided in an embodiment of the present application. As an example but not a limitation, the method is applied to a server and includes the following steps:

[0090] Step 201, obtaining a query statement.

[0091] The query statement may refer to the query content input by the user into the RAG system. The present application does not limit the specific content of the query statement. For example, the query statement may be a question that the user needs to query or content that needs to be translated.

[0092] Step 202: Determine a plurality of target data indexes matching the query statement from a plurality of data index sets.

[0093] The data index set corresponds to the data block, each data index set includes one or more data indexes, and different data indexes of the same data index set are used to represent the data block corresponding to the data index set from different semantic perspectives.

[0094] Since different data indexes in the same data index set are used to represent the data blocks corresponding to the data index set from different semantic perspectives, the server matches the query statement with each data index in multiple data index sets, which can enable the query statement to perform multi-angle semantic matching with the data block, thereby improving the quality of data retrieval.

[0095] Step 203: Based on the multiple target data indexes, one or more candidate data blocks matching the query statement are queried from different semantic perspectives.

[0096] Among them, the candidate data block is the data block corresponding to the target data index.

[0097] The server may pre-store data blocks and correspondence information between data blocks and data indexes. Based on multiple target data indexes, one or more candidate data blocks matching the query statement may be retrieved from the correspondence information.

[0098] In an embodiment of the present application, by obtaining a query statement, multiple data indexes matching the query statement can be determined from multiple data index sets. Since the data index set corresponds to a data block, each data index set includes one or more data indexes. Different data indexes of the same data index set are used to represent the data block corresponding to the data index set from different semantic perspectives. Therefore, based on the multiple target data indexes matching the query statement in the data index set, the data block can be queried from different semantic perspectives, thereby determining the target data block that is more closely matched with the query statement, thereby improving the data retrieval quality.

[0099] Step 204: determine a target data block based on one or more candidate data blocks.

[0100] The target data block is a data block selected from one or more candidate data blocks and relatively matching the query statement.

[0101] In a possible implementation, after executing step 204, the query statement and the retrieved target data block may be input into a large language model to obtain a query result.

[0102] As an example but not limitation, the query statement may be a question that the user needs to query, and the query result output by the large language model is the answer to the question.

[0103] Please participate Figure 3 , the data retrieval method also includes steps 301 to 304.

[0104] Step 301, split the data to be processed into multiple data blocks.

[0105] The data to be processed may refer to various data collected and related to at least one search scenario, and these data may come from documents, web pages, databases, etc. For example, if the search scenario is a question-answering system, then the data to be processed may refer to various data collected and related to the question-answering system.

[0106] Before splitting the data to be processed, the collected data to be processed needs to be cleaned to remove noise, duplicates, and irrelevant information to ensure the quality and accuracy of the data to be processed.

[0107] It should be understood that the number of the above-mentioned data to be processed is at least one. By splitting more data to be processed, more data blocks can be obtained, thereby providing rich data support for data retrieval and improving the accuracy and richness of data retrieval.

[0108] It should be noted that the present application does not limit the method of splitting the data to be processed.

[0109] Step 302: construct data index sets for multiple data blocks respectively.

[0110] By building a data index set for each data block, the server can represent the corresponding data block from different semantic perspectives, thereby performing multi-angle semantic matching between the query statement and each data block, thereby improving the quality of data retrieval.

[0111] In one possible implementation, before constructing data index sets for multiple data blocks respectively, the server may first determine one or more index categories to be constructed corresponding to the data to be processed, and then construct a data index set for the corresponding data block based on the one or more index categories to be constructed and each data block.

[0112] Since a large number of data to be processed are required to build a knowledge base, and the contents and styles of different data to be processed are different, in one possible implementation method, each data to be processed can be analyzed and processed to determine one or more index categories to be constructed corresponding to each data to be processed, and a data index is constructed for the data block corresponding to the data to be processed based on the index category to be constructed, thereby constructing a data index for the data block that can more accurately represent its semantics, thereby realizing different semantic expressions of the same data block.

[0113] Optionally, the above analysis and processing of each data to be processed is as follows:

[0114] Determine the matching scores between the data to be processed and N candidate index categories, where N is greater than or equal to 2;

[0115] If there are one or more target index categories among the N candidate index categories, the target index category is determined to be the index category to be constructed, and the target index category is a candidate index category whose matching score is greater than a first preset score.

[0116] Among them, the above-mentioned N candidate index categories are pre-set index categories. As an example but not limitation, the above-mentioned N candidate index categories include but are not limited to six index categories such as summary, hypothetical question, parent-child block, title extraction, synonym conversion, and metadata. The data indexes corresponding to the above-mentioned six index categories can be respectively referred to as summary index, hypothetical question index, parent-child block index, title index, synonym conversion index, metadata index, etc. The above-mentioned six data indexes can be constructed by corresponding index builders. The index builders corresponding to the above-mentioned six data indexes can be respectively referred to as summary index builders, hypothetical question index builders, parent-child block index builders, title extraction index builders, synonym conversion index builders, and metadata index builders.

[0117] During the construction of the first database and the second database, the server may analyze and process the content of each data to be processed, thereby automatically identifying the index category to be constructed corresponding to each data to be processed from the N candidate index categories.

[0118] As an example but not limitation, if the content of the data to be processed includes the content of a leader's speech, then the corresponding index category to be constructed may include an abstract. The abstract is not only brief in content, but also accurately describes the main content of the leader's speech. If the content of the data to be processed includes a user manual, then the corresponding index category to be constructed may include hypothetical questions. After the data to be processed is split, the difference between different data blocks can be increased by hypothetical questions. If the content of the data to be processed includes long paragraphs of text, then the corresponding index category to be constructed may include parent-child blocks. After the data to be processed is split, there may be a lot of descriptions of other content in the data block (i.e., content that is not related to the retrieval), which makes it difficult to match the data block in subsequent semantic matching. In this case, the description of other content in the child data block can be reduced through the parent-child blocks, and the child data block can be matched in subsequent semantic matching, thereby matching the data block corresponding to the child data block. If the content of the data to be processed includes a user guide or a document containing multi-level titles, then the corresponding index category to be constructed may include title extraction. As Figure 4 The following is an example of a document containing multi-level headings. Figure 4 The front panel and the back panel both contain three parts of content, but all the content in the entire front panel or back panel is relatively long and needs to be split. After the interface is split from the front panel and the back panel, the interface content loses its association with the information such as the front panel and the back panel. When the user enters the front panel interface, the data block matching is chaotic and may match the interface of the back panel. In this case, title extraction can avoid data block matching confusion and improve data retrieval quality. If some words in the data to be processed are nouns in a proprietary field, the noun may have multiple names (such as potatoes, which can also be called ground eggs and potatoes), then the corresponding index category to be constructed can include synonym conversion, thereby solving the problem of difficulty in matching due to multiple names of some words. If the content of the data to be processed includes movie ranking documents, personnel list data documents, etc., the corresponding index category to be constructed can include metadata, and the metadata can be matched to the data block containing metadata during subsequent semantic matching.

[0119] In this embodiment, the server can determine the matching score between each to-be-processed data and N candidate index categories through an index selector. The index selector can include LLM and reference templates corresponding to the N candidate index categories. The reference templates are prompt word templates in the form of a few samples for different candidate index categories. The implementation method is as follows:

[0120] Each data to be processed is input into the index selector, and the index selector determines the matching score between the data to be processed and the N candidate index categories based on the reference templates corresponding to the N candidate index categories. The matching score represents the matching degree between the data to be processed and the corresponding candidate index category.

[0121] After the server obtains the matching scores between the data to be processed and N candidate index categories based on the index selector, it can screen out a target index category that is more closely matched or has a better matching degree with the data to be processed from the N candidate index categories by comparing the matching scores with the first preset scores, and determine the target index category as the index category to be constructed, thereby screening out an index category to be constructed that can more accurately represent the semantics of each data to be processed.

[0122] Optionally, the first preset score can be set according to actual scene requirements, and this application does not limit this. For example, the lowest score of the matching score is 0 points and the highest score is 100 points, so the first preset score can be set to 60 points.

[0123] In another possible implementation, one or more index categories to be constructed may be pre-set, and the index category to be constructed is determined to be the index category to be constructed corresponding to each data to be processed. This method is simple to operate and does not require analysis and processing of the content of the data to be processed.

[0124] After determining one or more index categories to be constructed corresponding to each data to be processed, the server can select the required index builder from N index builders based on the one or more index categories to be constructed, and construct a data index belonging to the corresponding index category to be constructed based on the selected index builder, and construct a data index set of the data block based on the constructed data index and data block (that is, the data block itself is also a kind of data index) belonging to the corresponding index category to be constructed. That is, on the basis of the existing data retrieval method that only matches the query statement with the data block itself, the embodiment of the present application also matches the query statement with the constructed data index, so that the query statement can be matched with each data block from different semantic angles, and the target data block that is more matched with the query statement is determined from the second database, thereby improving the data retrieval quality.

[0125] In one embodiment, the construction of six types of data indexes, namely, summary index, hypothetical question index, parent-child block index, title index, synonym conversion index, and metadata index, is introduced by taking the i-th data block as an example. The i-th data block is any data block of the data to be processed, and i is an integer greater than 0 and less than or equal to the total number of data blocks corresponding to the data to be processed.

[0126] The summary index of the i-th data block can be obtained by inputting the i-th data block into the summary index builder and summarizing the i-th data block through the summary index builder. The summary index is more refined than the i-th data block, which can not only well represent the semantics of the i-th data block, but also reduce the impact of other content (i.e., non-summary content) in the i-th data block on data retrieval, and can improve the ranking of the i-th data block in the matching score ranking corresponding to the summary, thereby increasing the probability of retrieving the i-th data block in subsequent data retrieval.

[0127] The hypothetical question index of the ith data block may be a hypothetical question constructed for the ith data block by the hypothetical question type index builder after the ith data block is input into the hypothetical question type index builder. In this embodiment, the hypothetical question is used as the data index of the ith data block, which can increase the difference between the ith data block and other data blocks, and can improve the ranking of the ith data block in the matching score ranking corresponding to the hypothetical question, thereby increasing the probability of retrieving the ith data block in subsequent data retrieval.

[0128] The parent-child block index of the ith data block may be inputting the ith data block into the parent-child block class index builder, splitting the ith data block through the parent-child block class index builder, obtaining two or more child data blocks of the ith data block, and establishing the corresponding relationship information between the child data blocks and the ith data block. In this embodiment, by adopting the parent-child block index, the child data block described in a short language is used as the index of the ith data block, which can improve the ranking of the ith data block in the matching score ranking corresponding to the parent-child block, thereby increasing the probability of retrieving the ith data block in subsequent data retrieval.

[0129] The title extraction index of the i-th data block can be obtained by inputting the i-th data block into the title extraction index builder, and extracting the title of the i-th data block through the title extraction index builder. This embodiment can avoid problems such as confusion in data block matching due to lack of title content by using the extracted title as the data index of the i-th data block.

[0130] The synonym conversion index of the i-th data block can be obtained by inputting the i-th data block into the synonym conversion index builder and performing synonym conversion on the i-th data block through the synonym conversion index builder. In this embodiment, the synonym conversion index is used as the data index of the i-th data block, which can perform semantic matching more accurately when the i-th data block has multiple expressions.

[0131] The metadata index of the i-th data block can be obtained by inputting the i-th data block into a metadata index builder and extracting metadata from the i-th data block through the metadata index builder. The metadata includes but is not limited to information such as time, person, country, and location. In this embodiment, the extracted metadata is used as the data index of the i-th data block, which can improve the ranking of the i-th data block in the matching score ranking corresponding to the metadata, thereby increasing the probability of retrieving the i-th data block in subsequent data retrieval.

[0132] Step 303: Store the data index set in the first database.

[0133] Storing the data index set in the first database can facilitate subsequent determination of the data index matching the query statement based on the first database.

[0134] It should be noted that the data index stored in the first database can be a vector or a non-vector, which is related to the matching method used in semantic matching. For example, if vector matching is used, the data index stored in the first database is a vector. For example, the data index can be input into the embedded model for processing, and after the vectorized data index is obtained, the vectorized data index is stored in the first database.

[0135] Step 304: Store the plurality of data blocks and the correspondence information between the data blocks and the data indexes in the second database.

[0136] Storing the data blocks and the corresponding relationship information between the data blocks and the data indexes in the second database can facilitate subsequent retrieval of the data blocks matching the query statement from the second database.

[0137] It should be understood that the above-mentioned first database and the above-mentioned second database can be understood as knowledge bases that provide data support when performing data retrieval through the RAG system.

[0138] like Figure 5 The following is an example diagram of building a data index. The server can Figure 5 The index selector in the data processing unit selects the index category to be constructed based on the content of the data to be processed, and constructs corresponding data indexes for the data blocks based on a variety of index builders such as summary index builders, hypothesis index builders, parent-child index builders, title extraction index builders, synonym conversion index builders, and metadata index builders. The constructed data indexes are vectorized through the embedded model and stored in the first database, and the data blocks and the corresponding relationship information between the data blocks and the data indexes are stored in the second database. It should be noted that in this embodiment, the data block itself is also used as a data index, and is vectorized through the embedded model and stored in the first database (in Figure 5 not shown).

[0139] like Figure 6 FIG. 1 is a flow chart of a data retrieval method according to another embodiment of the present invention. The method includes: Figure 6 Steps 601 to 611 in . Taking the question-answering system as an example, the data to be processed is a document. Step 601, split the document into S data blocks, where S is an integer greater than 1. Step 602, input the document into the index selector to determine three index categories to be constructed. Step 603, determine the index builder group to be used based on the three index categories to be constructed, and the index builder group includes three index builders required to build the data index belonging to the index category to be constructed. Step 604, build three corresponding data indexes for the S data blocks based on the index builder group. For example, the data indexes of data block 1 are data index 1.1, data index 1.2 and data index 1.3, respectively, the data indexes of data block 2 are data index 2.1, data index 2.2 and data index 2.3, respectively, and the data indexes of data block S are data index S.1, data index S.2 and data index S.3, respectively. Step 605, after constructing the data indexes of all data blocks, input the data indexes into the embedded model to obtain vectorized data indexes. Step 606, storing the vectorized data index in the first database. Step 607, storing the data block and the corresponding relationship information between the data block and the data index in the second database. After executing steps 606 and 607, the user can enter the question to be queried in the RAG system. Step 608, matching the question to be queried with the data index in the first database to obtain multiple target data indexes matching the query statement. Step 609, based on the multiple target data indexes matching the query statement, one or more candidate data blocks matching the query statement can be retrieved from the second database, and the target data block that is more closely matched with the query statement can be determined based on the one or more candidate data blocks. Step 610, embedding the user's query question and the retrieved target data block matching the query statement into a preset prompt word template to obtain the prompt word content enhanced by the retrieval. Step 611, inputting the prompt word content enhanced by the retrieval into the LLM to generate the required output.

[0140] In one possible implementation, the above step 202 may include: matching the query statement with each data index in a plurality of data index sets to obtain a matching score for each data index; based on the matching score of each data index, screening out a plurality of target data indexes from the plurality of data index sets, the matching scores of the plurality of target data indexes being greater than a second preset score or being arranged in the first M positions, where M is a preset integer greater than 1.

[0141] Optionally, the similarity between the query statement and the data index can be used to measure the matching degree between the query statement and the data index, that is, the matching score can be characterized by the similarity. For example, the matching score can be characterized by cosine similarity. Of course, it is understandable that the matching degree between the query statement and the data index can also be measured by other methods (such as Euclidean distance), and this application does not limit this.

[0142] The server can compare the matching score of each data index with the second preset score, or arrange each data index in the multiple data index sets in descending order of matching scores, and can filter out the target data index that matches the query statement from the multiple data index sets.

[0143] Optionally, the second preset score can be set according to actual scenario requirements, and this application does not limit this.

[0144] In a possible implementation, before executing step 204, the matching scores of one or more candidate data blocks may be determined based on the matching scores of multiple target data indexes, thereby determining the target data block based on the matching scores of one or more candidate data blocks.

[0145] Since the second database stores data blocks and the correspondence information between data blocks and data indexes, the matching score of the data block represents the degree of matching between the data block and the query statement. Therefore, the server can determine the matching score of the candidate data block based on the matching score of the target data index and the correspondence information stored in the second database. The candidate data block is the data block corresponding to the target data index; based on the matching score of the candidate data block, the target data block that is more closely matched with the query statement can be determined.

[0146] The server may determine the matching scores of one or more candidate data blocks in the following two ways.

[0147] Method 1: determine one or more target data indexes belonging to the same candidate data block; and determine the matching score of the candidate data block based on the matching scores of the one or more target data indexes belonging to the same candidate data block.

[0148] In this manner, one or more target data indexes belonging to the same candidate data block can be determined based on the correspondence information between the data block and the data index in the second database. The maximum value among the matching scores of the one or more target data indexes belonging to the same candidate data block can be used as the matching score of the candidate data block, or different weights can be set for the matching scores of different data indexes, and based on the weights, the matching scores of the one or more target data indexes belonging to the same candidate data block are weighted and calculated to obtain a comprehensive score, which is the matching score of the candidate data block.

[0149] like Figure 7 The figure shows an example flow chart of a data retrieval method of another embodiment provided by the present application. The method includes Figure 7 Steps 701 to 704 in the RAG system. Taking the question-answering system as an example, the user can input questions (i.e., query statements) in the RAG system. Step 701, convert the input question into a vector through the embedded model, calculate the matching score between the vector corresponding to the question and each data index in the first database, and based on the matching score, filter out multiple target data indexes that match the query statement from the first database. Step 702, take the maximum value of the matching score of at least one target data index belonging to the same candidate data block as the matching score of the candidate data block, filter out the data blocks with the matching score in the top K positions from the candidate data blocks, and obtain the matching score ranking of these K data blocks, where K is an integer greater than zero. Step 703, calculate the matching score between the input question and each data block based on the BM25 algorithm. Since the number of data blocks is large, K data blocks that are more closely matched with the question can be filtered out from more data blocks based on the matching score, and the distribution score ranking of these K data blocks can be obtained. In step 704, the two matching score rankings obtained based on the two retrieval models, such as the embedded model and the BM25 algorithm, are ranked and fused through the RRF algorithm to obtain the final matching score ranking between the input question and some data blocks in the second database. Based on the matching score ranking, the data block most relevant to the question can be retrieved from the second database.

[0150] Method 2: When there are target data indexes corresponding to different index categories among the multiple target data indexes, matching score rankings corresponding to different index categories are generated based on the matching scores of the multiple target data indexes and the index categories to which the multiple target data indexes belong;

[0151] The matching score rankings corresponding to different index categories are merged to obtain the matching scores of one or more candidate data blocks.

[0152] The server can use the RRF algorithm to merge the matching score rankings corresponding to different index categories to obtain the matching scores of candidate data blocks. This can fully utilize the matching advantages of different index categories, improve the overall quality and relevance of semantic matching, and thus enhance the performance of LLM in generating accurate responses.

[0153] The embodiment of the present application can determine the data index that matches the query statement based on the first database by obtaining the query statement. Since the first database includes multiple data index sets corresponding to different data blocks, the same data index set contains one or more data indexes corresponding to the same data block, and different data indexes in the same data index set are used to represent the corresponding data blocks from different semantic perspectives, based on the different data indexes in the first database, the data blocks can be queried from different semantic perspectives, thereby determining the data blocks that are more closely matched with the query statement from the second database, thereby improving the data retrieval quality.

[0154] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0155] Corresponding to the data retrieval method described in the above embodiment, Figure 8 A schematic diagram of the structure of a data retrieval device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0156] Reference Figure 8 , the data retrieval device comprises:

[0157] A query acquisition module 81, used to acquire a query statement;

[0158] An index determination module 82 is used to determine a plurality of target data indexes matching the query statement from a plurality of data index sets, wherein the data index sets correspond to data blocks, each of the data index sets includes one or more data indexes, and different data indexes of the same data index set are used to represent the data blocks corresponding to the data index set from different semantic perspectives;

[0159] A data determination module 83 is used to query and obtain one or more candidate data blocks matching the query statement from different semantic perspectives based on multiple target data indexes;

[0160] The target determination module 84 determines a target data block based on one or more candidate data blocks.

[0161] Optionally, the data retrieval device further includes:

[0162] A data splitting module is used to split the data to be processed into multiple data blocks;

[0163] An index building module, used to build the data index set for the plurality of data blocks respectively;

[0164] An index storage module, used to store the data index set in a first database;

[0165] The information storage module is used to store the plurality of data blocks and the corresponding relationship information between the data blocks and the data indexes in the second database.

[0166] Optionally, the data retrieval device further includes:

[0167] A category determination module, used to determine one or more categories of indexes to be constructed corresponding to the data to be processed;

[0168] The above index building module is specifically used for:

[0169] Based on one or more of the to-be-constructed index categories and each of the data blocks, the data index set is constructed for the corresponding data block.

[0170] Optionally, the category determination module is specifically used for:

[0171] Determine the matching scores between the data to be processed and N candidate index categories, where N is greater than or equal to 2;

[0172] If there are one or more target index categories among the N candidate index categories, the target index category is determined to be the index category to be constructed, and the target index category is a candidate index category whose matching score is greater than a first preset score.

[0173] Optionally, the index determination module 82 includes:

[0174] A data matching unit, used for matching the query statement with each of the data indexes in the plurality of data index sets to obtain a matching score for each of the data indexes;

[0175] An index screening module is used to screen out multiple target data indexes from multiple data index sets based on the matching score of each data index, and the matching scores of the multiple target data indexes are greater than a second preset score or are arranged in the first M positions, where M is a preset integer greater than 1.

[0176] Optionally, the data retrieval device further includes:

[0177] A score determination module, configured to determine a matching score of one or more candidate data blocks based on the matching scores of the plurality of target data indexes;

[0178] The target determination module 84 is specifically used for:

[0179] The target data block is determined based on the matching scores of one or more candidate data blocks.

[0180] Optionally, the score determination module is specifically used for:

[0181] Determine one or more target data indexes belonging to the same candidate data block;

[0182] The matching score of the candidate data block is determined based on the matching scores of one or more target data indexes belonging to the same candidate data block.

[0183] Optionally, when there are target data indexes corresponding to different index categories among the plurality of target data indexes, the score determination module is specifically used to:

[0184] Based on the matching scores of the plurality of target data indexes and the index categories to which the plurality of target data indexes belong, generating matching score rankings corresponding to different index categories;

[0185] The matching score rankings corresponding to the different index categories are merged to obtain the matching scores of one or more candidate data blocks.

[0186] Optionally, the data retrieval device further includes:

[0187] The data input module is used to input the query statement and the target data block into the large language model to obtain the query result.

[0188] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0189] Fig. 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device 9 of this embodiment includes: at least one processor 90 ( Fig. 9 Only one is shown), a memory 91, and a computer program 92 stored in the memory 91 and executable on the at least one processor 90, wherein the processor 90 implements the steps of any of the above-mentioned data retrieval method embodiments when executing the computer program 92.

[0190] The electronic device may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will appreciate that Fig. 9 It is only an example of the electronic device 9 and does not constitute a limitation on the electronic device 9. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, it may also include input and output devices, network access devices, etc.

[0191] The processor 90 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0192] In some embodiments, the memory 91 may be an internal storage unit of the electronic device 9, such as a hard disk or memory of the electronic device 9. In other embodiments, the memory 91 may also be an external storage device of the electronic device 9, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 9. Further, the memory 91 may also include both an internal storage unit of the electronic device 9 and an external storage device. The memory 91 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program, etc. The memory 91 may also be used to temporarily store data that has been output or is to be output.

[0193] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0194] An embodiment of the present application also provides an electronic device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps of any of the above-mentioned method embodiments when executing the computer program.

[0195] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0196] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0197] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0198] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0199] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0200] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0201] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0202] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A data retrieval method, characterized in that: include: Get the query statement; Determine, from a plurality of data index sets, a plurality of target data indexes matching the query statement, wherein the data index sets correspond to data blocks, each of the data index sets includes one or more data indexes, and different data indexes of the same data index set are used to represent the data blocks corresponding to the data index set from different semantic perspectives; Based on the plurality of target data indexes, query from different semantic angles to obtain one or more candidate data blocks matching the query statement; Based on one or more of the candidate data blocks, a target data block is determined.

2. The data retrieval method according to claim 1, characterized in that: Also includes: Split the data to be processed into multiple data blocks; Construct the data index set for each of the plurality of data blocks; storing the data index set in a first database; The plurality of data blocks and the correspondence information between the data blocks and the data indexes are stored in a second database.

3. The data retrieval method according to claim 2, characterized in that: Before respectively constructing the data index set for the plurality of data blocks, the method further includes: Determine one or more index categories to be constructed corresponding to the data to be processed; The step of respectively constructing the data index set for the plurality of data blocks comprises: Based on one or more of the to-be-constructed index categories and each of the data blocks, the data index set is constructed for the corresponding data block.

4. The data retrieval method according to claim 3, characterized in that: The determining one or more index categories to be constructed corresponding to the data to be processed includes: Determine the matching scores between the data to be processed and N candidate index categories, where N is greater than or equal to 2; If there are one or more target index categories among the N candidate index categories, the target index category is determined to be the index category to be constructed, and the target index category is a candidate index category whose matching score is greater than a first preset score.

5. The data retrieval method according to any one of claims 1 to 4, characterized in that: The step of determining, from a plurality of data index sets, a plurality of target data indexes matching the query statement comprises: Matching the query statement with each of the data indexes in the plurality of data index sets to obtain a matching score for each of the data indexes; Based on the matching score of each of the data indexes, multiple target data indexes are screened out from the multiple data index sets, and the matching scores of the multiple target data indexes are greater than a second preset score or are arranged in the first M positions, where M is a preset integer greater than 1.

6. The data retrieval method according to claim 5, characterized in that: Before determining the target data block based on one or more candidate data blocks, the method further includes: Determining matching scores of one or more candidate data blocks based on matching scores of the plurality of target data indexes; The determining of the target data block based on one or more candidate data blocks includes: The target data block is determined based on the matching scores of one or more candidate data blocks.

7. The data retrieval method according to claim 6, characterized in that: The determining, based on the matching scores of the plurality of target data indexes, the matching scores of one or more candidate data blocks comprises: Determine one or more target data indexes belonging to the same candidate data block; The matching score of the candidate data block is determined based on the matching scores of one or more target data indexes belonging to the same candidate data block.

8. The data retrieval method according to claim 6, characterized in that: In the case that there are target data indexes corresponding to different index categories among the plurality of target data indexes, determining the matching scores of one or more candidate data blocks based on the matching scores of the plurality of target data indexes includes: Based on the matching scores of the plurality of target data indexes and the index categories to which the plurality of target data indexes belong, generating matching score rankings corresponding to different index categories; The matching score rankings corresponding to the different index categories are merged to obtain the matching scores of one or more candidate data blocks.

9. The data retrieval method according to any one of claims 1 to 4, characterized in that: After determining the target data block, it also includes: The query statement and the target data block are input into a large language model to obtain a query result.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the electronic device implements the data retrieval method according to any one of claims 1 to 9.

Images