Data retrieval method, device, electronic device and storage medium

Abstract

The data retrieval method, device, electronic device and storage medium provided by the present invention relate to the technical field of electronic digital data processing. By obtaining the matching fields of the query statement through feature information, accurate matching according to the query intent, key information and context information is achieved, and the accuracy of the matching field group is improved. The matching field group is obtained according to the field-topic mapping table, and the query statement is quickly located, which is beneficial to improving the efficiency and accuracy of data retrieval. Multi-dimensional retrieval is performed according to the extended matching field group, and retrieval according to the entities of the matching field group is achieved, which is beneficial to improving the accuracy and efficiency of data retrieval. The retrieval results are weighted and fused according to the weight of the extended matching field group, so that the weight of each retrieval result can be accurately obtained, which is convenient for users to quickly find the retrieval data most relevant to the query statement.

Description

Technical Field

[0001] The present invention relates to the technical field of electronic digital data processing, and in particular to a data retrieval method, device, electronic equipment and storage medium. Background Art

[0002] With the deepening development of the digital age, the amount of data is exploding and the data structure is becoming increasingly complex. Traditional data retrieval methods are unable to cope with large-scale, multi-field databases. Although the Retrieval-Augmented Generation (RAG) system has made some progress by combining retrieval and generation technologies, it still has obvious limitations when dealing with complex queries and multi-dimensional data.

[0003] Currently, mainstream RAG systems usually only vectorize the main text of the database, and simply use other attributes as metadata for full-word matching. This approach leads to a series of problems: retrieval flexibility is limited, and users often need to explicitly specify matching content. In addition, existing systems are also deficient in query understanding capabilities, making it difficult to accurately capture the user's true intentions, especially when dealing with complex and multifaceted queries. Finally, most data retrieval systems lack effective result evaluation and optimization mechanisms.

[0004] The above problems lead to low efficiency of existing data retrieval. Summary of the invention

[0005] The present invention provides a data retrieval method, device, electronic device and storage medium, which are used to solve the defect of low data retrieval efficiency in the prior art and improve the efficiency of data retrieval.

[0006] The present invention provides a data retrieval method, comprising the following steps: obtaining feature information of a query statement based on a semantic analysis result of a user's query statement based on a language model, the feature information comprising query intent, key information and context information; obtaining multiple matching field groups of the query statement in a field-topic mapping table of a database based on the feature information; the field-topic mapping table comprising a mapping relationship between preset topics of multiple databases and fields of the database; the matching field groups correspond to the preset topics one by one; extending the matching fields in each matching field group to obtain multiple extended matching field groups of the query statement; performing multi-dimensional retrieval in the database based on the multiple extended matching field groups to obtain retrieval results of each extended matching field group; and weighted fusion of the retrieval results based on the weights of the extended matching field groups to obtain retrieval data of the query statement.

[0007] According to the data retrieval method provided by the present invention, multiple matching field groups of the query statement are obtained in the field-subject mapping table of the database based on the feature information, including: obtaining multiple matching field sequences of the query statement in the field-subject mapping table based on the feature information; obtaining the correlation between the matching field and the query statement in each matching field sequence, sorting the matching fields in order of correlation from large to small, and obtaining a sorted matching field sequence; in each sorted matching field sequence, at least one matching field that is ranked first is used as a matching field group.

[0008] According to the data retrieval method provided by the present invention, after the retrieval results are weighted and fused based on the weights of the extended matching field groups to obtain the retrieval data of the query statement, the method further includes: when the retrieval score of the retrieval data is lower than the set score, the matching field group is updated based on each sorted matching field sequence; iterative retrieval is performed based on the updated matching field group until the retrieval score is greater than or equal to the set score, or the number of iterative retrievals reaches the set number; and the retrieval data with the highest retrieval score is used as the final retrieval data.

[0009] According to the data retrieval method provided by the present invention, the matching fields in each matching field group are expanded to obtain multiple extended matching field groups of the query statement, including: performing named entity recognition analysis on each matching field of the matching field group to obtain the associated fields of the matching field; and obtaining each extended matching field group based on all the matching fields in each matching field group and the associated fields of the matching fields.

[0010] According to the data retrieval method provided by the present invention, the weight of the extended matching field group is determined based on the following steps: obtaining the basic weight of each extended matching field group based on a preset theme; determining the importance level of each extended matching field group based on feature information; and adjusting the basic weight based on the importance level to obtain the weight of the extended matching field group.

[0011] According to the data retrieval method provided by the present invention, the database is determined based on the following steps: data cleaning is performed on the initial text involved in the retrieval in the initial database to unify the format of the initial text; the cleaned initial text is converted into a text vector; if the number of repetitions of the text vector in the initial database is lower than the set number of repetitions, an index structure of the text vector is constructed; and a database is obtained based on the index structure and the text vector.

[0012] According to the data retrieval method provided by the present invention, a multi-dimensional search is performed in a database based on multiple extended matching field groups to obtain a search result for each extended matching field group, including: performing a one-dimensional search in the database based on one extended matching field group; performing a multi-dimensional search in the database based on multiple extended matching field groups; wherein, if there is an index structure matching the extended matching field in the database, the search result is determined based on the similarity between the text vector of the index structure and the extended matching field; if there is an unstructured text vector matching the extended matching field in the database, fuzzy matching is performed on the unstructured text vector to obtain a search result.

[0013] The present invention also provides a data retrieval device, including: a feature information determination module, which is used to obtain feature information of a query statement based on a semantic analysis result of a user's query statement by a language model, wherein the feature information includes query intent, key information and context information; a matching module, which is used to obtain multiple matching field groups of the query statement in a field-topic mapping table of a database based on the feature information; the field-topic mapping table includes a mapping relationship between preset topics of multiple databases and fields of the database; the matching field groups correspond to the preset topics one by one; an extension module, which is used to extend the matching fields in each matching field group to obtain multiple extended matching field groups of the query statement; a retrieval module, which is used to perform multi-dimensional retrieval in the database based on the multiple extended matching field groups to obtain retrieval results of each extended matching field group; and a fusion module, which is used to perform weighted fusion on the retrieval results based on the weights of the extended matching field groups to obtain retrieval data of the query statement.

[0014] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-mentioned data retrieval methods when executing the computer program.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements any of the above-mentioned data retrieval methods.

[0016] The data retrieval method, device, electronic device and storage medium provided by the present invention obtain the matching fields of the query statement through feature information, realize accurate matching according to the query intent, key information and context information, and improve the accuracy of the matching field group. The matching field group is obtained according to the field-topic mapping table, and the query statement is quickly located, which is beneficial to improving the efficiency and accuracy of data retrieval. Multi-dimensional retrieval is performed according to the extended matching field group, and retrieval according to the entities of the matching field group is realized, which is beneficial to improving the accuracy and efficiency of data retrieval. The retrieval results are weighted and fused according to the weight of the extended matching field group, so that the weight of each retrieval result can be accurately obtained, which is convenient for users to quickly find the retrieval data most relevant to the query statement. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the data retrieval method provided by the present invention.

[0019] Figure 2 It is a structural schematic diagram of the data retrieval device provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Combine the following Figure 1-Figure 3 The data retrieval method, device, electronic device and storage medium of the present invention are described.

[0023] Figure 1 It is a flow chart of the data retrieval method provided by the present invention, such as Figure 1 As shown, the data retrieval method includes steps S100 to S500, and each step is specifically described as follows.

[0024] S100: Acquire feature information of a query statement based on a semantic analysis result of a user's query statement based on a language model. The feature information includes query intent, key information, and context information.

[0025] A language model is a model that can understand and express language. It is used to process natural language. Language models use machine learning and deep learning technologies to process the semantic information of text. Language models include the Chat Generation Pre-training Model (ChatGPT), Tongyi Qianwen Language Model, Doubao Language Model, etc.

[0026] Perform basic text cleaning and standardization on the user's query statements. Use language models to analyze the query intent, key information, and context information of the query statement. Among them, the query intent includes factual query, relationship query, and exploratory query. The key information includes the keywords, entities, and relationships in the query statement. The context information includes the potential semantics and context information of the query statement.

[0027] S200: Acquire multiple matching field groups of the query statement in a field-subject mapping table of a database based on the feature information.

[0028] The field-topic mapping table includes mapping relationships between preset topics of multiple databases and fields of the databases; matching field groups correspond one-to-one to preset topics.

[0029] Pre-build the field-subject mapping table of the database. The field-subject mapping table is used to describe the semantics and applicable scenarios of each field in the database. The field-subject mapping table includes multiple preset topics, such as title, text, author, date, journal quality, etc. Among them, the title includes multiple title fields for general query and keyword matching. The text includes multiple text fields for detailed content query and full-text search. The author includes multiple author fields for specific person-related queries. The date includes multiple date fields for time-related queries.

[0030] Using the language model as a route, the user's query statement is matched with the field-topic mapping table, and multiple matching preset topics are selected. The fields that are most relevant to the query statement and the matching preset topics are selected as the matching field group.

[0031] For example, the query statement is "I want to find review articles on the application of artificial intelligence in agricultural modernization published in top journals in the past five years. Pay special attention to research using computer vision and deep learning technology, preferably from authors from unit A or unit B. It would be better if the article discusses the application difficulties of technology for small farmers or environmental sustainability issues." It can be seen that the query intent is to find matching literature. The key information is artificial intelligence, agricultural modernization, computer vision and deep learning technology. The context information is "want to find", "pay special attention to", "preferably" and "it would be better if..."

[0032] According to the feature information (including query intent, key information and contextual information), multiple matching field groups of the query statement are obtained in the field-topic mapping table, including time matching field group (in the past five years), type matching field group (review articles), title matching field group (artificial intelligence, agricultural modernization), text matching field group (computer vision and deep learning), author matching field group (authors from unit A or unit B), journal quality matching field group (top journals) and other matching field groups (application difficulties for small farmers, environmental sustainability).

[0033] S300: Expand the matching fields in each matching field group to obtain multiple extended matching field groups of the query statement.

[0034] The matching fields in each matching field group are expanded to obtain multiple extended matching field groups of the query statement. Specifically, named entity recognition analysis is performed on each matching field in the matching field group to obtain the associated fields of the matching field; each extended matching field group is obtained based on all matching fields in each matching field group and the associated fields of the matching fields.

[0035] Named Entity Recognition (NER) analysis is to analyze entities of matching fields, such as names of people, organization names, place names, dates, quantities, etc. The NER analysis of the present invention is used to accurately extract entities related to matching fields, thereby improving the accuracy of retrieval.

[0036] For example, if the matching field is the past five years, the extended matching field obtained after NER analysis is 2019 to 2024. By analogy, the extended matching fields corresponding to review articles are review, comment, review, and meta-analysis. The extended matching fields corresponding to artificial intelligence are artificial intelligence, intelligent technology, machine learning, and smart agriculture. The extended matching fields corresponding to agricultural modernization are agricultural modernization, Agriculture 4.0, smart agriculture, and precision agriculture. The extended matching fields corresponding to computer vision are computer vision, image recognition, target detection, and remote sensing image analysis. The extended matching fields corresponding to deep learning are neural networks, convolutional neural networks, and recurrent neural networks. The extended matching fields corresponding to the authors of unit A or unit B are authors of unit A, unit B, subordinate units of unit A, and subordinate units of unit B. The extended matching fields corresponding to top journals are journals C, D, and E. The extended matching fields corresponding to the application problems of small farmers are small farmers, technology popularization, and cost issues. The extended matching fields corresponding to environmental sustainability are environmental science sustainability, ecological agriculture, and green agriculture.

[0037] All the matching fields in the matching field group and the associated fields of the matching fields are used as extended matching fields to obtain an extended matching field group.

[0038] The present invention obtains the associated fields of the matching fields according to the named entity recognition analysis, realizes the entity mining of the matching fields, improves the accuracy and relevance of the associated fields, and is conducive to improving the precision of retrieval.

[0039] S400: Perform a multi-dimensional search in a database based on multiple extended matching field groups to obtain a search result for each extended matching field group.

[0040] Each dimension search is performed based on each extended match field group. For example, there are multiple extended match field groups, namely, time extended match field group, type extended match field group, title extended match field group, text extended match field group, author extended match field group, journal quality extended match field group and other extended match field groups. One extended match field group corresponds to one dimension search, and one search result is obtained.

[0041] For example, if the time extension matching field group is 2019 to 2024, all documents from 2019 to 2024 will be matched, and a document list set (search result) will be obtained.

[0042] S500: Perform weighted fusion on the search results based on the weights of the extended matching field groups to obtain search data of the query statement.

[0043] Get the weight of each extended match field group. For example, the weight of the title extended match field group is 1, the weight of the text extended match field group is 0.8, the weight of the type extended match field group is 0.7, the weight of the time extended match field group is 0.6, the weight of the author extended match field group is 0.4, the weight of the journal quality extended match field group is 0.5, and the weight of other extended match field groups is 0.3.

[0044] The weight of the search results after weighted fusion is determined according to the weight of each extended matching field group, and the search results are sorted according to the weighted fusion weight, for example, the search results are sorted in descending order according to the weighted sum weight to obtain the search data.

[0045] For example, if document A matches the time extended match field group, type extended match field group, title extended match field group, text extended match field group, author extended match field group, journal quality extended match field group and other extended match field groups at the same time, the weighted fusion weight of document A is 1+0.8+0.7+0.6+0.5+0.4+0.3=4.3 (the maximum weight), and it ranks first.

[0046] The data retrieval method provided by the embodiment of the present invention obtains the matching fields of the query statement through feature information, realizes accurate matching according to the query intent, key information and context information, and improves the accuracy of the matching field group. The matching field group is obtained according to the field-subject mapping table, and the query statement is quickly located, which is beneficial to improving the efficiency and accuracy of data retrieval. Multi-dimensional retrieval is performed according to the extended matching field group, and retrieval according to the entities of the matching field group is realized, which is beneficial to improving the accuracy and efficiency of data retrieval. The retrieval results are weighted and fused according to the weight of the extended matching field group, so that the weight of each retrieval result can be accurately obtained, which is convenient for users to quickly find the retrieval data most relevant to the query statement.

[0047] Based on the above embodiment, multiple matching field groups of the query statement are obtained in the field-subject mapping table of the database based on the feature information, including steps S210 to S230, and each step is specifically as follows.

[0048] S210: Acquire multiple matching field sequences of the query statement in the field-topic mapping table based on the feature information.

[0049] S220: Obtain the correlation between the matching field and the query statement in each matching field sequence, sort the matching fields in descending order of correlation, and obtain a sorted matching field sequence.

[0050] S230: In each sorted matching field sequence, at least one matching field that is at the front is used as a matching field group.

[0051] According to the query intent, key information and context information of the query statement, multiple matching field sequences matching the query statement are obtained in the field-topic mapping table. For example, the query statement is "highly cited papers on artificial intelligence in medical applications published in the past five years". The matching field sequence includes the title and keyword matching field sequence (including artificial intelligence and medical fields, artificial intelligence, medical fields), the citation count matching field sequence (including the citation count greater than 100 times, the citation count between 100 times and 10 times, and the citation count less than 10 times) and the time field matching sequence (including the past 1 year, the past 3 years, the past 5 years, and the past 10 years).

[0052] In the matching field sequence, the matching fields are sorted in descending order of relevance to obtain a sorted matching field sequence. For example, the sorted reference count matching field sequence is reference count greater than 100 times, reference count between 100 times and 10 times, and reference count less than 10 times.

[0053] In each sorted matching field sequence, at least one matching field that is ranked first is used as a matching field group. For example, the first matching field that is ranked first is used as a matching field group. The matching field group is a matching field group that is referenced more than 100 times.

[0054] The present invention obtains multiple matching field sequences of the query statement, preliminarily locks the scope of the query statement, determines the matching field group according to the correlation, improves the correlation between the matching field group and the query statement, and is conducive to improving the efficiency and accuracy of the retrieval.

[0055] Based on the above embodiment, after weighted fusion of the search results is performed based on the weights of the extended matching field groups to obtain the search data of the query statement, steps S600 to S800 are further included, and the details of each step are as follows.

[0056] S600: When the search score of the searched data is lower than the set score, the matching field group is updated based on each sorted matching field sequence.

[0057] S700: Perform an iterative search based on the updated matching field group until the search score is greater than or equal to the set score, or the number of iterative searches reaches the set number.

[0058] S800: The search data with the highest search score is used as the final search data.

[0059] The semantic relevance of the search data is scored according to the language model to obtain the search score of the query statement. The search score is compared with the set score to determine whether optimization is needed. If the search score is lower than the set score, it means that the current search data is not ideal and needs to be searched again.

[0060] Update the matching field group based on each sorted matching field sequence. For example, the sorted matching field sequence is cited more than 100 times, cited between 100 and 10 times, and cited less than 10 times. In the current search, the matching field group is cited more than 100 times. Then, in the next search, the updated matching field group is cited between 100 and 10 times.

[0061] Search again based on all updated matching field groups until the search score is greater than or equal to the set score, or the number of iterative searches reaches the set number, then stop searching. The search data with the highest search score is used as the final search data.

[0062] The present invention improves the final retrieval accuracy by updating the matching field group and performing iterative retrieval.

[0063] Based on the above embodiment, the weight of the extended matching field group is determined based on steps S510 to S530, and each step is specifically as follows.

[0064] S510: Obtain a basic weight of each extended matching field group based on a preset theme.

[0065] S520: Determine the importance level of each extended matching field group based on the feature information.

[0066] S530: Adjust the basic weight based on the importance level to obtain the weight of the extended matching field group.

[0067] The basic weight of each extended match field group is preset. For example, the basic weight of the title extended match field group is 1, the basic weight of the text extended match field group is 0.8, the basic weight of the type extended match field group is 0.7, the basic weight of the time extended match field group is 0.6, and the basic weight of the author extended match field group is 0.4.

[0068] The importance level of each extended match field group is determined based on the feature information (including query intent, key information, and context information). For example, if the query statement is determined to be time-sensitive based on the feature information, the importance level of the time extended match field group is determined to be the highest importance level, and the basic weight of the time extended match field group is adjusted to 1 based on the highest importance level, so the final weight of the time extended match field group is 1.

[0069] The present invention adjusts the basic weight according to the feature information, realizes the adjustment of the basic weight according to the actual needs of the query statement, ensures that the retrieval results can meet the actual needs of the user to the greatest extent, and is conducive to improving the accuracy of data retrieval.

[0070] Based on the above embodiment, the database is determined based on steps S410 to S440, and each step is specifically as follows.

[0071] S410: Perform data cleaning on the initial texts involved in the retrieval in the initial database to unify the format of the initial texts.

[0072] S420: Convert the cleaned initial text into a text vector.

[0073] S430: If the number of repetitions of the text vector in the initial database is lower than the set number of repetitions, construct an index structure of the text vector.

[0074] S440: Obtain a database based on the index structure and the text vector.

[0075] Identify all the initial texts that may be involved in the retrieval in the initial database, including but not limited to the title, text, abstract, keywords, author, date, etc. Perform data cleaning on the initial text to remove special characters and unify the format of the initial text. Use a pre-trained language model, such as the BGE-M3 language model, to convert the cleaned initial text into a text vector. For relatively long initial texts, segment the initial text to obtain segmented texts, and convert each segmented text into a text vector.

[0076] If the number of repetitions of the text vector in the initial database is lower than the set number of repetitions, it means that the text vector is a text vector that is not repeated in large quantities, such as the text and the title. Construct an index structure for text vectors that are not repeated in large quantities. For example, use the Hierarchical Navigable Small World (HNSW) algorithm to construct an index structure. The HNSW algorithm can complete the search within logarithmic time complexity, improving the retrieval speed of large-scale vector data. The HNSW algorithm supports dynamic insertion of new vectors and is suitable for processing growing data sets. The index structure obtained according to the HNSW algorithm is relatively compact and can effectively utilize memory resources.

[0077] If the number of repetitions of a text vector in the initial database is greater than the set number of repetitions, then this type of text vector is a text vector with a large number of repetitions, such as author, tag, etc. For such a large number of repetitions, no index structure is constructed, and a direct match is performed during the query.

[0078] The present invention constructs an index structure for text vectors whose repetition times are lower than a set repetition times, which is beneficial to improving the retrieval speed of the database and also improves the utilization rate of the memory resources of the database.

[0079] The present invention vectorizes all the initial texts involved in matching or retrieval in the database, greatly expanding the database's semantic understanding ability and laying the foundation for subsequent accurate retrieval. At the same time, in order to support efficient retrieval, the database also establishes an efficient index structure (such as HNSW) for non-massively repeated fields after vectorization to support fast retrieval. Fields with a large number of repeated categories do not need to be indexed, thereby improving the effect and allowing categories to be quickly added or subtracted without frequently repeating the construction of index structures.

[0080] Based on the above embodiment, a multi-dimensional search is performed in a database based on multiple extended matching field groups to obtain a search result for each extended matching field group, including steps S540 to S550, and each step is specifically as follows.

[0081] S540: Perform a dimension search in the database based on an extended matching field group.

[0082] S550: Perform multi-dimensional search in the database based on multiple extended matching field groups; if there is an index structure matching the extended matching field in the database, determine the search result based on the similarity between the text vector of the index structure and the extended matching field; if there is an unstructured text vector matching the extended matching field in the database, perform fuzzy matching on the unstructured text vector to obtain the search result.

[0083] Perform a one-dimensional search in the database based on an extended match field group. Perform a multi-dimensional search in the database based on multiple extended match field groups. If an index structure is retrieved based on the extended match field in the extended match field group, the nearest neighbor algorithm (for example, HNSW) is used to calculate the similarity between the extended match field and the text vector in the index structure, and the text vector with the highest similarity is used as the search result of the extended match field. If an unstructured text vector is retrieved based on the extended match field, fuzzy matching is performed on the unstructured text vector. Fuzzy matching allows a certain degree of error in string comparison. Fuzzy matching also uses text vectors with spelling errors, homophones, synonyms, etc. as matching text vectors, thereby improving the flexibility and recall rate of the search. If the extended match field matches the category field, exact or fuzzy matching is performed directly.

[0084] The present invention performs multi-dimensional retrieval based on multiple extended matching field groups, thereby improving the retrieval speed and comprehensiveness of the retrieval. Similarity matching and fuzzy matching are introduced into the retrieval, thereby improving the accuracy of the retrieval.

[0085] The data retrieval device provided by the present invention is described below. The data retrieval device described below and the data retrieval method described above can be referenced to each other.

[0086] like Figure 2 As shown, a data retrieval device includes: a feature information determination module 201, which is used to obtain feature information of a query statement based on a semantic analysis result of a user's query statement by a language model, and the feature information includes query intent, key information and context information.

[0087] Matching module 202 is used to obtain multiple matching field groups of the query statement in the field-topic mapping table of the database based on the feature information; the field-topic mapping table includes the mapping relationship between the preset topics of multiple databases and the fields of the database; the matching field groups correspond to the preset topics one by one.

[0088] The expansion module 203 is used to expand the matching fields in each matching field group to obtain multiple expanded matching field groups of the query statement.

[0089] The search module 204 is used to perform a multi-dimensional search in the database based on the multiple extended matching field groups to obtain a search result for each extended matching field group.

[0090] The fusion module 205 is used to perform weighted fusion on the search results based on the weights of the extended matching field groups to obtain the search data of the query statement.

[0091] The data retrieval device provided by the embodiment of the present invention obtains the matching fields of the query statement through feature information, realizes accurate matching according to the query intent, key information and context information, and improves the accuracy of the matching field group. The matching field group is obtained according to the field-subject mapping table, and the query statement is quickly located, which is beneficial to improving the efficiency and accuracy of data retrieval. Multi-dimensional retrieval is performed according to the extended matching field group, and retrieval according to the entities of the matching field group is realized, which is beneficial to improving the accuracy and efficiency of data retrieval. The retrieval results are weighted and fused according to the weight of the extended matching field group, so that the weight of each retrieval result can be accurately obtained, which is convenient for users to quickly find the retrieval data most relevant to the query statement.

[0092] In one embodiment, the matching module 202 is used to: obtain multiple matching field sequences of the query statement in the field-subject mapping table based on the feature information; obtain the correlation between the matching field and the query statement in each matching field sequence, sort the matching fields in descending order of correlation, and obtain a sorted matching field sequence; in each sorted matching field sequence, at least one matching field that is ranked first is used as a matching field group.

[0093] In one embodiment, the fusion module 205 is also used to: when the retrieval score of the retrieved data is lower than the set score, update the matching field group based on each sorted matching field sequence; perform iterative retrieval based on the updated matching field group until the retrieval score is greater than or equal to the set score, or the number of iterative retrievals reaches the set number; and use the retrieval data with the highest retrieval score as the final retrieval data.

[0094] In one embodiment, the extension module 203 is used to: perform named entity recognition analysis on each matching field of the matching field group to obtain the associated fields of the matching field; and obtain each extended matching field group based on all matching fields in each matching field group and the associated fields of the matching fields.

[0095] In one embodiment, the fusion module 205 is used to: obtain a basic weight of each extended matching field group based on a preset theme; determine an importance level of each extended matching field group based on feature information; and adjust the basic weight based on the importance level to obtain a weight of the extended matching field group.

[0096] In one embodiment, the retrieval module 204 is also used to: perform data cleaning on the initial text involved in the retrieval in the initial database to unify the format of the initial text; convert the cleaned initial text into a text vector; if the number of repetitions of the text vector in the initial database is lower than the set number of repetitions, construct an index structure of the text vector; and obtain a database based on the index structure and the text vector.

[0097] In one embodiment, the retrieval module 204 is used to: perform a one-dimensional search in the database based on an extended matching field group; perform a multi-dimensional search in the database based on multiple extended matching field groups; wherein, if there is an index structure matching the extended matching field in the database, the retrieval result is determined based on the similarity between the text vector of the index structure and the extended matching field; if there is an unstructured text vector matching the extended matching field in the database, fuzzy matching is performed on the unstructured text vector to obtain the retrieval result.

[0098] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the data retrieval method, which includes: obtaining feature information of the query statement based on the semantic analysis result of the user's query statement by the language model, wherein the feature information includes query intent, key information and context information; obtaining multiple matching field groups of the query statement in the field-topic mapping table of the database based on the feature information; the field-topic mapping table includes mapping relationships between preset topics of multiple databases and fields of the database; the matching field group corresponds to the preset topic one by one; expanding the matching field in each matching field group to obtain multiple extended matching field groups of the query statement; performing multi-dimensional retrieval in the database based on the multiple extended matching field groups to obtain the retrieval result of each extended matching field group; performing weighted fusion on the retrieval results based on the weight of the extended matching field group to obtain the retrieval data of the query statement.

[0099] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0100] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the data retrieval method provided by the above-mentioned methods, the method comprising: obtaining feature information of the query statement based on the semantic analysis result of the user's query statement based on a language model, the feature information including query intent, key information and context information; obtaining multiple matching field groups of the query statement in a field-topic mapping table of a database based on the feature information; the field-topic mapping table includes a mapping relationship between preset topics of multiple databases and fields of the database; the matching field groups correspond one-to-one to the preset topics; expanding the matching fields in each matching field group to obtain multiple extended matching field groups of the query statement; performing multi-dimensional search in the database based on the multiple extended matching field groups to obtain the retrieval results of each extended matching field group; and weighted fusion of the retrieval results based on the weights of the extended matching field groups to obtain the retrieval data of the query statement.

[0101] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0103] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data retrieval method, characterized in that: include: Acquire feature information of the query statement based on the semantic analysis result of the user's query statement by the language model, wherein the feature information includes query intent, key information and context information; Acquire multiple matching field groups of the query statement in a field-subject mapping table of a database based on the feature information; The field-topic mapping table includes a mapping relationship between a plurality of preset topics of the database and fields of the database; the matching field group corresponds to the preset topics one by one, and the field-topic mapping table is used to describe the semantics and applicable scenarios of each field in the database; the database is determined based on the following steps: Performing data cleaning on the initial texts involved in the retrieval in the initial database to unify the format of the initial texts; Convert the cleaned initial text into a text vector; If the number of repetitions of the text vector in the initial database is lower than the set number of repetitions, constructing an index structure of the text vector; the index structure is constructed based on a hierarchically navigable small world HNSW algorithm; Obtaining the database based on the index structure and the text vector; Expanding the matching fields in each of the matching field groups to obtain multiple expanded matching field groups of the query statement; Perform a multi-dimensional search in the database based on the plurality of extended matching field groups to obtain a search result for each of the extended matching field groups; The search results are weighted and fused based on the weights of the extended matching field groups to obtain the search data of the query statement.

2. The data retrieval method according to claim 1, characterized in that: The acquiring a plurality of matching field groups of the query statement in a field-subject mapping table of a database based on the feature information includes: Acquire multiple matching field sequences of the query statement in the field-topic mapping table based on the feature information; Obtaining the correlation between the matching field and the query statement in each matching field sequence, and sorting the matching fields in descending order of correlation to obtain a sorted matching field sequence; In each of the sorted matching field sequences, at least one matching field that is at the front is used as the matching field group.

3. The data retrieval method according to claim 2, characterized in that: After performing weighted fusion on the search results based on the weights of the extended matching field groups to obtain the search data of the query statement, the method further includes: When the search score of the search data is lower than a set score, updating the matching field group based on each of the sorted matching field sequences; Performing iterative search based on the updated matching field group until the search score is greater than or equal to the set score, or the number of iterative searches reaches the set number; The retrieval data with the highest retrieval score is taken as the final retrieval data.

4. The data retrieval method according to claim 1, characterized in that: The step of expanding the matching fields in each matching field group to obtain multiple extended matching field groups of the query statement includes: Performing named entity recognition analysis on each of the matching fields in the matching field group to obtain associated fields of the matching field; Each of the extended matching field groups is obtained based on all of the matching fields in each of the matching field groups and associated fields of the matching fields.

5. The data retrieval method according to claim 1, characterized in that: The weight of the extended match field group is determined based on the following steps: Obtaining a basic weight of each of the extended matching field groups based on the preset subject; determining an importance level of each of the extended matching field groups based on the feature information; The basic weight is adjusted based on the importance level to obtain the weight of the extended matching field group.

6. The data retrieval method according to claim 1, characterized in that: The performing a multi-dimensional search in the database based on the plurality of the extended matching field groups to obtain a search result for each of the extended matching field groups includes: Perform a dimensional search in the database based on one of the extended matching field groups; A multi-dimensional search is performed in the database based on a plurality of the extended matching field groups; wherein, if an index structure matching the extended matching field exists in the database, a search result is determined based on the similarity between a text vector of the index structure and the extended matching field; and if an unstructured text vector matching the extended matching field exists in the database, a fuzzy match is performed on the unstructured text vector to obtain the search result.

7. A data retrieval device, characterized in that: include: A feature information determination module, used to obtain feature information of a query statement based on a semantic analysis result of a user's query statement by a language model, wherein the feature information includes query intent, key information, and context information; A matching module, used for acquiring a plurality of matching field groups of the query statement in a field-subject mapping table of a database based on the feature information; The field-topic mapping table includes mapping relationships between a plurality of preset topics of the database and fields of the database; the matching field groups correspond to the preset topics one by one, and the field-topic mapping table is used to describe the semantics and applicable scenarios of each field in the database; An expansion module, used for expanding the matching fields in each of the matching field groups to obtain multiple extended matching field groups of the query statement; A retrieval module, configured to perform a multi-dimensional search in the database based on the plurality of extended matching field groups, and obtain a search result for each of the extended matching field groups; A fusion module, used for performing weighted fusion on the search results based on the weights of the extended matching field groups to obtain the search data of the query statement; The matching module is also used to clean the initial text involved in the retrieval in the initial database to unify the format of the initial text; convert the cleaned initial text into a text vector; if the number of repetitions of the text vector in the initial database is lower than the set number of repetitions, construct an index structure of the text vector; the index structure is constructed based on a hierarchically navigable small world HNSW algorithm; and obtain the database based on the index structure and the text vector.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the data retrieval method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data retrieval method according to any one of claims 1 to 6 is implemented.

Images