Data retrieval method, device and system

Abstract

The embodiment of the present invention provides a data retrieval method, device and system, which relates to the field of data processing technology. The method includes: obtaining an original search statement and an associated search statement to obtain a target search statement; performing keyword extraction on the target search statement to obtain a search keyword. Based on the search keyword, a search is performed in a knowledge graph database to obtain a first search result including a first data file; based on the search keyword and the target search statement, a search is performed in a basic database to obtain a second search result including a second data file; based on the statement feature vector of the target search statement, a search is performed in a vectorized database to obtain a third search result including a third data file; based on the interactive information of each data file, a target search result including the target data file is determined from the first search result, the second search result and the third search result to improve the accuracy of data retrieval and enhance user experience.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data retrieval method, device and system. Background Art

[0002] With the development of information technology, the application of knowledge-based service systems is becoming increasingly widespread. For example, knowledge sharing systems provide data retrieval services. These systems can provide users with data files that meet their search requirements. Today's knowledge sharing systems contain massive amounts of data files. These multi-source data files are distributed and stored in different databases due to their varying data structure. Therefore, knowledge sharing systems must accurately retrieve the data files that meet user needs from this complex and multi-source data set.

[0003] In related technologies, word matching is performed between the user's search statements and data files. When dealing with complex search scenarios (such as search scenarios for broad topics), this search method only focuses on keyword matching and cannot deeply understand the user's deep intentions, making it difficult to capture the user's deep intentions. As a result, data files that are highly relevant to user needs cannot be effectively retrieved, which reduces the accuracy of the search and leads to a poor user experience. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to provide a data retrieval method, device, and system to improve the accuracy of data retrieval. The specific technical solution is as follows:

[0005] In a first aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides a data retrieval method, which is applied to a server; the method includes: obtaining an original search statement input by a user, and an associated search statement of the original search statement, to obtain a target search statement; wherein the keywords in the associated search statement have an associated relationship with the keywords in the original search statement; performing keyword extraction on the target search statement to obtain a search keyword; based on the search keyword, searching in a knowledge graph database to obtain a first search result including a first data file; wherein the knowledge graph database includes various types of data files and the associated relationship between each data file; based on the search keyword and the target search statement, searching in a basic database to obtain a second search result including a second data file; wherein the basic database includes various types of data files; generating a sentence feature vector of the target search statement, and based on the sentence feature vector, searching in a vectorized database to obtain a third search result including a third data file; wherein the vectorized database includes text feature vectors of texts of multiple data files; based on the user's interactive information on each data file, determining a target search result including the target data file from the first search result, the second search result, and the third search result.

[0006] In a second aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides a data retrieval device, which is applied to a server; the device includes: a retrieval statement generation module, which is used to obtain an original retrieval statement input by a user, and an associated retrieval statement of the original retrieval statement, to obtain a target retrieval statement; wherein the keywords in the associated retrieval statement are associated with the keywords in the original retrieval statement; a keyword extraction module, which is used to extract keywords from the target retrieval statement to obtain retrieval keywords; a first retrieval module, which is used to search in a knowledge graph database based on the retrieval keywords to obtain a first retrieval result including a first data file; wherein the knowledge graph database includes various types of data files, and the associations of each data file relationship; a second retrieval module, used to search in the basic database based on the retrieval keyword and the target retrieval statement, and obtain a second retrieval result including a second data file; wherein the basic database includes various types of data files; a third retrieval module, used to generate a sentence feature vector of the target retrieval statement, and based on the sentence feature vector, search in the vectorized database, and obtain a third retrieval result including a third data file; wherein the vectorized database includes text feature vectors of texts of multiple data files; a target retrieval result acquisition module, used to determine a target retrieval result including a target data file from the first retrieval result, the second retrieval result and the third retrieval result based on the user's interactive information on each data file.

[0007] In a third aspect, in order to achieve the above-mentioned purpose, an embodiment of the present application provides a data retrieval system, which includes: a terminal and a server; the terminal is used to obtain an original search statement input by a user and send the original search statement to the server; the server is used to obtain a target retrieval result of the original search statement after receiving the original search statement, and send the target retrieval result to the terminal; wherein the target retrieval result includes a target data file; the target retrieval result is determined according to any of the data retrieval methods described above; the terminal is also used to display the target data file in a display interface after receiving the target retrieval result.

[0008] An embodiment of the present invention also provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor is used to implement any of the above-mentioned data retrieval method steps when executing the program stored in the memory.

[0009] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned data retrieval methods is implemented.

[0010] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned data retrieval methods.

[0011] The technical solution provided by the embodiment of the present invention expands the original search statement to obtain the associated search statement of the original search statement, and uses the original search statement and the associated search statement to perform retrieval, which can not only process the needs clearly expressed by the user, but also process and meet the user's potential deep needs, understand the user's deep intentions, and then provide more comprehensive and accurate retrieval results to improve the user experience. In addition, searching in a vectorized database effectively processes unstructured and colloquial search statements, and by converting natural language search statements into vectorized statement feature vectors, deep semantic understanding and fast similarity retrieval of complex searches are achieved, thereby improving the processing power and response speed of natural language searches and improving the user experience. The target retrieval results are determined in combination with the user's personalized information, so as to provide the user with a data file that is more in line with the user's retrieval intention, further improve the accuracy of data retrieval, and improve the user experience.

[0012] Of course, it is not necessary to achieve all of the advantages described above simultaneously in order to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0014] Figure 1 A structural diagram of a data retrieval system provided in an embodiment of the present application;

[0015] Figure 2 A flowchart of the first data retrieval method provided in an embodiment of the present application;

[0016] Figure 3 A schematic diagram of a method for generating an associated search statement provided in an embodiment of the present application;

[0017] Figure 4 A flowchart of constructing a vectorized database and performing retrieval provided in an embodiment of the present application;

[0018] Figure 5 A schematic diagram of a data retrieval method provided in an embodiment of the present application;

[0019] Figure 6 A flowchart of a second data retrieval method provided in an embodiment of the present application;

[0020] Figure 7 A structural diagram of a data retrieval device provided in an embodiment of the present application;

[0021] Figure 8 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0022] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on this application are within the scope of protection of the present invention.

[0023] like Figure 1As shown, an embodiment of the present application provides a data retrieval system, including a server 101 and a terminal 102. Server 101 is in communication with terminal 102. Terminal 102 is a user terminal, such as a mobile phone or computer. When a user needs to retrieve a data file, they input a search statement (i.e., an original search statement) into terminal 102. The user's original search statement may vary depending on the user's needs in different application scenarios. After receiving the original search statement, terminal 102 sends the original search statement to server 101. Server 101 is deployed with multiple databases storing various types of data files. Server 101 searches each database based on the received original search statement to obtain target search results. The target search results include target data files that match the original search statement. For example, in a financial business scenario, when a user searches for information related to opening a securities account, the original search statement may be "I want to inquire about the securities account opening process" or "How to register a securities account," and the target data files may be multiple articles describing the securities account opening process. Server 101 sends the target search results to terminal 102. The terminal 102 displays the received target data file on the display interface.

[0024] Based on the data retrieval system provided in the embodiment of the present application, the original search statement is expanded to obtain the associated search statement of the original search statement, and the original search statement and the associated search statement are used for retrieval, which can not only process the needs clearly expressed by the user, but also process and meet the user's potential deep needs, realize understanding the user's deep intentions, provide more comprehensive and accurate retrieval results, and improve user experience. In addition, searching in a vectorized database effectively processes unstructured and colloquial search statements, and realizes deep semantic understanding and fast similarity retrieval of complex retrieval by converting natural language retrieval statements into vectorized sentence feature vectors, thereby improving the processing power and response speed of natural language retrieval and improving user experience. The target retrieval results are determined in combination with the user's personalized information, so as to provide the user with a data file that is more in line with the user's retrieval intention, further improve the accuracy of data retrieval, and improve user experience.

[0025] See also Figure 2 , Figure 2 A flowchart of a data retrieval method provided in an embodiment of the present application, applied to Figure 1 The data retrieval system shown. The method comprises the following steps:

[0026] S201: The terminal obtains an original search statement input by the user.

[0027] S202: The terminal sends an original search statement to the server.

[0028] S203: After receiving the original search statement, the server generates a related search statement of the original search statement to obtain a target search statement. The keywords in the related search statement are related to the keywords in the original search statement.

[0029] S204: The server extracts keywords from the target search statement to obtain search keywords.

[0030] S205: The server searches the knowledge graph database based on the search keyword and obtains a first search result including the first data file. The knowledge graph database includes various types of data files and the association relationship between the data files.

[0031] S206: The server searches the basic database based on the search keyword and the target search statement to obtain a second search result including a second data file. The basic database includes various types of data files.

[0032] S207: The server generates a sentence feature vector of the target search sentence, and searches the vectorized database based on the sentence feature vector to obtain a third search result including a third data file. The vectorized database includes text feature vectors of texts of multiple data files.

[0033] S208: The server determines a target search result including the target data file from the first search result, the second search result, and the third search result based on the user's interaction information on each data file.

[0034] S209: The server determines the display order of the target data files.

[0035] S2010: The server sends the target data file and display order to the terminal.

[0036] S2011: The terminal displays the target data files in a display order.

[0037] Based on the data retrieval method provided in the embodiment of the present application, the original search statement is expanded to obtain the associated search statement of the original search statement, and the original search statement and the associated search statement are used for retrieval, which can not only process the needs clearly expressed by the user, but also process and meet the user's potential deep needs, and understand the user's deep intentions, thereby providing more comprehensive and accurate retrieval results and improving the user experience. In addition, by searching in a vectorized database, unstructured and colloquial search statements can be effectively processed, and by converting natural language search statements into vectorized statement feature vectors, deep semantic understanding and fast similarity retrieval of complex searches can be achieved, thereby improving the processing power and response speed of natural language searches and improving the user experience. The target retrieval results are determined in combination with the user's personalized information, so as to provide the user with a data file that is more in line with the user's retrieval intention, further improving the accuracy of data retrieval and improving the user experience.

[0038] With respect to step S201 and step S202, when the user enters a search statement in the terminal, the terminal obtains the characters entered by the user and sends the characters entered by the user to the server. The server obtains each search statement containing the characters entered by the user from the high-frequency search statement library, selects the first fourth number of search statements in descending order of the frequency of use of each search statement by the user, obtains candidate search statements, and sends the candidate search statements to the terminal. The fourth number is set by the technical staff according to the needs. The high-frequency search statement library is set based on each search statement of the user's historical search and the frequency of use of each search statement. For example, the search statements and frequencies that are used more frequently in the user's historical search are recorded in the high-frequency search statement library.

[0039] The terminal displays candidate search statements on a display interface for the user to select. If the candidate search statement is the search statement the user intends to use, the user directly selects the candidate search statement. Upon receiving a selection instruction from the user for the displayed candidate search statement, the terminal determines that the candidate search statement selected by the user is the original search statement. If the candidate search statement is not the search statement the user intends to use, the user continues to input characters into the terminal. Upon receiving a completion instruction from the user, the terminal determines that the characters input by the user are the original search statement. The terminal sends the original search statement to the server. The server receives the original search statement sent by the terminal.

[0040] In the field of data retrieval, quickly and accurately guiding users to complete their search intentions is the key to improving the search experience. In the embodiment of the present application, an advanced intelligent search field association completion mechanism is proposed. Based on the user's historical search data and user behavior analysis, the high-frequency search statement library is screened and regularly updated. The immediacy and relevance of the search statement completion suggestions are ensured through high-frequency search statements. In addition, the search statement completion mechanism takes effect immediately when the user starts to enter the search statement. By matching the characters entered by the user with the search statements in the high-frequency search statement library, accurate completion suggestions (i.e., candidate search statements) are provided in real time, which significantly shortens the time it takes for the user to enter the search statement. By accurately matching the user's search intention, the convenience and efficiency of the search process are greatly improved.

[0041] In step S203, after the server obtains the original search statement, it generalizes the original search statement to obtain the associated search statements of the original search statement. The original search statement and the associated search statements are collectively referred to as target search.

[0042] In one implementation, the server obtains the original search statement entered by the user as the target search statement. It then uses a word segmentation algorithm to extract keywords from the original search statement, obtaining original keywords. The word segmentation algorithm used is the Jieba (jieba) word segmentation algorithm. For different application scenarios, a word segmentation library tailored to that application scenario is pre-configured. For example, for a financial application scenario, a word segmentation library containing proper nouns in the financial field is pre-configured. The word segmentation algorithm is then used to segment the original search statement using the dedicated word segmentation library, obtaining original keywords that better match the application scenario and improving the accuracy of keyword extraction.

[0043] Determine the node to which the original keyword belongs in the knowledge graph database and the nodes associated with that node. The keyword represented by the determined node is the associated keyword. Use the associated keyword as a prompt for the Large Language Model (LLM), or generate a phrase containing the associated keyword as a prompt for the LLM. Input the original search statement and the prompt into the LLM to generate the associated search statement output by the LLM. The associated search statement also serves as the target search statement.

[0044] A knowledge graph database is a graph database that includes multiple nodes representing different entities, edges connecting the nodes, and node attribute information. Each node in a knowledge graph database is a potential query point, and the multiple nodes together form a systematic and structured approach to information exploration. Entities include keywords, articles, videos, etc. A knowledge graph database also includes keywords and the relationships between keywords. The existence of an edge between two nodes indicates a relationship between the entities represented by the two nodes. The node attribute information represents the relevant content of the entity. For example, if the entity is an article, the attribute information includes the article's tags, type, keywords, and so on. The knowledge graph database is Nebula. Nebula is an open-source, distributed, and easily scalable native knowledge graph database.

[0045] See also Figure 3 The query content refers to the original search statement. Query keyword extraction and analysis are performed on the original search statement "I want to inquire about the securities account opening process." This involves extracting keywords from the original search statement, resulting in original keywords (i.e., primary keywords) including "securities account number," "account opening process," "query," "account opening," and "process." The original keywords are used as entities for inference association. This involves searching the knowledge graph database to identify other entities with edges to the entity, i.e., associated keywords associated with the original keywords. These associated keywords include "document requirements," "verification process," "account activation," "international account opening," "account opening steps," and "securities account type."

[0046] Based on the query and related keywords, the question is generalized. Specifically, the related keywords are used as prompts, and LLM is used to generate new search statements based on the original search statement, resulting in four related search statements (i.e., Generalized Question 1, Generalized Question 2, Generalized Question 3, and Generalized Question 4). The prompts can be directly the related keywords mentioned above, or phrases containing the related keywords. For example, using the prompt "I want to inquire about the securities account opening process" as a template, a search statement containing "document requirements" is generated. Generalized Question 1 is described as "What personal information and materials are required to open a securities account?" Keywords for Generalized Question 1 include "securities account," "document requirements," and "personal information." Generalized Question 2 is described as "What are the verification and activation steps in the securities account opening process?" Keywords for Generalized Question 2 include "verification process," "account activation," and "account opening steps." Generalized Question 3 is described as "What are the differences in the opening process for different types of securities accounts (e.g., personal accounts and institutional accounts)?" Keywords for Generalized Question 3 include "securities account type," "account opening process," "personal account," and "institutional account." Generalized question 4 is described as “What are the differences in the procedures and requirements for opening securities accounts in different countries or regions?” The keywords of generalized question 4 include “international account opening,” “process differences,” and “regional requirements.”

[0047] In the embodiment of the present application, the knowledge in the knowledge graph database is stored in a structured form and naturally has the ability of logical reasoning. In the embodiment of the present application, the knowledge graph is innovatively introduced as an auxiliary tool, and its structured knowledge and logical reasoning ability are used to obtain related keywords that have an association relationship with the original search statement, and the related keywords are used as prompt words for the large language model to set a clear path for the generalization process of the large language model and provide a clear guide. That is, it cleverly combines the logical reasoning ability of the knowledge graph database and the semantic understanding of the large language model, performs deep semantic expansion on the original search statement input by the user, and gives the search statement a clear generalization direction through the precise logical reasoning of the knowledge graph database, generating more relevant and exploratory related search statements.

[0048] Intelligently associating raw keywords with the knowledge graph database not only retrieves existing relevant knowledge within the database but also further refines related keywords. This processing method, which integrates the logical reasoning capabilities of the knowledge graph database and leverages the rich relationships and attribute information between entities within the knowledge graph database, ensures that the related keywords used in generalized search statements are not merely superficial linguistic connections, but are based on deep, accurate logical relationships between entities. This can expand the breadth of user data retrieval, greatly enrich the in-depth application of the knowledge graph database, and improve the recall rate of search algorithms.

[0049] In step S204, after obtaining the target search statement, the server performs keyword extraction on the target search statement to obtain search keywords. The method for extracting keywords from the target search statement is similar to the method for extracting keywords from the original search statement, as described in the previous embodiment. The search keywords include the keywords from the original search statement and the keywords from the associated search statement.

[0050] against Figure 3 In the embodiment, keywords are extracted for generalized question 1, generalized question 2, generalized question 3, and generalized question 4, and the obtained search keywords include: "securities account", "query", "account opening", "process", "document requirements", "personal information", "verification process", "account activation", "account opening steps", "securities account type", "account opening process", "institutional account", "personal account", "international account opening", "process differences", and "regional requirements".

[0051] For step S205, the server uses the search keyword as a node and searches for a node with an edge to the node in the knowledge graph database. The data file represented by the determined node is the first data file that matches the search keyword, and the first data file also matches the original search statement.

[0052] In some embodiments, the first search result also includes: the initial correlation between the first data file and the original search statement. The server obtains the first search result in the following manner: a data file that is associated with the search keyword is obtained from the knowledge graph database to obtain the first data file. For example, the node whose attribute information contains the search keyword is determined, and the node that has an edge with the node to which the search keyword belongs is determined. The data file represented by the determined node is the first data file. The server determines the search keyword contained in the text of the first data file, and the first data file includes at least one search keyword.

[0053] Various data file types include text files, video files, audio files, and image files. Text files include TXT (Text Document) files, JSON (JavaScript Object Notation) files, HTML (Hypertext Markup Language) web pages, and PDF (Portable Document Format) files. For text files, such as TXT files, the text in the data file is the data file itself. For other file types, the text in the data file is the attribute information of the data file. For example, for video files, the text in the video file is the video file's tags and description.

[0054] Calculate a target parameter for each search keyword in the text of the first data file. The target parameter indicates the importance of the search keyword in the text. For example, the target parameter for the search keyword is the TF-IDF (Term Frequency–Inverse Document Frequency) of the search keyword. TF (Term Frequency) indicates the frequency of the search keyword in the text of the first data file, and IDF (Inverse Document Frequency) indicates the rarity of the search keyword in the text of each first data file.

[0055] The weighted sum of the target parameters of each search keyword in the text of each first data file is calculated to obtain the initial relevance of the first data file to the original search statement. The keyword weight is pre-set by the technician based on business needs. If the search keyword weight is not pre-set, the default value is determined for the search keyword weight.

[0056] Based on the above processing, the search keywords are combined with the knowledge graph database for intelligent association. This not only obtains data files related to the search keywords in the knowledge graph database, but also further obtains data files related to the associated keywords. This not only expands the breadth of user data retrieval, greatly enriches the in-depth application of the knowledge graph database, but also improves the recall rate of the search algorithm. After calculating the initial relevance of the first data file and the original search statement, the initial relevance of the first data file and the original search statement is used to filter out the target data files, improving the accuracy of data retrieval.

[0057] In step S206, the basic database is a database that stores various types of data files. For example, if an Elastic Search engine is deployed on the server, the basic database is a relational database accessible by the Elastic Search engine. Then, a search is performed in the basic database using the Elastic Search engine to obtain the second data file. The basic database is constructed based on user question and answer files, experience files, and product files.

[0058] In some embodiments, the second search result also includes: the initial relevance of the second data file to the original search statement. The server obtains the second search result in the following manner: in order to improve the scope of data retrieval, the server obtains synonyms of the search keyword, and uses the search keyword and its synonyms as target keywords. Determine the data files that meet the first search condition or the second search condition from the basic database, and obtain candidate data files, and the candidate data files match the original search statement. The first search condition includes: the text of the data file includes each keyword in the original search statement or the associated search statement, and the candidate data file that meets the first search condition matches the original search statement or the associated search statement. The second search condition is: the text of the data file includes at least one target keyword, the target keyword is obtained by keyword extraction of the original search statement and the associated search statement, and the candidate data file that meets the second search condition matches the original search statement or the associated search statement.

[0059] The built-in word segmenter of the Elastic Search engine does not perform well when processing complex Chinese searches and cannot meet the requirements for accuracy and efficiency. In the face of growing and changing language usage habits, the customization and maintenance workload of the built-in word segmenter is huge, which increases the cost of data retrieval. In the embodiment of the present application, the search keywords are obtained by processing the search statements using the jieba word segmentation algorithm, that is, by deeply integrating NLP (Natural Language Processing) technology (i.e., the jieba word segmentation algorithm) into the search process of the Elastic Search engine, efficient and accurate word segmentation for Chinese searches is achieved. Compared with the built-in word segmenter of the Elastic Search engine, the jieba word segmentation can provide more flexible and accurate word segmentation strategies, support custom dictionaries (i.e., word segmentation libraries for different application scenarios), can quickly adapt to emerging words and proprietary words, and greatly improve the processing capabilities of Chinese texts. Through deep NLP processing of search statements, not only does this address the accuracy issues of Chinese search, but it also dynamically adjusts word segmentation strategies to effectively address the diversity and complexity of Chinese search statements, ensuring high relevance and accuracy of search results. This reduces the time and cost of manually maintaining custom word segmenters in the Elastic Search engine, enabling more efficient adaptation and processing of complex natural language searches, significantly improving user search experience and satisfaction. Furthermore, combining the match_phrase method (i.e., the first search condition) with the multi_match method significantly enhances search accuracy and flexibility.

[0060] After obtaining the candidate data files, the server calculates the target parameter for each target keyword in the text of each candidate data file. The target parameter for a keyword represents the importance of the keyword in the text and is the TF-IDF of the target keyword in the text. The server then calculates the weighted sum of the target parameters for each target keyword in the text of the candidate data file as the initial relevance of the candidate data file to the original search statement. Keyword weights are pre-set by technical personnel based on business needs.

[0061] To improve the accuracy of the second search results and eliminate irrelevant information caused by fuzzy searches, the candidate data files are screened for irrelevant search results. For example, the server identifies, from each candidate data file, those whose initial relevance to the original search statement is greater than a first threshold, and obtains the second data file. Data files with initial relevance greater than the first threshold are highly consistent with the original search statement and are therefore likely to be included in the target search results ultimately displayed to the user. By setting a dynamic threshold (i.e., the first threshold), the relevance and quality of the second search results are significantly enhanced. The first threshold is determined based on the maximum relevance of each candidate data file to the original search statement. For example, the first threshold can be 10% of the maximum relevance of each candidate data file to the original search statement, or 15% of the maximum relevance of each candidate data file to the original search statement.

[0062] In the embodiments of the present application, complex Boolean logic is used to carefully construct a series of search conditions (i.e., the first search condition and the second search condition). These search conditions can accurately locate the characteristics of specific data files and perform in-depth filtering of the search results to ensure that the selected data files accurately match the user's search intent. This multi-level, dynamically adjusted screening mechanism not only effectively eliminates information that does not match the user's search intent, but also greatly shortens the time it takes for users to find the required data files in the massive search results. Furthermore, through the integrated design of multiple strategies such as precise phrase priority search (i.e., setting the first search condition), retaining keyword search flexibility (i.e., setting the second search condition), and dynamic weight adjustment (i.e., setting different weights for different keywords), not only does it improve the responsiveness to precise search needs, but it also provides greater flexibility for handling a wide range of search needs.

[0063] In some embodiments, when the Elastic Search engine is deployed in a server, a high-frequency search statement library is set by adding a completion type field to the ElasticSearch engine, and the high-frequency search statement library can be stored and managed by the Elastic Search engine.

[0064] For step S207, the vectorized database includes the cluster centers of each vector group; each vector group includes text feature vectors from multiple data files. The vectorized database is built based on data files uploaded to the server by business personnel. Uploaded data files include user-published question and answer files, experience files, product files, and so on.

[0065] In some embodiments, the server establishes a vectorized database in the following manner:

[0066] Due to the different types of data files, the server identifies different types of data files and uses different methods to obtain the text of the data files for different types of data files. For example, data files include TXT files, PDF files, JSON files, and HTML web page files. For TXT files and JSON files, the text is read directly. For PDF files, OCR (Optical Character Recognition) is used to parse the PDF file to obtain the text of the PDF file. For HTML web page files, the content of the HTML web page file is converted into plain text format through a parser, and the converted text is cleaned. For example, HTML tags are removed, character encoding is converted, date and number formats are normalized, and redundant spaces and line breaks are eliminated to obtain the text of the HTML web page file.

[0067] If the retrieved data files contain text in different languages, language detection is performed and text in languages ​​other than the specified language is translated to produce text in the specified language. The specified language is set by technical personnel based on business needs. For example, if the data retrieval service is provided to Chinese users, the specified language is Chinese. If the data retrieval service is provided to English users, the specified language is English.

[0068] Based on the above processing, considering the high flexibility and adaptability required when reading various types of data files, it is possible to process various data files and set specific content extraction strategies for different types of data files to obtain the text of the data file, ensuring the cleanliness and consistency of the obtained text.

[0069] Since the output results (i.e., text) of text normalization for different types of data files have different lengths, a recursive segmentation strategy is adopted to process longer texts in order to preserve the rich semantic hierarchy in long texts. For example, the text is divided into multiple text blocks through TextSplitter, and there is an overlap between two adjacent text blocks. This avoids semantic faults near the cutting points when dividing the file, and better maintains the overall contextual coherence of the text. For example, the text of the data file is divided into multiple text blocks in a way that 300 words are a text block and two adjacent text blocks overlap by 50 words.

[0070] Based on the above processing, not only can texts exceeding the length limit be processed, but it also ensures that when faced with massive text data, the semantic information of each part can be captured and utilized as much as possible, thereby improving the accuracy of subsequent data retrieval.

[0071] Each text block is vectorized independently to ensure that the semantics of each part can be preserved. For each text block in the data file, the text block is processed based on the text vectorization model to obtain the text feature vector of the text block.

[0072] The core principle of text embedding technology is to convert natural language text into feature vectors in a high-dimensional space that represent semantic information. This process of converting text into feature vectors typically relies on deep learning models (i.e., text vectorization models) for natural language processing (NLP). By learning from large amounts of text data, deep learning models can understand the meaning of words, phrases, and even entire paragraphs, and convert this understanding into feature vectors. Examples of text vectorization models include the BERT-Chinese (Bidirectional Encoder Representations from Transformers-Chinese) model and the GPT (Generative Pre-trained Transformer) model.

[0073] Chinese text lacks distinct spaces to separate words, so character processing is more suited to the linguistic characteristics of Chinese. In this embodiment, the text vectorization model used is the BERT-Chinese model. BERT-Chinese typically performs character tokenization, treating each Chinese character (including punctuation) as a token. By pre-training on a wide range of Chinese corpora, BERT-Chinese is able to understand and process a wide variety of Chinese text and contexts.

[0074] The text vectorization model includes a word segmenter and an encoder. The word segmenter is the WordPiece algorithm, and the encoder is the Transformer. For each text block, the WordPiece algorithm is used to divide the text block into a series of tokens. Assuming that the text block T is divided into N tokens, the text block T is expressed as the following formula (1):

[0075] (1);

[0076] t N The Nth word in the text block. Based on the i-th word t i The position in the word segmenter's vocabulary is obtained to represent the word t i The one-dimensional vector of , and the one-dimensional vector is reduced in dimension to obtain the word t i Vectorized basic vector representation. i The vectorized basic vector representation is mapped to obtain the word t i The initial eigenvector M i , M i It is expressed as the following formula (2):

[0077] (2);

[0078] E i Represents the basic vector representation of the i-th word element vectorization; WE i represents the word embedding of the i-th word; PE i Represents the position embedding of the i-th word; SE i Represents the fragment embedding of the i-th word; fragment embedding is used to distinguish different sentences when the i-th word appears in multiple sentences; W, P, S represent different model parameters of the text vectorization model.

[0079] For each word, the initial feature vector of the word is input to the encoder. Each layer of encoder will fuse the information of all words around the word and output the word feature vector of the word obtained by the encoder of this layer. If the text vectorization model has L layers of encoders, the input of each layer of encoder is the output of the previous layer. The word feature vector of the i-th word output by the k-th layer encoder is expressed as the following formula (3):

[0080] (3);

[0081] Represents the word-unit feature vector of the i-th word-unit output by the k-th layer encoder; Represents the word feature vector of the i-th word output by the k-1 layer encoder. TransformerLayer represents the k-th layer encoder. =M i , that is, the input of the first layer encoder is M i (i.e. the output of the input layer).

[0082] For each text block, the weighted sum of the word element feature vectors of each word element in the text block is calculated according to the preset weights to obtain the text feature vector of the text block. The weights of each word element are set by technical personnel according to business needs.

[0083] Each text block of the text in each data file is vectorized to obtain multiple text feature vectors for the text in the data file. Dimensionality reduction is then performed on the multiple text feature vectors to reduce the high-dimensional vector space and simplify the subsequent quantization process. For example, each text feature vector is divided into M sub-vectors. For example, if the text feature vector is D-dimensional, the M sub-vectors obtained by division are D / M-dimensional. The weighted sum of the M sub-vectors is then calculated to obtain the final text feature vector.

[0084] Cluster multiple text feature vectors to obtain multiple vector groups. For example, use the k-means clustering algorithm (k-means clustering algorithm) to cluster multiple text feature vectors to obtain multiple vector groups. Each vector group includes a cluster center and multiple text feature vectors.

[0085] A random algorithm is used to generate the ID (identification) of the data file to which the text belongs, as well as the ID of each text feature vector. Furthermore, the cluster center of each vector group, the text feature vectors contained in that vector group and their IDs, as well as the data file to which each text feature vector belongs and its ID are recorded to obtain a vectorized database.

[0086] Based on the above processing, the BERT model can convert massive data files into text feature vectors containing rich semantic information through the above process, that is, it can convert various types of data files (such as structured market data, semi-structured storage knowledge, unstructured document files, etc.) into vectorized data in a unified format. The feature vectors are stored in the vectorized database for search, and the unique vector index (ie ID) of each data file is also stored to facilitate subsequent searches, thereby improving the efficiency and accuracy of data retrieval.

[0087] In some embodiments, the third search result further includes: an initial relevance between the third data file and the original search statement; cluster centers of each vector group in the vectorized database; and text feature vectors of the text of each vector group including multiple data files. Accordingly, the server determines the third search result in the following manner:

[0088] The server vectorizes the target search sentence based on the text vectorization model to obtain the sentence feature vector of the target search sentence. Specifically, the server inputs the target search sentence into the text vectorization model, uses the word segmenter of the text vectorization model to segment the target search sentence, and obtains each word element of the target search sentence. For each word element of the target search sentence, the word element is mapped to obtain the initial feature vector of the word element. The encoder of the text vectorization model is used to encode the initial feature vector of the word element to obtain the word element feature vector of the word element. According to the preset weight, the weighted sum of the word element feature vectors of each word element of the target search sentence is calculated to obtain the sentence feature vector of the target search sentence.

[0089] For each sentence feature vector, calculate the similarity between the sentence feature vector and the cluster centers of each vector group in the vectorized database as the first similarity. The first similarity represents the degree of relevance between the text feature vector in the vector group and the target search sentence. According to the order of the first similarity from high to low, determine the vector groups to which the first number of cluster centers belong, and obtain the candidate vector group of the sentence feature vector. For example, the sentence feature vector of the target search sentence is denoted as V Q , then the candidate vector group is recorded as q(V Q ),q(V Q ) includes multiple text feature vectors.

[0090] The similarity between the sentence feature vector and each text feature vector in the candidate vector group is calculated as the second similarity. The second similarity between the sentence feature vector of the i-th target retrieval sentence and each text feature vector in the candidate vector group is expressed as the following formula (4):

[0091] (4);

[0092] d(V Q , V i ) represents the second similarity between the sentence feature vector of the i-th target retrieval sentence and each text feature vector in the candidate vector group; q(V Qm ) represents the mth text feature vector in the candidate vector group; M represents the number of text feature vectors; q(V im ) represents the sentence feature vector of the i-th target retrieval sentence; Indicates the second similarity between the sentence feature vector of the i-th target retrieval sentence and the m-th text feature vector in the candidate vector group.

[0093] The text feature vectors in each candidate vector group are sorted in descending order of the second similarity, and the first second number of text feature vectors are determined from each text feature vector after being sorted side by side to obtain the matching feature vector of the sentence feature vector. If the text feature vectors in the candidate vector group have a high degree of correlation with the sentence feature vector of the target search sentence, that is, the data file to which the text feature vectors in the candidate vector group belong has a high degree of correlation with the target search sentence, then the data file to which the matching feature vector belongs has a high degree of correlation with the target search sentence, and then the data file to which the matching feature vector belongs is determined as the third data file. There may be multiple matching feature vectors in the text feature vector of the third data file, and then the maximum value of the second similarity between the sentence feature vector in the third data file and the matching feature vector is determined as the initial correlation between the third data file and the original search sentence. The above-mentioned method for calculating the similarity between feature vectors is a cosine similarity algorithm, Euclidean distance, etc.

[0094] See also Figure 4 , Figure 4 This is a flowchart for constructing and searching a vectorized database, provided in an embodiment of the present application. The server retrieves multi-source knowledge from the knowledge center. The knowledge center is a platform for users to publish data files. This multi-source knowledge includes user-published question-and-answer files, experience files, and product files. The server reads the content, i.e., retrieves the text of each data file.

[0095] The server segments the data file's text and constructs Documents, which are the resulting text blocks. For each text block, the server performs Text2vec (text-to-vector) text vectorization (emdedding), mapping the text blocks to obtain the vectorized features (i.e., text feature vectors) for each Document. A Faiss index (i.e., the data file ID) is constructed, recording the cluster center of each vector group, the text feature vectors contained in that vector group, the ID of each text feature vector, and the data file to which each text feature vector belongs and its ID, resulting in a vectorized database.

[0096] The server receives the user's search query (i.e., the original search statement and related search statements) and obtains the vectorized features of the query, that is, the statement feature vectors of the original search statement and related search statements. Using the statement feature vectors, it searches the Faiss database (i.e., the vectorized database) to obtain the top K relevant documents (i.e., text blocks) and their context. The context of a text block refers to other text blocks that belong to the same data file as the text block. Together, the text block and the context define the data file.

[0097] In this embodiment, a vectorized database is created to convert various data files into text features in a unified format. The BERT-Chinese model converts the user's target search sentence into a sentence feature vector with the same dimension as the vectorized data volume. Using the IPQ (Inverted Product Quantization) search algorithm, the vectorized database is searched to obtain a third data file, enabling understanding and analysis of the deep semantics of the user's search query and returning more relevant and accurate search results.

[0098] Regarding step S208, after obtaining the first search results, the second search results, and the third search results, the server combines the user's interaction information on each data file and determines a target search result that includes the target data file from the first search results, the second search results, and the third search results. The user's interaction information on a data file includes at least one of the number of likes, favorites, views, and comments on the data file.

[0099] In one implementation, the server filters out duplicate data files from the first data file, the second data file, and the third data file to obtain matching data files. Based on the user's interaction information for each matching data file, the interaction factor of the matching data file is calculated. For example, the average of the number of likes, favorites, views, and comments of the user for the data file is calculated, and the calculated interaction factor is normalized to [0,1] to obtain the interaction factor of the matching data file. The statistical value (such as the average, weighted sum) of the interaction factor of the matching data file and the initial relevance of the matching data file to the original search statement is calculated to obtain the target relevance of the matching data file to the original search statement. In descending order of the target relevance to the original search statement, the first five matching data files are selected from the matching data files to obtain the target data file, thereby obtaining the target search result.

[0100] In another implementation, a multi-channel recall strategy involves using multiple algorithms simultaneously to retrieve search results from multiple databases, aiming to enhance the comprehensiveness and accuracy of the search results. This strategy may recall duplicate data files. Therefore, the search results from the multi-channel recall are deduplicated. This involves filtering out duplicate data files from the first, second, and third data files to obtain matching data files. This prevents duplicate data files from impacting the user experience and reduces the burden of subsequent processing.

[0101] For example, each data file has a unique identifier (i.e., ID). Create a set to record the identifiers of each matching data file. Iterate through each recalled data file. If a data file's identifier already exists in the set, discard the data file. If a data file's identifier does not exist in the set, add the data file's identifier to the set and retain the data file. By cleaning the retrieval results of multiple recalls, the uniqueness of the data files in the retrieval results is guaranteed.

[0102] After obtaining matching data files, they need to be reranked. The results of this reranking directly impact the user's search experience and satisfaction. The goal of this reranking process is to leverage the context of the user's search and incorporate user interaction information to further filter and sort the matching data files, ensuring that the most relevant and useful data files are presented to the user first.

[0103] In the embodiment of the present application, an interaction factor and a time decay factor are introduced to re-screen and re-rank the matching data files. Exemplarily, based on the initial relevance of each matching data file to the original search statement and the user's interaction information for the matching data file, the intermediate relevance of the matching data file to the original search statement is calculated. User interaction information is an important indicator for measuring the quality and popularity of data files. The server regularly obtains the interaction information of each data file to ensure the accuracy and timeliness of the sorting. For each matching data file, the bonus factor of the matching data file is calculated based on the user's interaction information for the matching data file and the following formula (5).

[0104] (5);

[0105] SL represents the addition factor; likes represents the interaction factor; the interaction factor is determined based on at least one of the number of likes, favorites, views, and comments of the user on the data file; min() represents the minimum function; log() represents the logarithmic function; otherwise represents other cases except that likes is 0.

[0106] Based on the bonus factor of the matching data file, the initial relevance of the matching data file to the original search statement and the following formula (6), the intermediate relevance of the matching data file to the original search statement is calculated.

[0107] (6);

[0108] TS represents the intermediate relevance between the matching data file and the original search statement; BS represents the initial relevance between the matching data file and the original search statement.

[0109] In data retrieval systems, it's crucial not only to ensure that the data files retrieved are what the user needs but also their timeliness. Newly released data files are more valuable and relevant than older ones. By setting a time decay factor, newly released data files are prioritized for display to users, balancing the relevance and freshness of documents. This not only improves user satisfaction but also enhances the practicality and timeliness of search results, ensuring that users have access to the most up-to-date data files.

[0110] In one implementation, the intermediate relevance between each matching data file and the original search statement is attenuated based on the release time and current time of the matching data file to obtain a target relevance between the matching data file and the original search statement. For example, the time attenuation factor for the matching data file is calculated based on the release time and current time of the matching data file according to the following formula (7).

[0111] (7);

[0112] DF represents the time decay factor of the matching data file; DaysDiff represents the number of days between the release time of the matching data file and the current time; decay represents the decay rate; and scale_days represents the decay scale.

[0113] Using the time decay factor of the matching data file, the intermediate relevance between the matching data file and the original search statement is decayed according to the following formula (8), and the target relevance between the matching data file and the original search statement is obtained.

[0114] (8);

[0115] CS represents the target relevance between the matching data file and the original search statement; TS represents the intermediate relevance between the matching data file and the original search statement; and t represents the time weight.

[0116] In descending order of target relevance, the first third number of matching data files are selected to obtain a target data file, and the target relevance between the target data file and the original search statement is obtained to obtain a target search result.

[0117] Based on the above processing, the time-decay-based adjustment method allows the target data files in the retrieval results to maintain relevance while also taking into account the freshness of the data files, which can better meet the user's demand for the latest information and further improve user experience and satisfaction.

[0118] At step S209, the target search results include the target data files and the target relevance of the target data files to the original search statement. The server determines the display order of the target data files and sends the target data files and the display order to the terminal. The terminal displays the target data files in the display order for the user to browse.

[0119] For example, the target data files are displayed in descending order of their relevance to the original search statement. Alternatively, after the multi-way recall strategy generates the target search results, the large language model is used to further integrate and optimize the target search results. This allows for a more accurate and user-satisfied search result display by deeply understanding the semantic relationship between the search statement and the data files, and taking into account the user's personalized needs and feedback.

[0120] Even if the target search results precisely match the user's search intent, different users may have different preferences for the same search query, necessitating personalized adjustments to the order of search results. Therefore, by incorporating data such as the user's historical search behavior and click patterns (i.e., interactive information), combined with a large language model to predict user preferences for different data files, the display order of target data files can be adjusted accordingly to provide a more personalized search experience.

[0121] After obtaining the target search results, the server determines the target data files in descending order of relevance to the original search query, which serves as the initial display order for the target data files. Based on the target data files' interactive information and business needs, the server adjusts the initial display order according to a pre-set sorting strategy to determine the final display order for the target data files.

[0122] Exemplarily, the preset sorting strategy may include the following:

[0123] Method 1: Business personnel set quality labels for data files when managing them. For example, data files with high user interaction and good quality are labeled "Essence," while data files with low user interaction or poor quality are labeled "Normal." If the target data file has the "Essence" label, a higher weight is assigned to the target data file. If the target data file has the "Normal" label, a lower weight is assigned to the target data file. The weight of a data file is a number greater than 1. The weight of the target data file is calculated by multiplying the target data file's relevance to the original search statement. The resulting products are sorted from high to low as the final display order.

[0124] Method 2: Business personnel set type labels for data files when managing each data file. For example, if the data file is an article introducing the process of opening a securities account, the article is labeled "Securities Account" or "Account Opening". Alternatively, if the data file is an article introducing knowledge related to stock purchases, the article is labeled "Stocks". The server determines the business type to which the original search statement entered by the user belongs. The business type of the original search statement is entered by the user when entering the original search statement. If the target data file has a label of the business type to which the original search statement belongs, a higher weight is set for the target data file. If the target data file does not have a label of the business type to which the original search statement belongs, a lower weight is set for the target data file. The weight of the data file is a numerical value greater than 1. Calculate the product of the weight of the target data file and the target relevance of the target data file to the original search statement, and sort the calculated products from high to low as the final display order.

[0125] Method 3: Obtain user interaction information for the target data file. For example, if the interaction information is the number of times a user has viewed the target data file, the initial display order of the target data files is adjusted in descending order of the number of times the user has viewed the target data files to obtain the final display order.

[0126] Method 4: Input the target data file's interactive information, business requirements, the target data file's initial display order, the user's original search statement, and prompt words into the large language model. The large language model is used to adjust the initial display order to obtain the target data file's final display order. Prompt words are set by technical personnel based on business requirements. For example, the prompt word could be "Combining the target data file's interactive information, business requirements, and the user's original search statement, adjust the target data file's initial display order to output a target data file arrangement order that meets the user's requirements."

[0127] In some embodiments, due to the limited data files stored in the database, even the retrieval results obtained through multiple recalls may not fully meet the user's needs. It is necessary to determine whether the target retrieval results meet the supplementary retrieval conditions and perform corresponding processing based on the determination results.

[0128] If the target search results do not meet the supplementary search criteria, the server directly sends the target data files and the display order to the terminal. The terminal displays the target data files on the display interface in the display order. If the target search results meet the supplementary search criteria, the server invokes a preset search engine and performs an online search using the original search statement to obtain the online search results. The server sends the target data files, the display order, and the online search results to the terminal. The terminal displays the target data files in the first display area of ​​the display interface in the display order and displays the online search results in the second display area of ​​the display interface.

[0129] The supplementary search condition is that the number of target data files is less than a second threshold. The server determines whether the number of target data files is less than the second threshold. If the number of target data files is less than the second threshold, the target search result is determined to meet the supplementary search condition. If the number of target data files is greater than the second threshold, the target search result is determined to meet the supplementary search condition.

[0130] Alternatively, the supplementary search condition is: the target data file does not contain any data files that match the original search statement. Large language models are used to generate supplementary search results (i.e., online search results). Search engine tools are used to supplement search results, ensuring that users can obtain the most comprehensive information possible.

[0131] The target data file, original search statement, and prompt word are input into the large language model. The large language model is used to determine whether there are data files in the target data file that match the original search statement. The prompt word is set by technical personnel based on business needs. For example, the prompt word is "Determine whether the target data file is related to the original search statement, and output whether there are data files in the target data file that match the original search statement." If the large language model outputs that there are no data files in the target data file that match the original search statement, the target search result is determined to meet the supplementary search conditions. If the large language model outputs that there are data files in the target data file that match the original search statement, the target search result is determined to meet the supplementary search conditions.

[0132] See also Figure 5 , Figure 5 This is a schematic diagram of a data retrieval method provided in an embodiment of the present application. This embodiment of the present application optimizes the existing data retrieval method from multiple aspects.

[0133] First, when performing data searches in existing technologies, users can only retrieve data files directly corresponding to their specific search. When processing complex or multi-dimensional search statements, users may not be able to accurately express their information needs, or their search intent may involve multiple related concepts. Existing technologies are unable to obtain data files that meet users' deeper needs.

[0134] In order to understand the user's search needs and automatically expand related search statements to improve the coverage and accuracy of the search, in an embodiment of the present application, a search is performed based on a knowledge graph and a large language model. The server obtains the query input by the user (i.e., the original search statement), and uses the structured data and the relationship between entities in the knowledge graph database, combined with the large language model to generate multiple sub-queries (sub-questions) associated with the query. The sub-questions are associated search statements. Then, keywords are extracted from the query and sub-query, and knowledge is stored. That is, the search keywords are obtained. The search keywords are used to search in the Nebula graph database (i.e., the knowledge image database) to obtain the first search result.

[0135] This approach not only connects seemingly unrelated information points but also ensures that the associated keywords used in generalized searches are based on deep, accurate relationships between entities, rather than relying solely on linguistic co-occurrence or semantic similarity. By introducing a knowledge graph database, this provides clear guidance for query generalization in large language models, enhancing the effectiveness of generalization problem formulation and ensuring that each generalization operation leads to valuable information mining, thereby enriching search results.

[0136] Secondly, while existing Elastic Search engines excel at processing large amounts of data and delivering fast search results, they still have room for improvement in handling complex queries, understanding user intent, and providing precise matches. Users expect a more relevant, comprehensive, and easy-to-understand search experience, especially when searching for specialized terms or across multiple search dimensions.

[0137] In order to meet these needs, the retrieval of the Elastic Search engine is optimized in the embodiment of the present application to improve the quality of search results and user satisfaction through a more intelligent search mechanism and ranking algorithm. The server uses Query and Sub-query, as well as search keywords to search in the Elastic Search engine database to obtain the results of the Elastic Search search answer (i.e., the second search result). The Elastic Search engine database is created based on the question and answer files, experience files, and product files in the knowledge center. The knowledge center is a platform for users to publish data files. In the above processing method, phrase matching (i.e., using Query and Sub-query for retrieval) and multi-field query (i.e., using search keywords for retrieval) are combined to give priority to the matching of complete search statements, and the user experience is improved through the automatic completion function. These optimization measures work together to provide more accurate, efficient and user-friendly search services.

[0138] Third, in the prior art, users typically use colloquial search statements to initiate searches, expecting the system to accurately understand and quickly return relevant results. Existing keyword matching methods face challenges when processing such unstructured, colloquial queries, as colloquial search statements lack an understanding of the deeper meaning of natural language. In addition, with the growth and diversification of data volumes, users have higher expectations for the accuracy and response speed of search systems. To address the above issues, the present application introduces a vector knowledge base framework in an embodiment to establish a vectorized index library (i.e., construct a vectorized database). Data files (i.e., question and answer files, experience files, and product files) are obtained from the knowledge center, and the text of the data files is obtained. The text of the data files is segmented and vectorized, that is, the text of the data files is vectorized using a text emdedding model to obtain text feature vectors. The cluster center of each vector group, the text feature vectors contained in the vector group, the ID of each text feature vector, and the data file to which each text feature vector belongs and the ID of the data file are stored to obtain a vectorized database. Then, the TextEmdedding Model is used to vectorize the user's query and sub-query, generating a sentence feature vector. This sentence feature vector is then used to search the Vector Database, obtaining the top K similar vectors (i.e., matching feature vectors). The matching feature vectors are then used to determine the search results (i.e., the third search result). Figure 7 The circles in the Vector Database represent different text feature vectors.

[0139] In this approach, based on retrieval from a vectorized knowledge base, a text embedding model (i.e., the BERT-Chinese model) was selected, capable of processing long sentences and capturing deep semantic information. This text embedding model effectively converts natural language into feature vectors, thus supporting complex query understanding. Furthermore, the text embedding model is used to preprocess and vectorize the user's colloquial query, ensuring that even unstructured natural language input can be effectively converted into searchable feature vectors. Using vectorized database technology, large-scale vector data, including vectors generated from natural language queries, can be efficiently stored and indexed, enabling fast similarity retrieval.

[0140] Fourthly, a multi-source search algorithm is used to obtain a series of preliminary search results (i.e., the first, second, and third search results). These results may be derived from keyword matching, vectorized search, or other relevance algorithms. Then, the multi-source recall results (i.e., the first, second, and third search results) are filtered to eliminate irrelevant or duplicate data files, generating the final Top-K search results (i.e., the target search results). Furthermore, when ranking the target search results, the highest-quality combination of recall results is selected from the multiple recall results. Specifically, personalized reranking is performed on the search results, combining user context and result traffic markers (such as number of likes and favorites) to determine the display order of the target search results. When the target search results meet the supplementary search criteria, network agent empowerment is performed, i.e., a network search is performed. The target search results and the network search results are combined to obtain the final Top N data files.

[0141] The data retrieval method provided in the embodiments of this application achieves the following results by integrating knowledge graphs, large language models, the search optimization strategy of the Elastic Search engine, and vectorized knowledge base technology:

[0142] First, enhanced query and understanding capabilities: By combining the knowledge graph and a large language model, it can deeply understand user query intent and automatically expand related queries, providing more comprehensive and accurate search results. This not only addresses users' explicitly expressed needs but also explores and satisfies their latent information needs. Second, improved search result quality and accuracy: The Elastic Search engine's search strategy optimization, combined with multi-field queries, filter mechanisms, and document relevance assessment, significantly improves the quality and accuracy of search results, especially for complex and specialized terminology queries, providing a more relevant and intuitive search experience. Third, efficient processing of colloquial queries: The introduction of a vector knowledge base effectively handles unstructured, colloquial queries. By converting natural language into vector representations, it supports deep semantic understanding of complex queries and fast similarity retrieval, improving the processing power and response speed of natural language queries. Fourth, personalized and optimized search results: By re-ranking and filtering search results, combining user context and the semantic analysis capabilities of a large language model, it can personalize and optimize search results, ensuring that the most relevant and valuable information is presented to users first.

[0143] See also Figure 6 , Figure 6A flowchart of a data retrieval method provided for an embodiment of the present application. The server obtains the second search result of the ES (i.e., Elastic Search Engine) search, the third search result of the vector library search, and the first search result of the graph index. The above search results are combined with business needs, sub-tag priority strategy, data file creation time, the relevance of the data file to the search statement, and traffic tags for multi-way recall. Traffic tags include the number of likes and collections of the data file by the user using the terminal (mobile phone or computer). Then, a sorting decision is made, and the multi-way recall is first sorted by deduplication and heterogeneous merging, that is, the duplicate data files in the first search result, the second search result, and the third search result are filtered out, and the filtered data files are merged and sorted to obtain the target data file and the initial display order of the target data file. Then, the sorting rule is adjusted, that is, the initial display order is adjusted to obtain the final display order of the target data file, and the terminal displays the results according to the display order.

[0144] Figure 6 The terminal's display interface shows the target search results. The user uses "securities account" as the original search query. The terminal displays the corresponding target search results. The target search results include multiple articles related to "securities account." For example, "Securities Account Opening Process," "What Documents Are Required for Opening a Securities Account," "What Types of Securities Accounts Are There," and "Securities Account Verification and Activation Process." When the user clicks "Online Results," the terminal also displays online search results.

[0145] Based on the data retrieval method provided in the embodiment of the present application, the problem is generalized through a large language model and a knowledge graph database, and the understanding of the deep intention of the user query is enhanced, thereby providing more accurate and relevant search results. Through the multi-way recall mechanism, information silos are broken to ensure that users can access and retrieve comprehensive knowledge information. As well as through the vectorized search algorithm and user feedback mechanism, the relevance and accuracy of the search results are improved, the time for user screening and verification is reduced, and the user experience is improved. It provides users with a more intelligent, accurate and personalized search experience, allowing users to find the information they need more easily while reducing interactions with irrelevant or duplicate results, significantly improving user satisfaction and efficiency. After multi-way recall, the large language model is used to integrate and optimize the retrieval results to improve the retrieval quality. By deeply understanding the query intention, personalizing the user experience, and dynamically optimizing the retrieval results, user satisfaction is greatly improved, and the retrieval technology is promoted to develop in a more intelligent and accurate direction.

[0146] In the embodiments of the present application, the order of the steps of searching the knowledge graph database, the basic database, and the vectorized database is not limited. For example, the knowledge graph database, the basic database, and the vectorized database are searched simultaneously, or the knowledge graph database, the basic database, and the vectorized database are searched in any order. For example, the knowledge graph database is searched first, the basic database is searched second, and the vectorized database is searched last. Alternatively, the vectorized database is searched first, the basic database is searched second, and the knowledge graph database is searched last.

[0147] See also Figure 7 , Figure 7 A structural diagram of a data retrieval device provided in an embodiment of the present application. The device is applied to a server, and the device includes: a search statement generation module 701, which is used to obtain the original search statement input by the user, and the associated search statement of the original search statement, to obtain the target search statement; wherein the keywords in the associated search statement have an associated relationship with the keywords in the original search statement; a keyword extraction module 702, which is used to extract keywords from the target search statement to obtain search keywords; a first search module 703, which is used to search in the knowledge graph database based on the search keywords to obtain a first search result including a first data file; wherein the knowledge graph database includes various types of data files and the associated relationships between the data files; a second search module 704, which is used to Based on the search keywords and the target search statement, a search is performed in the basic database to obtain a second search result including a second data file; wherein the basic database includes data files of various types; a third search module 705 is used to generate a sentence feature vector of the target search statement, and based on the sentence feature vector, a search is performed in the vectorized database to obtain a third search result including a third data file; wherein the vectorized database includes text feature vectors of texts of multiple data files; a target search result acquisition module 706 is used to determine a target search result including a target data file from the first search result, the second search result and the third search result based on the user's interactive information on each data file.

[0148] Optionally, the first search result further includes: an initial relevance between the first data file and the original search statement;

[0149] The first retrieval module 703 is specifically used to obtain data files that are associated with the search keywords from the knowledge graph database to obtain first data files; for each first data file, calculate the target parameters of each search keyword in the text of the first data file; wherein the target parameter of a keyword represents the importance of the keyword in the text; calculate the weighted sum of the target parameters of each search keyword in the text of the first data file, and obtain the initial relevance of the first data file to the original search statement.

[0150] Optionally, the second search result further includes: an initial relevance between the second data file and the original search statement;

[0151] The second search module 704 is specifically configured to generate target keywords including search keywords and synonyms of the search keywords; determine data files that meet the first search condition or the second search condition from the basic database as candidate data files; wherein the first search condition includes: the text of the data file includes each keyword in the original search statement or the associated search statement; the second search condition includes: the text of the data file contains at least one target keyword; for each candidate data file, calculate the target parameter of each target keyword in the text of the candidate data file in the text; wherein the target parameter of a keyword represents the importance of the keyword in the text; calculate the weighted sum of the target parameters of each target keyword in the text of the candidate data file as the initial relevance of the candidate data file to the original search statement; and determine, from each candidate data file, a candidate data file whose initial relevance to the original search statement is greater than a first threshold value to obtain a second data file; wherein the first threshold value is determined based on the maximum relevance of each candidate data file to the original search statement.

[0152] Optionally, the third search result further includes: an initial relevance between the third data file and the original search statement; cluster centers of each vector group in the vectorized database; each vector group includes text feature vectors of texts of multiple data files;

[0153] The third retrieval module 705 is specifically used to vectorize the target retrieval sentence based on the text vectorization model to obtain the sentence feature vector of the target retrieval sentence; for each sentence feature vector, calculate the similarity between the sentence feature vector and the cluster center of each vector group in the vectorization database as the first similarity; determine the vector groups to which the first number of cluster centers belong in descending order of the first similarity to obtain the candidate vector group of the sentence feature vector; calculate the similarity between the sentence feature vector and each text feature vector in the candidate vector group as the second similarity; determine the first second number of text feature vectors in each candidate vector group in descending order of the second similarity to obtain the matching feature vector of the sentence feature vector; determine the data file to which the matching feature vector belongs as the third data file; determine the maximum value of the second similarity between the sentence feature vector in the third data file and the matching feature vector as the initial relevance of the third data file to the original retrieval sentence.

[0154] Optionally, the third retrieval module 705 is specifically used to input the target retrieval sentence into the text vectorization model, use the word segmenter of the text vectorization model to segment the target retrieval sentence, and obtain each word unit of the target retrieval sentence; for each word unit of the target retrieval sentence, map the word unit to obtain the initial feature vector of the word unit; use the encoder of the text vectorization model to encode the initial feature vector of the word unit to obtain the word unit feature vector of the word unit; calculate the weighted sum of the word unit feature vectors of each word unit of the target retrieval sentence according to the preset weights to obtain the sentence feature vector of the target retrieval sentence.

[0155] Optionally, the device also includes: a database construction module, used to obtain the text of each data file; for each data file, dividing the text of the data file to obtain multiple text blocks of the data file; wherein there is an overlapping part between two adjacent text blocks; for each text block of the data file, processing the text block based on a text vectorization model to obtain a text feature vector of the text block; clustering the text feature vectors of each text block of each data file to obtain multiple vector groups; recording the cluster center of each vector group, the text feature vectors contained in the vector group, and the data file to which each text feature vector belongs.

[0156] Optionally, the first search result further includes the initial relevance between the first data file and the original search statement; the second search result further includes the initial relevance between the second data file and the original search statement; and the third search result further includes the initial relevance between the third data file and the original search statement.

[0157] The target search result acquisition module 706 is specifically used to filter out duplicate data files from the first data file, the second data file, and the third data file to obtain matching data files; calculate the target relevance of the matching data files to the original search statement based on the initial relevance of the matching data files to the original search statement, the user's interactive information on the matching data files, and the release time of the matching data files; and select the first third number of matching data files in descending order of target relevance as target data files to obtain the target search result.

[0158] The device also includes: a display module, which is used to, after the target retrieval result acquisition module 706 executes the target retrieval result including the target data file from the first retrieval result, the second retrieval result and the third retrieval result based on the user's interactive information on each data file, input the target relevance of the target data file to the original search statement, the interactive information of the target data file and the business requirements into the large language model to obtain the display order of the target data files; and send the target data file and the display order to the terminal so that the terminal displays the target data file according to the display order.

[0159] Optionally, the search statement generation module 701 is specifically used to obtain the original search statement input by the user as the target search statement; extract the keywords in the original search statement to obtain the original keywords; obtain keywords that have an association relationship with the original keywords from the knowledge graph database as associated keywords; wherein the knowledge graph database also includes keywords, and the association relationship between keywords; input the associated keywords and the original search statement into the large language model to obtain the associated search statement of the original search statement as the target search statement.

[0160] Based on the data retrieval device provided in the embodiment of the present application, the original search statement is expanded to obtain the associated search statement of the original search statement, and the original search statement and the associated search statement are used for retrieval, which can not only process the needs clearly expressed by the user, but also process and meet the user's potential deep needs, and understand the user's deep intentions, thereby providing more comprehensive and accurate retrieval results and improving the user experience. In addition, searching in a vectorized database effectively processes unstructured and colloquial search statements, and by converting natural language search statements into vectorized sentence feature vectors, deep semantic understanding and fast similarity retrieval of complex searches are achieved, thereby improving the processing power and response speed of natural language searches and improving the user experience. The target retrieval results are determined in combination with the user's personalized information, so as to provide the user with a data file that is more in line with the user's retrieval intention, further improving the accuracy of data retrieval and improving the user experience.

[0161] The embodiment of the present invention further provides an electronic device, such as Figure 8As shown, the system includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804. The processor 801, the communication interface 802, and the memory 803 communicate with each other via the communication bus 804. The memory 803 is used to store computer programs; the processor 801 is used to implement the steps of the data retrieval method in the above embodiment when executing the program stored in the memory 803.

[0162] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. This communication bus can be divided into an address bus, a data bus, a control bus, and so on. For ease of illustration, the figure uses only a single thick line, but this does not imply a single bus or type of bus. The communication interface is used for communication between the electronic device and other devices. The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the processor. The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

[0163] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data retrieval methods are implemented.

[0164] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer is enabled to execute any one of the data retrieval methods in the above embodiments.

[0165] In the above embodiments, all or part of the embodiments can be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state disk (SSD)).

[0166] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0167] Each embodiment in this specification is described in a related manner. Similar portions between embodiments can be referenced to each other. Each embodiment focuses on the differences between other embodiments. In particular, since the device, system, computer-readable storage medium, and computer program product embodiments are generally similar to the method embodiments, their descriptions are relatively simplified. For related portions, reference can be made to the descriptions of the method embodiments.

[0168] The above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention are included in the scope of protection of the present invention.

Claims

1. A data retrieval method, characterized in that: The method is applied to a server; the method comprises: Acquire an original search statement input by a user and a related search statement of the original search statement to obtain a target search statement; wherein the keywords in the related search statement are associated with the keywords in the original search statement; Perform keyword extraction on the target search statement to obtain search keywords; Based on the search keyword, a search is performed in a knowledge graph database to obtain a first search result including a first data file; wherein the knowledge graph database includes various types of data files and associations between the data files; Generate a target keyword containing the search keyword and a synonym of the search keyword; determine a data file that meets a first search condition or a second search condition from a basic database as a candidate data file; the first search condition includes: the text of the data file includes each keyword in the original search statement or the associated search statement; the second search condition includes: the text of the data file includes at least one target keyword; screen the candidate data files to obtain a second search result including a second data file; wherein the basic database includes various types of data files; The target retrieval sentence is vectorized based on the text vectorization model to obtain the sentence feature vector of the target retrieval sentence; for each sentence feature vector, the similarity between the sentence feature vector and the cluster center of each vector group in the vectorization database is calculated as the first similarity; according to the first similarity from high to low, the vector group to which the first number of cluster centers belong is determined to obtain the candidate vector group of the sentence feature vector; the similarity between the sentence feature vector and each text feature vector in the candidate vector group is calculated as the second similarity; according to the second similarity from high to low, the first second number of text feature vectors in each candidate vector group are determined to obtain the matching feature vector of the sentence feature vector; the data file to which the matching feature vector belongs is determined to obtain a third retrieval result including a third data file; wherein, the vectorization database includes text feature vectors of texts of multiple data files; the vectorization database includes cluster centers of each vector group; each vector group includes text feature vectors of texts of multiple data files; Duplicate data files are filtered out from the first data file, the second data file, and the third data file to obtain matching data files; for each matching data file, a bonus factor of the matching data file is calculated based on user interaction information for the matching data file; an intermediate relevance of the matching data file to the original search statement is calculated based on the bonus factor of the matching data file and the initial relevance of the matching data file to the original search statement; a time decay factor of the matching data file is calculated based on the release time of the matching data file and the current time; the intermediate relevance of the matching data file to the original search statement is decayed using the time decay factor of the matching data file to obtain a target relevance of the matching data file to the original search statement; the first third number of matching data files are selected in descending order of target relevance to obtain a target search result including the target data file; the first search result also includes the initial relevance of the first data file to the original search statement; the second search result also includes the initial relevance of the second data file to the original search statement; and the third search result also includes the initial relevance of the third data file to the original search statement.

2. The method according to claim 1, characterized in that The first search result also includes: an initial relevance between the first data file and the original search statement; The searching in the knowledge graph database based on the search keyword to obtain a first search result including the first data file includes: Acquire a data file associated with the search keyword from the knowledge graph database to obtain a first data file; For each first data file, calculating a target parameter of each search keyword in the text of the first data file; wherein the target parameter of a keyword represents the importance of the keyword in the text; The weighted sum of the target parameters of each search keyword in the text of the first data file is calculated to obtain the initial relevance between the first data file and the original search statement.

3. The method according to claim 1, characterized in that The second search result also includes: an initial relevance of the second data file to the original search statement; The screening of the candidate data files to obtain a second search result including a second data file includes: For each candidate data file, calculate the target parameter of each target keyword in the text of the candidate data file in the text; wherein the target parameter of a keyword represents the importance of the keyword in the text; Calculating a weighted sum of target parameters of each target keyword in the text of the candidate data file as an initial relevance between the candidate data file and the original search statement; From each candidate data file, a candidate data file having an initial relevance greater than a first threshold to the original search statement is determined to obtain a second data file; wherein the first threshold is determined based on the maximum relevance of each candidate data file to the original search statement.

4. The method according to claim 1, wherein The third search result also includes: an initial relevance of the third data file to the original search statement; The method further comprises: The maximum value of the second similarity between the sentence feature vector in the third data file and the matching feature vector is determined as the initial relevance between the third data file and the original search sentence.

5. The method according to claim 1, wherein The vectorization processing of the target search sentence based on the text vectorization model to obtain the sentence feature vector of the target search sentence includes: Inputting the target search sentence into a text vectorization model, and using a word segmenter of the text vectorization model to segment the target search sentence to obtain each word element of the target search sentence; For each word element of the target search sentence, mapping the word element to obtain an initial feature vector of the word element; Encode the initial feature vector of the word unit using the encoder of the text vectorization model to obtain the word unit feature vector of the word unit; According to preset weights, a weighted sum of the word-gram feature vectors of each word-gram of the target search sentence is calculated to obtain a sentence feature vector of the target search sentence.

6. The method according to claim 1, characterized in that The vectorized database is established in the following manner: Get the text of each data file; For each data file, the text of the data file is divided to obtain a plurality of text blocks of the data file; wherein there is an overlapping portion between two adjacent text blocks; For each text block of the data file, processing the text block based on the text vectorization model to obtain a text feature vector of the text block; Clustering the text feature vectors of each text block of each data file to obtain multiple vector groups; Record the cluster center of each vector group, each text feature vector contained in the vector group, and the data file to which each text feature vector belongs.

7. The method according to claim 1, characterized in that The method further comprises: The target relevance between the target data file and the original search statement, the interactive information of the target data file, and the business requirements are input into a large language model to obtain a display order of the target data file; and the target data file and the display order are sent to a terminal so that the terminal displays the target data file according to the display order.

8. The method according to claim 1, characterized in that The obtaining of the original search statement input by the user and the associated search statements of the original search statement to obtain the target search statement includes: Obtain the original search statement input by the user as the target search statement; Extracting keywords from the original search statement to obtain original keywords; Obtaining keywords that are associated with the original keywords from the knowledge graph database as associated keywords; wherein the knowledge graph database also includes keywords and the associations between the keywords; The associated keywords and the original search sentence are input into a large language model to obtain an associated search sentence of the original search sentence as a target search sentence.

9. A data retrieval device, characterized in that: The device is applied to a server; the device includes: A search statement generation module is used to obtain an original search statement input by a user and a related search statement of the original search statement to obtain a target search statement; wherein the keywords in the related search statement are associated with the keywords in the original search statement; A keyword extraction module, configured to extract keywords from the target search statement to obtain search keywords; A first search module is configured to search a knowledge graph database based on the search keyword to obtain a first search result including a first data file; wherein the knowledge graph database includes various types of data files and associations between the data files; The second search module is configured to generate a target keyword including the search keyword and a synonym of the search keyword; determine a data file that meets the first search condition or the second search condition from the basic database as a candidate data file; the first search condition includes: the text of the data file includes each keyword in the original search statement or the associated search statement; the second search condition includes: the text of the data file includes at least one target keyword; screen the candidate data files to obtain a second search result including a second data file; wherein the basic database includes data files of various types; The third retrieval module is used to vectorize the target retrieval sentence based on the text vectorization model to obtain the sentence feature vector of the target retrieval sentence; for each sentence feature vector, calculate the similarity between the sentence feature vector and the cluster center of each vector group in the vectorization database as the first similarity; determine the vector group to which the first number of cluster centers belong in the order of the first similarity from high to low, and obtain the candidate vector group of the sentence feature vector; calculate the similarity between the sentence feature vector and each text feature vector in the candidate vector group as the second similarity; determine the first second number of text feature vectors in each candidate vector group in the order of the second similarity from high to low; determine the data file to which the matching feature vector belongs, and obtain a third retrieval result including a third data file; wherein, the vectorization database includes text feature vectors of texts of multiple data files; the vectorization database includes cluster centers of each vector group; each vector group includes text feature vectors of texts of multiple data files; The target search result acquisition module is configured to filter out duplicate data files from the first data file, the second data file, and the third data file to obtain matching data files; calculate, for each matching data file, a bonus factor of the matching data file based on user interaction information with respect to the matching data file; calculate, based on the bonus factor of the matching data file and the initial relevance of the matching data file to the original search statement, an intermediate relevance of the matching data file to the original search statement; calculate, based on the release time and current time of the matching data file, a time decay factor of the matching data file; use the time decay factor of the matching data file to decay the intermediate relevance of the matching data file to the original search statement to obtain a target relevance of the matching data file to the original search statement; select the first third number of matching data files in descending order of target relevance to obtain a target search result containing the target data file; the first search result also includes the initial relevance of the first data file to the original search statement; the second search result also includes the initial relevance of the second data file to the original search statement; and the third search result also includes the initial relevance of the third data file to the original search statement.

10. A data retrieval system, characterized in that: The data retrieval system includes: a terminal and a server; The terminal is configured to obtain an original search statement input by a user and send the original search statement to the server; The server is configured to, after receiving the original search statement, obtain a target search result of the original search statement and send the target search result to the terminal; wherein the target search result includes a target data file; and the target search result is determined according to the data search method according to any one of claims 1 to 8; The terminal is further configured to display the target data file on a display interface after receiving the target search result.

Images