Text retrieval method, device and system, storage medium and product

Abstract

The invention discloses a text retrieval method, device and system, a storage medium and a product, and relates to the technical field of text retrieval, the method comprises the following steps: in response to a text retrieval instruction, obtaining a current user question input by a user; and performing vector matrix conversion on the current user question through a pre-created text model to obtain a vector matrix of the current user question, and matching the vector matrix with a vector matrix in a pre-constructed ES knowledge base to obtain a knowledge answer. According to the method, knowledge base scenes and business requirements are combined, knowledge is converted into the vector matrix to be stored in the ES knowledge base, user questions are converted into vectors, massive text knowledge can be converted into spatial vector representation, knowledge information dimensions included in the vectorized feature matrix are more complete and specific, retrieval accuracy is improved, and user experience is improved. Meanwhile, due to the adoption of an off-line vectorization storage scheme based on ES storage, the waiting time of real-time feature extraction and repeated resource calculation in retrieval are reduced, and retrieval is more efficient and more accurate.

Description

Technical Field

[0001] The present application relates to the field of text retrieval technology, and in particular to a text retrieval method, device, system, storage medium and product. Background Art

[0002] With the rapid advancement of deep learning and natural language processing technology, big models have developed rapidly, such as GPT, Tongyi, GLM, etc. It is an inevitable trend for application fields to use big models for AI intelligence. However, due to the limitation of token length, big model applications cannot input massive data for big models to understand. Therefore, how to provide accurate knowledge for big models to refer to from massive data based on user input is a key issue that needs to be solved in the development of big models.

[0003] However, for efficient retrieval of massive text data, the retrieval scripts provided by traditional databases and storage devices such as ES, whether relying on the inaccuracy of fuzzy matching or the limitations of precise retrieval on keywords, are difficult to meet the needs of high-precision and comprehensive retrieval. Especially when processing long texts, the neglect of subject information and the low matching hit rate highlight the limitations of existing retrieval technology. Therefore, it is necessary to propose a solution to improve text retrieval accuracy and retrieval speed. Summary of the invention

[0004] The main purpose of this application is to provide a text retrieval method, device, system, storage medium and product, aiming to improve text retrieval accuracy and retrieval speed.

[0005] To achieve the above purpose, the present application proposes a text retrieval method, which comprises:

[0006] In response to the text search instruction, obtaining the current user question input by the user;

[0007] Performing vector matrix conversion on the current user question through a pre-created text model to obtain a current user question vector matrix;

[0008] The current user question vector matrix is ​​matched with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer.

[0009] In one embodiment, the step of matching the current user question vector matrix with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer includes:

[0010] Calculate the similarity distance between the current user question vector matrix and the vector matrix in the knowledge base;

[0011] Score the similarity distance to obtain the similarity score retrieval result;

[0012] The vector matrix in the knowledge base is filtered and screened according to the similarity score retrieval result, and the knowledge is finally recalled to obtain the knowledge answer with similarity hit.

[0013] In one embodiment, the method further comprises:

[0014] In the large-model domain question-answering scenario, the large-model domain question-answering recall engine is called to retrieve text knowledge vectors, analyze the recalled knowledge hit by similarity matching, format the recalled knowledge text, and output the processed knowledge answer;

[0015] In the QA knowledge domain question-answering scenario, the general QA knowledge domain question-answering recall engine is called to retrieve QA knowledge vectors and recall QA answers with similarity hits.

[0016] In one embodiment, before the step of matching the current user question vector matrix with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer, the step further includes:

[0017] Determine the application scenarios of current knowledge storage;

[0018] If the application scenario of the current knowledge storage is a large-model domain question-answering scenario based on document knowledge, the ES knowledge base is bound to the large-model domain question-answering recall engine for retrieval;

[0019] Preprocessing the pre-collected text knowledge to obtain preprocessed text knowledge;

[0020] According to the business scenario, an adaptive text slicing rule template is selected to perform a slicing operation on the content of the preprocessed text knowledge to obtain slice content;

[0021] The text2Vec feature extraction model is used to extract features from the slice content, and the extracted text features are converted into a text vector matrix, which is then stored in the ES knowledge base.

[0022] In one embodiment, the text model is a text2Vec feature extraction model, the slice content is a knowledge fragment of a preset length, and the slice content contains complete text paragraph information and meets the token restrictions of the large model.

[0023] In one embodiment, the step of preprocessing the pre-collected text knowledge to obtain the preprocessed text knowledge includes:

[0024] Clean the pre-collected text knowledge and remove invalid tags and format contents in the text knowledge content;

[0025] The privacy content of the text information of the entire text knowledge is detected, and the relevant privacy information and / or non-compliant content are removed to obtain the preprocessed text knowledge.

[0026] In one embodiment, the step of selecting an adaptive text slicing rule template to perform a slicing operation on the content of the preprocessed text knowledge according to the business scenario, and obtaining the sliced ​​content also includes:

[0027] The content of text knowledge is classified according to different business scenarios.

[0028] In one embodiment, before the step of determining the application scenario of the current knowledge storage, the step further includes:

[0029] If the application scenario of the current knowledge storage is based on the QA knowledge domain question and answer scenario, the ES knowledge base is bound to the general QA knowledge domain question and answer recall engine for retrieval;

[0030] Get pre-collected questions and answers;

[0031] The questions and answers are converted into vector matrices and stored in the ES knowledge base.

[0032] This embodiment also provides a text search device, the device comprising:

[0033] An acquisition module, used for acquiring a current user question input by a user in response to a text search instruction;

[0034] A vector conversion module, used for performing vector matrix conversion on the current user question through a pre-created text model to obtain a vector matrix of the current user question;

[0035] The retrieval matching module is used to match the vector matrix of the current user question with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer.

[0036] This embodiment further provides a text retrieval system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the text retrieval method described above.

[0037] This embodiment further provides a storage medium, which is a computer-readable storage medium. A computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the text retrieval method described above are implemented.

[0038] This embodiment further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the text retrieval method described above are implemented.

[0039] One or more technical solutions proposed in this application have at least the following technical effects: by responding to the text search instruction, the current user question input by the user is obtained; the current user question is converted into a vector matrix through a pre-created text model to obtain the current user question vector matrix; the current user question vector matrix is ​​matched with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer. This application combines the knowledge base scenario and business needs, takes ES as the knowledge base, converts the knowledge into a vector matrix and stores it in the ES knowledge base, and converts the user question into a vector, and performs similarity matching and recall in the knowledge base. Due to the use of vector conversion, massive text knowledge can be converted into a spatial vector representation. The vectorized feature matrix can include both the complete text subject content and keywords, so that the knowledge information dimension is more complete and specific, making the entire retrieval more accurate than the fuzzy matching and exact matching of ES. At the same time, due to the use of an offline vectorized storage solution based on ES storage, the waiting time for real-time feature extraction in retrieval is reduced, repeated resource calculations are reduced, and the delay of real-time extraction is reduced, making the retrieval more efficient and more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 A flowchart diagram of the first embodiment of the text retrieval method of the present application;

[0043] Figure 2 A flowchart diagram of the second embodiment of the text retrieval method of the present application;

[0044] Figure 3 A complete flowchart of the text search method provided in the embodiment of the present application;

[0045] Figure 4 A flowchart of text parsing, preprocessing and slicing in the text retrieval method provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the search process of the search engine in different application scenarios in the text search method provided in the embodiment of the present application;

[0047] Figure 6This is a schematic diagram of the module structure of the text retrieval device in the embodiment of the present application;

[0048] Figure 7 Schematic diagram of the system structure of the hardware operating environment involved in the text retrieval method in the embodiment of the present application.

[0049] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0050] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0051] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0052] The main solution of the embodiment of the present application is: by responding to the text search instruction, the current user question input by the user is obtained; the current user question is converted into a vector matrix through a pre-created text model to obtain the current user question vector matrix; the current user question vector matrix is ​​matched with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer. This application combines the knowledge base scenario and business needs, takes ES as the knowledge base, converts the knowledge into a vector matrix and stores it in the ES knowledge base, and converts the user question into a vector, and performs similarity matching and recall in the knowledge base. Due to the use of vector conversion, a large amount of text knowledge can be converted into a spatial vector representation. In the vectorization process, the model can accurately capture the core features of the text information. The vectorized feature matrix can include both the complete text subject content and keywords, so that the knowledge information dimension is more complete and specific. In the similarity search, high-precision matching hits can be achieved, making the entire search more accurate than the fuzzy matching and exact matching of ES. At the same time, due to the use of an offline vectorized storage solution based on ES storage, the waiting time for real-time feature extraction in the search is reduced, repeated resource calculations are reduced, and the delay of real-time extraction is reduced, making the search more efficient and more accurate.

[0053] In the existing technology, in the large model application scenario, due to the limitation of token length, it is impossible to input the entire domain knowledge into the large model for understanding, and only knowledge of appropriate length can be given to the large model for analysis. However, the large model cannot directly and quickly retrieve the knowledge content most relevant to the user's question from the massive knowledge content. The traditional technical solution splits the text knowledge, but cannot accurately match the knowledge related to the current user question to the large model, so the matching accuracy is low and the matching cost is high. In addition, in the QA knowledge field question and answer scenario, quickly retrieving the QA knowledge most relevant to the user's question from the knowledge base is the most basic and critical function of the search engine. However, at present, whether it is ES's fuzzy matching or exact matching, or keyword search, it is impossible to accurately match the hit.

[0054] Therefore, the embodiments of the present application provide a solution. In the large model application scenario, the present application combines the large model token input length limit, slices the domain knowledge, extracts the sliced ​​knowledge as feature vectors and stores them in ES. When the user asks a question, the user question is converted into a vector matrix for vector similarity matching and recall, and the knowledge most relevant to the user question is found and input into the large model for analysis and understanding; in the QA knowledge domain question and answer scenario, the present application combines the knowledge base scenario and business needs, converts questions and answers into vector matrices respectively, and converts user questions into vectors, and performs similarity matching and recall in the knowledge base, thereby improving retrieval accuracy and retrieval speed.

[0055] In addition, this embodiment takes into account that: At present, feature extraction models such as Bert can convert unstructured data such as text, images, and audio into high-dimensional vectors, opening up a new data foundation and application scenario for vector similarity retrieval. In this context, vector similarity retrieval has become a research hotspot in the field of text retrieval because of its ability to quickly locate the most similar vectors in high-dimensional space. However, when the algorithm processes large-scale data sets, its retrieval efficiency and accuracy decrease, especially the "dimensionality disaster" problem faced in high-dimensional space, making similarity calculation extremely complex and time-consuming. In addition, the high cost of index construction and updating, the challenge of updating the index in real time for dynamically changing data sets, and the difficulty of interpreting retrieval results all reveal the limitations of the nearest neighbor algorithm in practical applications. More importantly, the effect of vector-based similarity retrieval depends to a large extent on the feature extraction capability of the BERT model. Only in the vectorization process, the model can accurately capture the core features of text information, and can high-precision matching hits be achieved in similarity retrieval.

[0056] Therefore, this application combines deep learning and natural language processing technology to design and implement a vector retrieval processing device based on text feature matrix. The device uses ES as a storage medium for massive data. Before the data is stored in the warehouse, the data will first be sliced ​​to divide the text knowledge into knowledge fragments of appropriate length; secondly, the segmented text data will be feature extracted using the Bert model (text2Vec feature extraction model), converted into a text vector matrix, and stored in the ES knowledge base; then, when searching, the user input question is vectorized using the same Bert model, and the vectorized user question is fully matched in the ES vector library, and finally the knowledge answer of the text similarity hit is returned.

[0057] The present application embodiment provides a text retrieval method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the text retrieval method of the present application.

[0058] In this embodiment, the text search method includes steps S10 to S30, each of which is explained in detail below.

[0059] like Figure 1 As shown, the first embodiment of the present application proposes a text retrieval method, the method comprising:

[0060] Step S10, in response to the text search instruction, obtaining the current user question input by the user;

[0061] The technical solution of this embodiment can improve the accuracy and speed of text retrieval.

[0062] The execution subject of this embodiment may be a text search device, which may be carried on a terminal or a server in the form of hardware or software.

[0063] The text retrieval device of this embodiment is a vector retrieval processing device based on a text feature matrix, and the device uses ES as a storage medium for massive data.

[0064] As an implementation method, the text search device can be configured with a search engine or mounted with a search engine, and the search engine can be selected according to different application scenarios, such as a large model domain question and answer recall engine in a large model application scenario, or a common QA knowledge domain question and answer recall engine in a QA knowledge domain question and answer scenario. This embodiment can call the corresponding search engine for text search according to different application scenarios, thereby improving the search accuracy and search speed of the search engine.

[0065] The text data in this embodiment can be applied in the financial field, but is not limited to the financial field.

[0066] As an implementation mode, after receiving a search instruction from a user, the text search device may call a corresponding search engine to perform a search, and obtain the current user question input by the user through the search engine in response to the user's text search instruction.

[0067] Step S20, performing vector matrix conversion on the current user question using a pre-created text model to obtain a vector matrix of the current user question;

[0068] In this embodiment, a text model is pre-created, and the text model is used for feature extraction and vector conversion. As an implementation method, the text model can be a Bert model or a text2Vec feature extraction model. In this embodiment, the text2Vec feature extraction model is used as an example. The text2Vec feature extraction model is used to perform vector matrix conversion on the current user question to obtain the current user question vector matrix.

[0069] As mentioned above, the current feature extraction models such as BERT can convert unstructured data such as text, images, and audio into high-dimensional vectors, opening up a new data foundation and application scenario for vector similarity retrieval. In this context, vector similarity retrieval has become a research hotspot in the field of text retrieval due to its ability to quickly locate the most similar vectors in high-dimensional space. However, when the algorithm processes large-scale data sets, its retrieval efficiency and accuracy decrease, especially the "dimensionality disaster" problem faced in high-dimensional space, making similarity calculation extremely complex and time-consuming. In addition, the high cost of index construction and updating, the challenge of real-time index updating for dynamically changing data sets, and the difficulty of interpreting retrieval results all reveal the limitations of the nearest neighbor algorithm in practical applications. More importantly, the effect of vector-based similarity retrieval depends largely on the feature extraction capability of the BERT model. Only when the model can accurately capture the core features of text information during the vectorization process can high-precision matching hits be achieved in similarity retrieval.

[0070] Therefore, this embodiment converts user questions into vectors through the text2Vec feature extraction model, which can convert massive amounts of text knowledge into spatial vector representation. During the vectorization process, the model can accurately capture the core features of text information. The feature matrix after vectorization can include both the complete text subject content and keywords, so that the knowledge information dimension is more complete and specific. In similarity retrieval, high-precision matching hits can be achieved, making the entire retrieval more accurate than ES's fuzzy matching and exact matching.

[0071] Step S30, matching the current user question vector matrix with the vector matrix in the pre-built ES knowledge base to obtain a matched knowledge answer.

[0072] This embodiment pre-builds an ES knowledge base, which stores knowledge answers to questions asked by different users, and the knowledge answers are stored in the ES knowledge base in the form of vector matrices. The vector retrieval processing device based on the text feature matrix uses ES as a storage medium for massive data. By adopting this offline vectorized storage solution based on ES storage, the waiting time for real-time feature extraction in retrieval can be reduced, and repeated resource calculations can be reduced, thereby reducing the delay of real-time extraction and making retrieval more efficient and accurate.

[0073] As an implementation method, the step of matching the current user question vector matrix with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer may include:

[0074] Step A, calculating the similarity distance between the vector matrix of the current user question and the vector matrix in the knowledge base;

[0075] Step B, scoring the similarity distance to obtain a similarity score retrieval result;

[0076] Step C, filtering and screening the vector matrix in the knowledge base according to the similarity score retrieval result, and finally recalling the knowledge to obtain the knowledge answer with similarity hit.

[0077] Specifically, in this embodiment, when similarity matching is performed, matching is achieved by calculating the similarity distance between vector matrices and vector matrices. That is, the user question is vectorized, and at the same time, the knowledge answer is stored in the ES knowledge base in the form of a vector matrix. When similarity matching is performed specifically, the similarity distance between the current user question vector matrix and the vector matrix in the knowledge base is calculated, and the similarity distance is scored to obtain a similarity score retrieval result; according to the similarity score retrieval result, the vector matrix in the knowledge base is filtered and screened, that is, one or more vector matrices with the highest similarity score are screened from the ES knowledge base to find the corresponding knowledge answer, or the knowledge answer that is most relevant or similar to the user question is found and recalled to obtain a knowledge answer with similarity hit.

[0078] Among them, the search engine configured by the text search device can be selected according to different application scenarios, for example, it can be a large model field question and answer recall engine in a large model application scenario, or it can be a common QA knowledge field question and answer recall engine in a QA knowledge field question and answer scenario. This embodiment scheme can call the corresponding search engine for text search according to different application scenarios to meet the different search needs of users and improve the search accuracy and search speed of the search engine.

[0079] When knowledge is stored in the database, the storage method will be different in different application scenarios. There are two main scenarios: large-model domain question and answer based on document knowledge and domain question and answer based on QA knowledge. The difference between these two scenarios is that document knowledge needs to be sliced, while in the domain question and answer scenario of QA knowledge, the questions and answers are already maintained. Therefore, when performing vector similarity retrieval based on the text feature matrix, the retrieval engine type is different.

[0080] Among them, as an implementation method, in the large model domain question and answer scenario, the text retrieval device calls the large model domain question and answer recall engine to perform text knowledge vector retrieval, and in the QA knowledge domain question and answer scenario, the text retrieval device calls the ordinary QA knowledge domain question and answer recall engine to perform QA knowledge vector retrieval.

[0081] Specifically, in the large-model domain question-answering scenario, the text retrieval device calls the large-model domain question-answering recall engine to perform text knowledge vector retrieval. First, the user question is converted into a spatial vector matrix representation, and then the similarity distance between the vector matrix of the current user question and the vector matrix of the knowledge base is calculated in the ES knowledge base. The retrieval results are filtered and screened according to the similarity score, and finally the knowledge is recalled.

[0082] Furthermore, in the large-model domain question-answering scenario, the text retrieval device calls the large-model domain question-answering recall engine to perform text knowledge vector retrieval, analyzes the recalled knowledge hit by similarity matching, formats the recalled knowledge text, and outputs the processed knowledge answer.

[0083] Among them, considering the application scenario of the large model, due to the limitation of token length, the entire domain knowledge cannot be input into the large model for understanding, and only knowledge of appropriate length can be given to the large model for analysis. However, the large model cannot directly and quickly retrieve the knowledge content most relevant to the user's question from the massive knowledge content. The traditional technical solution splits the text knowledge, but cannot accurately match the knowledge related to the current user question to the large model. The matching accuracy is low and the matching cost is high. In this embodiment, the domain knowledge is sliced ​​in combination with the input length limit of the token of the large model, and the sliced ​​knowledge is extracted as a feature vector and stored in ES. When the user asks a question, the user question is converted into a vector matrix for vector similarity matching and recall, and the knowledge most relevant to the user question is found and input into the large model for analysis and understanding.

[0084] In the QA knowledge domain question-answering scenario, the text retrieval device calls the general QA knowledge domain question-answering recall engine to perform QA knowledge vector retrieval and recall QA answers with similarity hits.

[0085] Among them, considering the question-answering scenario in the QA knowledge field, quickly retrieving the most relevant QA knowledge from the knowledge base is the most basic and critical function of the search engine. However, currently, whether it is ES's fuzzy matching or exact matching, or keyword search, it is impossible to accurately match the hit. This embodiment combines the knowledge base scenario and business needs, converts questions and answers into vector matrices respectively, and converts user questions into vectors, performs similarity matching and recall in the knowledge base, and improves retrieval accuracy.

[0086] This embodiment uses the above scheme, specifically by responding to the text search instruction, to obtain the current user question input by the user; converting the current user question into a vector matrix through a pre-created text model to obtain the current user question vector matrix; matching the current user question vector matrix with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer. This application combines the knowledge base scenario and business needs, uses ES as the knowledge base, converts the knowledge into a vector matrix and stores it in the ES knowledge base, and converts the user question into a vector, and performs similarity matching and recall in the knowledge base. Due to the use of vector conversion, massive text knowledge can be converted into a spatial vector representation. The vectorized feature matrix can include both the complete text subject content and keywords, so that the knowledge information dimension is more complete and specific, making the entire retrieval more accurate than the fuzzy matching and exact matching of ES. At the same time, due to the use of an offline vectorized storage solution based on ES storage, the waiting time for real-time feature extraction in retrieval is reduced, repeated resource calculations are reduced, and the delay of real-time extraction is reduced, making the retrieval more efficient and more accurate.

[0087] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the first embodiment can refer to the above introduction, and will not be repeated in the following. Figure 2 In step S30, the current user question vector matrix is ​​matched with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer, and the step also includes step S00:

[0088] Step S00, constructing an ES knowledge base.

[0089] Specifically, as an implementation method, the step S00 of building an ES knowledge base includes:

[0090] Step S01, determining the application scenario of the current knowledge storage;

[0091] Step S02: If the application scenario of the current knowledge storage is a large-model domain question-answering scenario based on document knowledge, the ES knowledge base is bound to the large-model domain question-answering recall engine for retrieval;

[0092] Step S03, preprocessing the pre-collected text knowledge to obtain preprocessed text knowledge;

[0093] Step S04, according to the business scenario, select an adaptive text slicing rule template to perform a slicing operation on the content of the preprocessed text knowledge to obtain slice content;

[0094] Step S05, using the text2Vec feature extraction model to perform feature extraction on the slice content, and converting the extracted text features into a text vector matrix, and storing the text vector matrix in the ES knowledge base.

[0095] Step S06: If the application scenario of the current knowledge storage is a question-answering scenario based on the QA knowledge domain, the ES knowledge base is bound to a common QA knowledge domain question-answering recall engine for retrieval;

[0096] Step S07, obtaining pre-collected questions and answers;

[0097] Step S08, convert the questions and answers into vector matrices and store them in the ES knowledge base.

[0098] In this embodiment, in the large model domain question answering scenario, the construction of the ES knowledge base involves slicing of text knowledge and vectorization of text features. In the QA knowledge domain question answering scenario, the construction of the ES knowledge base involves vectorization of questions and answers.

[0099] Among them, in the large-model domain question-answering scenario, the first step includes the preprocessing stage of text knowledge.

[0100] In the preprocessing stage of building the knowledge base, the text knowledge will first be cleaned to remove invalid tags and format content in the text content, and unnecessary symbols in the text will be reduced as much as possible while ensuring the basic format of the text. Secondly, the entire text information will be tested for privacy content and relevant privacy information will be removed. Then the relevant knowledge in the article will be checked to see if it contains non-compliant content. If the text content is found to be non-compliant, the corresponding text knowledge will not be able to be connected to the knowledge base.

[0101] After the text knowledge preprocessing is completed, the adaptive text slicing rule template will be selected according to the specific business scenario to slice the original content of the text knowledge. The most critical content in slicing is: the slicing fragments should reach the required maximum value as much as possible, while ensuring the semantic integrity of each fragment. That is, the slice content is a knowledge fragment of a preset length, and the slice content contains complete text paragraph information and meets the token restrictions of the large model. Only in the relative slice content, which contains both complete text paragraph information and meets the token restrictions of the large model, can the large model better serve the questions raised by users.

[0102] For example, in the banking business scenario, business knowledge has universal structural characteristics. Therefore, based on the characteristics of different text knowledge, slicing rules are summarized and formulated to form an adaptive text slicing rule template.

[0103] In addition, when slicing, the text knowledge content needs to be classified according to different business scenarios. The categories of text knowledge content include plain text, tables, and image information. Among them, the big model cannot directly understand the table information. For this reason, the table content needs to be converted into a markdown format that the big model can understand. Each sliced ​​paragraph after classification can be directly given to the big model as reference knowledge, ultimately serving the user's domain questions and answers.

[0104] This embodiment uses the above scheme to build an ES knowledge base, and when receiving a text search instruction, obtains the current user question input by the user; converts the current user question into a vector matrix through a pre-created text model to obtain a current user question vector matrix; matches the current user question vector matrix with the vector matrix in the constructed ES knowledge base to obtain a matched knowledge answer. This application combines the knowledge base scenario and business needs, uses ES as a knowledge base, converts knowledge into a vector matrix and stores it in the ES knowledge base, and converts user questions into vectors, and performs similarity matching and recall in the knowledge base. Due to the use of vector conversion, massive text knowledge can be converted into spatial vector representation. The vectorized feature matrix can include both the complete text subject content and keywords, so that the knowledge information dimension is more complete and specific, making the entire retrieval more accurate than the fuzzy matching and exact matching of ES. At the same time, due to the use of an offline vectorized storage solution based on ES storage, the waiting time for real-time feature extraction in retrieval is reduced, repeated resource calculations are reduced, and the delay of real-time extraction is reduced, making the retrieval more efficient and more accurate.

[0105] The following combination Figure 3 , Figure 4 and Figure 5 The present embodiment scheme is described in detail:

[0106] This embodiment combines deep learning and NLP technology to build a vector retrieval processing device based on text feature matrix. First, the text2Vec feature extraction model is used to classify domain knowledge and text knowledge according to different business scenarios, and slice them using adaptive length slicing rule templates. Then, the sliced ​​knowledge and QA knowledge are converted into vector matrices represented in space and stored offline in the ES knowledge base.

[0107] The offline construction process of the knowledge base resources is completed through the above steps, and knowledge retrieval is performed based on the existing knowledge base. In the large model scenario, the device first converts the user question into a spatial matrix representation, then calculates the similarity distance between the vector matrix of the current user question and the vector matrix of the knowledge base in the ES knowledge base, and filters and selects the retrieval results based on the similarity score, and finally recalls the knowledge. Figure 3 As shown, the overall process of vector search based on the text feature matrix in this embodiment is as follows:

[0108] First, build a knowledge base and store knowledge, that is, connect external knowledge to the ES knowledge base, which includes the preprocessing stage of text knowledge. In the preprocessing stage of building a knowledge base, the text knowledge will first be cleaned and parsed to remove invalid tags and format content in the text content, and unnecessary symbols in the text will be reduced as much as possible while ensuring the basic format of the text. Secondly, the entire text information will be tested for privacy content and relevant privacy information will be removed. Then check whether the relevant knowledge in the article contains non-compliant content. If the text content is found to be non-compliant, the corresponding text knowledge will not be able to be connected to the knowledge base.

[0109] Secondly, after the text knowledge preprocessing is completed, the adaptive text slicing rule template will be selected to slice the original content according to the specific business scenario. The most critical content when slicing is: the segmentation fragments should reach the required maximum value as much as possible, while ensuring the semantic integrity of each fragment. Only when the relative slice content contains both complete text paragraph information and meets the token restrictions of the big model can the big model better serve the questions raised by users. For example, in the banking business scenario, business knowledge has universal structural characteristics. To this end, based on different text knowledge features, slicing rules are summarized and formulated to form corresponding adaptive slicing templates. At the same time, the text content needs to be classified when slicing. The text content includes plain text, tables and picture information. Among them, the table information cannot be directly understood by the big model. For this reason, the table content needs to be converted into a markdown format that the big model can understand. Each slice paragraph after classification can be directly given to the big model as reference knowledge, and ultimately serve the user's domain questions and answers. Among them, the process of text parsing preprocessing and slicing can refer to Figure 4 shown.

[0110] Then, after paragraph segmentation, the text is vectorized using a feature extraction model, mapped to a high-dimensional space, and then stored in the corresponding ES knowledge base. The similarity between knowledge is then measured through spatial distance, so that knowledge content similar to the user's question can be recalled.

[0111] When knowledge is stored in the database, the storage method will be different in different application scenarios. There are two main application scenarios: large-scale domain question-answering scenarios based on document knowledge and domain question-answering scenarios based on QA knowledge. The difference between these two scenarios is that document knowledge needs to be sliced, while the domain question-answering of QA knowledge is the questions and answers that have been maintained. Therefore, when performing vector similarity retrieval based on text feature matrix, the retrieval engine type is also different. Figure 5 As shown, the specific process is as follows:

[0112] In the large model domain question and answer scenario, the text retrieval device calls the large model domain question and answer recall engine to perform text knowledge vector retrieval. In the QA knowledge domain question and answer scenario, the text retrieval device calls the ordinary QA knowledge domain question and answer recall engine to perform QA knowledge vector retrieval.

[0113] Specifically, in the large-model domain question-answering scenario, the text retrieval device calls the large-model domain question-answering recall engine to perform text knowledge vector retrieval. First, the user question is converted into a spatial vector matrix representation, and then the similarity distance between the vector matrix of the current user question and the vector matrix of the knowledge base is calculated in the ES knowledge base. The retrieval results are filtered and screened according to the similarity score, and finally the knowledge is recalled.

[0114] Furthermore, in the large-model domain question-answering scenario, the text retrieval device calls the large-model domain question-answering recall engine to perform text knowledge vector retrieval, analyzes the recalled knowledge hit by similarity matching, formats the recalled knowledge text, and outputs the processed knowledge answer.

[0115] Among them, considering the application scenario of the large model, due to the limitation of token length, the entire domain knowledge cannot be input into the large model for understanding, and only knowledge of appropriate length can be given to the large model for analysis. However, the large model cannot directly and quickly retrieve the knowledge content most relevant to the user's question from the massive knowledge content. The traditional technical solution splits the text knowledge, but cannot accurately match the knowledge related to the current user question to the large model. The matching accuracy is low and the matching cost is high. In this embodiment, the domain knowledge is sliced ​​in combination with the input length limit of the token of the large model, and the sliced ​​knowledge is extracted as a feature vector and stored in ES. When the user asks a question, the user question is converted into a vector matrix for vector similarity matching and recall, and the knowledge most relevant to the user question is found and input into the large model for analysis and understanding.

[0116] In the QA knowledge domain question-answering scenario, the text retrieval device calls the general QA knowledge domain question-answering recall engine to perform QA knowledge vector retrieval and recall QA answers with similarity hits.

[0117] Among them, considering the question-answering scenario in the QA knowledge field, quickly retrieving the most relevant QA knowledge from the knowledge base is the most basic and critical function of the search engine. However, currently, whether it is ES's fuzzy matching or exact matching, or keyword search, it is impossible to accurately match the hit. This embodiment combines the knowledge base scenario and business needs, converts questions and answers into vector matrices respectively, and converts user questions into vectors, performs similarity matching and recall in the knowledge base, and improves retrieval accuracy.

[0118] It should be noted that in the text retrieval device, the two most critical steps are the offline feature extraction of text knowledge and the real-time vector matrix conversion of user questions. In both processes, the text2vec feature extraction model is used to vectorize the text. The accuracy of the entire recall result is closely related to the feature extraction model. Only when the feature extraction model can better extract the complete information in the text can the vector search and recall process be more accurate.

[0119] Among them, since the feature vectorization process involves a large number of matrix operations, if the feature extraction of text knowledge is performed in real time, the waiting time will be longer during the user interaction process, and the retrieval efficiency will be low, resulting in poor user experience. To this end, this text retrieval device uses offline feature extraction of text knowledge. When searching, it only needs to vectorize the user question to directly calculate the similarity with the knowledge base knowledge, thereby improving the response speed of the engine recall and achieving more efficient and accurate recall.

[0120] This device uses ES as a storage platform for vector data. ES meets the native support for vector fields and provides effective management and retrieval functions for vector data. The storage structure design of ES is shown in Table 1 below:

[0121] Table 1-ES storage field design

[0122]

[0123]

[0124] This embodiment combines deep learning and natural language processing technology to design and implement a vector retrieval processing device (i.e., text retrieval device) based on a text feature matrix. The device uses ES as a storage medium for massive data. Before the data is stored in the database, the data will first be sliced ​​to divide the text knowledge into knowledge fragments of appropriate lengths; secondly, the segmented text data will be feature extracted using the text2Vec feature extraction model, converted into a text vector matrix, and stored in the ES knowledge base; then, when searching, the user-entered questions will be vectorized using the same text2Vec feature extraction model, and the vectorized user questions will be fully matched in the ES vector library, and finally the knowledge answers that match the text similarity will be returned.

[0125] Compared with the prior art, the scheme of this embodiment realizes the feature extraction vectorization method based on text2Vec, converts massive text knowledge into spatial vector representation, and the offline vectorized storage solution based on ES storage reduces the waiting time for real-time feature extraction in retrieval and improves the retrieval speed. The adaptive slicing scheme based on business scenarios and rule structures and the fully automatic slicing process provide more accurate context information for knowledge dialogues. The vector retrieval processing device based on the text feature matrix adopted in this embodiment combines the adaptive text slicing scheme with the feature vectorization method to retrieve and recall knowledge. The vectorized feature matrix includes both the complete text subject content and keywords, and the knowledge information dimension is more complete and specific. The entire retrieval is more accurate than the fuzzy matching and exact matching of ES. At the same time, due to the use of offline feature vector matrix extraction, repeated resource calculations are reduced, the delay of real-time extraction is reduced, and the retrieval is more efficient and accurate.

[0126] In addition, to achieve the above purpose, the present application also proposes a text retrieval device, such as Figure 6 As shown, the device comprises:

[0127] The acquisition module 10 is used to obtain the current user question input by the user in response to the text search instruction;

[0128] A vector conversion module 20, configured to perform vector matrix conversion on the current user question using a pre-created text model to obtain a vector matrix of the current user question;

[0129] The search and matching module 30 is used to match the vector matrix of the current user question with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer.

[0130] The text search device provided by the present application adopts the text search method in the above embodiment to improve the search accuracy and search speed. Compared with the prior art, the beneficial effects of the text search device provided by the present application are the same as the beneficial effects of the text search method provided by the above embodiment, and other technical features in the text search device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0131] In addition, to achieve the above objectives, the present application also proposes a text retrieval system, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the text retrieval method described above.

[0132] Reference below Figure 7 , which shows a schematic diagram of the structure of a text retrieval system suitable for implementing the embodiment of the present application. The text retrieval system in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The text retrieval system shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0133] like Figure 7As shown, the text retrieval system may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the text retrieval system are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. Communication device 1009 can allow the text retrieval system to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a text retrieval system with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0134] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0135] The text retrieval system provided by the present application adopts the text retrieval method in the above embodiment to improve the retrieval accuracy and retrieval speed. Compared with the prior art, the beneficial effects of the text retrieval system provided by the present application are the same as the beneficial effects of the text retrieval method provided by the above embodiment, and other technical features of the system are the same as those disclosed in the method of the previous embodiment, which will not be described in detail here.

[0136] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0137] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the text retrieval method described above are implemented.

[0138] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0139] The computer-readable storage medium may be included in the text retrieval system; or it may exist independently without being assembled into the text retrieval system.

[0140] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the text retrieval system, the text retrieval system: obtains the current user question input by the user by responding to the text retrieval instruction; performs vector matrix conversion on the current user question through a pre-created text model to obtain the current user question vector matrix; matches the current user question vector matrix with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer.

[0141] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0144] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned text retrieval method, which can improve the retrieval accuracy and retrieval speed. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the text retrieval method provided in the above-mentioned embodiment, and will not be repeated here.

[0145] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the text retrieval method described above are implemented.

[0146] The computer program product provided by this application can improve the search accuracy and search speed. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the text search method provided by the above embodiment, which will not be repeated here.

[0147] One or more technical solutions proposed in this application have at least the following technical effects: by responding to the text search instruction, the current user question input by the user is obtained; the current user question is converted into a vector matrix through a pre-created text model to obtain the current user question vector matrix; the current user question vector matrix is ​​matched with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer. This application combines the knowledge base scenario and business needs, takes ES as the knowledge base, converts the knowledge into a vector matrix and stores it in the ES knowledge base, and converts the user question into a vector, and performs similarity matching and recall in the knowledge base. Due to the use of vector conversion, massive text knowledge can be converted into a spatial vector representation. The vectorized feature matrix can include both the complete text subject content and keywords, so that the knowledge information dimension is more complete and specific, making the entire retrieval more accurate than the fuzzy matching and exact matching of ES. At the same time, due to the use of an offline vectorized storage solution based on ES storage, the waiting time for real-time feature extraction in retrieval is reduced, repeated resource calculations are reduced, and the delay of real-time extraction is reduced, making the retrieval more efficient and more accurate.

[0148] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A text retrieval method, characterized in that: The method comprises: In response to the text search instruction, obtaining the current user question input by the user; Performing vector matrix conversion on the current user question through a pre-created text model to obtain a current user question vector matrix; The current user question vector matrix is ​​matched with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer.

2. The method according to claim 1, characterized in that The step of matching the current user question vector matrix with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer includes: Calculate the similarity distance between the current user question vector matrix and the vector matrix in the knowledge base; Score the similarity distance to obtain the similarity score retrieval result; The vector matrix in the knowledge base is filtered and screened according to the similarity score retrieval result, and the knowledge is finally recalled to obtain the knowledge answer with similarity hit.

3. The method according to claim 2, characterized in that The method further comprises: In the large-model domain question-answering scenario, the large-model domain question-answering recall engine is called to retrieve text knowledge vectors, analyze the recalled knowledge hit by similarity matching, format the recalled knowledge text, and output the processed knowledge answer; In the QA knowledge domain question-answering scenario, the general QA knowledge domain question-answering recall engine is called to retrieve QA knowledge vectors and recall QA answers with similarity hits.

4. The method according to claim 1, characterized in that Before the step of matching the current user question vector matrix with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer, the step further includes: Determine the application scenarios of current knowledge storage; If the application scenario of the current knowledge storage is a large-model domain question-answering scenario based on document knowledge, the ES knowledge base is bound to the large-model domain question-answering recall engine for retrieval; Preprocessing the pre-collected text knowledge to obtain preprocessed text knowledge; According to the business scenario, an adaptive text slicing rule template is selected to perform a slicing operation on the content of the preprocessed text knowledge to obtain slice content; The text2Vec feature extraction model is used to extract features from the slice content, and the extracted text features are converted into a text vector matrix, which is then stored in the ES knowledge base.

5. The method according to claim 4, characterized in that The text model is a text2Vec feature extraction model, the slice content is a knowledge fragment of a preset length, the slice content contains complete text paragraph information and meets the token restrictions of the large model.

6. The method according to claim 4, characterized in that The step of preprocessing the pre-collected text knowledge to obtain the preprocessed text knowledge includes: Clean the pre-collected text knowledge and remove invalid tags and format contents in the text knowledge content; The privacy content of the text information of the entire text knowledge is detected, and the relevant privacy information and / or non-compliant content are removed to obtain the preprocessed text knowledge.

7. The method according to claim 4, characterized in that The step of selecting an adaptive text slicing rule template to perform a slicing operation on the content of the preprocessed text knowledge according to the business scenario to obtain the slicing content also includes: The content of text knowledge is classified according to different business scenarios.

8. The method according to claim 4, characterized in that Before the step of determining the application scenario of the current knowledge storage, the following steps are also included: If the application scenario of the current knowledge storage is based on the QA knowledge domain question and answer scenario, the ES knowledge base is bound to the general QA knowledge domain question and answer recall engine for retrieval; Get pre-collected questions and answers; The questions and answers are converted into vector matrices and stored in the ES knowledge base.

9. A text retrieval device, characterized in that: The device comprises: An acquisition module, used for acquiring a current user question input by a user in response to a text search instruction; A vector conversion module, used for performing vector matrix conversion on the current user question through a pre-created text model to obtain a vector matrix of the current user question; The retrieval matching module is used to match the vector matrix of the current user question with the vector matrix in the pre-built ES knowledge base to obtain the matched knowledge answer.

10. A text retrieval system, characterized in that: The system comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the text retrieval method according to any one of claims 1 to 8.

11. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the text retrieval method according to any one of claims 1 to 8 are implemented.

12. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the text search method according to any one of claims 1 to 8 are implemented.

Images