Text processing method and device

By using multiple recall units in RAG technology for text retrieval, combined with priority information of intervention and non-intervention recall units, the problems of low knowledge retrieval ability and long process in the professional field of RAG technology are solved, and the question-and-answer model is improved Answer accuracy and user experience.

CN118520098BActive Publication Date: 2024-11-08HUNDSUN TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411000017.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-08
Estimated Expiration
2044-07-24

AI Technical Summary

Technical Problem

RAG technology has low knowledge retrieval capabilities in professional fields, long process and low generalization, which can affect the accuracy of answers when problems arise in the intermediate processing link.

Method used

Using a multiple recall unit, including an intervention recall unit and a non-intervention recall unit, text search is performed by calculating the vector distance between the question text and the candidate question text in the question and answer text data, and selecting the target based on the priority information of the intervention and non-intervention recall unit. Text and input to the generation module for processing.

Benefits of technology

It improves the search ability and accuracy of the question-and-answer model, reduces the impact of the intermediate links on the output answers, and improves the user experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118520098B_ABST
    Figure CN118520098B_ABST
Patent Text Reader

Abstract

The embodiments of the present specification provide a text processing method and device, wherein the text processing method includes application to a question-answering model, wherein the question-answering model includes a retrieval module and a generation module, including: receiving a question text submitted to the question-answering model, and inputting the question text into the retrieval module; performing text retrieval on the question text through a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, a database of the intervention recall unit pre-stores question-answering text data, and the intervention recall unit performs text retrieval on the question text by calculating the vector distance between the question text and candidate question texts in the question-answering text data; selecting a target text from a first candidate text recalled by the intervention recall unit and a second candidate text recalled by the non-intervention recall unit according to the text retrieval result, inputting the target text and the question text into the generation module, and obtaining an answer text to the question text.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present specification relate to the field of text processing technology, and more particularly to a text processing method and device. Background Art

[0002] With the development of computer technology, natural language processing technology has been applied in more and more scenarios. RAG (Retrieval-Augmented Generation) is a natural language processing technology that combines information retrieval and text generation. It can enhance the generation ability of the language model by retrieving relevant information from a large number of documents, thereby improving the accuracy and richness of the generated text, so that it can have a good performance in the question-answering scenario. RAG technology mainly includes a retrieval module and a generation module; among them, the retrieval module is used to retrieve documents or document fragments related to the input query from the document library, and then handed over to the generation module for processing, so that the query-related answers can be output and output. However, since the retrieval module of RAG mainly retrieves knowledge through embedded vectors, and the differences between knowledge in professional fields are small, its knowledge retrieval ability in professional fields is greatly reduced; and because RAG has a long process and low generalization, when problems occur in the intermediate processing link, it will directly affect the accuracy of the final output answer. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of this, the embodiments of this specification provide a text processing method. One or more embodiments of this specification also relate to a text processing apparatus, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.

[0004] According to a first aspect of an embodiment of this specification, a text processing method is provided, which is applied to a question-answering model, wherein the question-answering model includes a retrieval module and a generation module, including:

[0005] Receiving a question text submitted to the question-answering model, and inputting the question text into the retrieval module;

[0006] Performing a text search on the question text by using a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, wherein a database of the intervention recall unit stores question-answer text data in advance, and the intervention recall unit performs a text search on the question text by calculating a vector distance between the question text and a candidate question text in the question-answer text data;

[0007] A target text is selected from the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result, and the target text and the question text are input into the generation module to obtain an answer text to the question text.

[0008] According to a second aspect of an embodiment of this specification, another text processing method is provided, which is applied to a server, wherein the server deploys a question-answering model, wherein the question-answering model includes a retrieval module and a generation module, including:

[0009] Receive a question text submitted by a client, and input the question text into the search module;

[0010] Performing a text search on the question text by using a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, wherein a database of the intervention recall unit stores question-answer text data in advance, and the intervention recall unit performs a text search on the question text by calculating a vector distance between the question text and a candidate question text in the question-answer text data;

[0011] Selecting a target text from the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result;

[0012] The target text and the question text are input into the generation module to obtain the answer text of the question text, and the answer text is fed back to the client.

[0013] According to a third aspect of an embodiment of this specification, a text processing device is provided, which is applied to a question-answering model, wherein the question-answering model includes a retrieval module and a generation module, including:

[0014] A receiving module, configured to receive a question text submitted to the question-answering model, and input the question text into the retrieval module;

[0015] A retrieval module is configured to perform text retrieval on the question text through a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, wherein a database of the intervention recall unit stores question-answer text data in advance, and the intervention recall unit performs text retrieval on the question text by calculating a vector distance between the question text and a candidate question text in the question-answer text data;

[0016] The determination module is configured to select a target text from the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result, input the target text and the question text into the generation module, and obtain the answer text of the question text.

[0017] According to a fourth aspect of an embodiment of this specification, another text processing device is provided, which is applied to a server, and the server deploys a question-answering model, wherein the question-answering model includes a retrieval module and a generation module, including:

[0018] A text receiving module is configured to receive a question text submitted by a client and input the question text into the search module;

[0019] A text retrieval module is configured to perform text retrieval on the question text through a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, wherein a database of the intervention recall unit stores question-answer text data in advance, and the intervention recall unit performs text retrieval on the question text by calculating a vector distance between the question text and a candidate question text in the question-answer text data;

[0020] A text selection module is configured to select a target text from the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result;

[0021] The feedback text module is configured to input the target text and the question text into the generation module, obtain the answer text of the question text, and feed back the answer text to the client.

[0022] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including:

[0023] Memory and processor;

[0024] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned text processing method are implemented.

[0025] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned text processing method are implemented.

[0026] According to a seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program or instructions, which implement the steps of the above-mentioned text processing method when executed by a processor.

[0027] The text processing method provided in this embodiment can improve the accuracy of the question answering model and reduce the impact of problems in the intermediate links on the output answer. After receiving the question text, the question text can be first input into the retrieval module in the question answering model; and in order to provide accurate candidate texts that match the question in the retrieval stage, the question text can be retrieved by the multi-way recall unit preset by the retrieval module, and the multi-way recall unit is composed of an intervention recall unit and a non-intervention recall unit, and the database of the intervention recall unit pre-stores the question and answer text data; therefore, it can be ensured that when the intervention recall unit performs text recall, the retrieval can be completed by calculating the vector distance between the question text and the candidate question text in the question and answer text data, ensuring that the retrieval is more accurate. On this basis, the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit can be determined according to the text retrieval results; thereafter, the target text can be selected from the first candidate text and the second candidate text, so that when screening the candidate text matching the question text, the priority information of the intervention recall unit and the non-intervention recall unit can be combined, and finally the target text and the answer text can be input into the generation module for processing, so that the answer text corresponding to the question text can be obtained according to the processing results. In the question-answering stage, the recall capability of the retrieval module can be improved through multi-way recall, thereby ensuring that the generation module can output answer text with higher accuracy, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a schematic diagram of a text processing method provided by an embodiment of this specification;

[0029] Figure 2 is a flowchart of a text processing method provided by an embodiment of this specification;

[0030] Figure 3 is a flowchart of another text processing method provided by an embodiment of this specification;

[0031] Figure 4 is a structural schematic diagram of a text processing device provided by an embodiment of this specification;

[0032] Figure 5 is a structural schematic diagram of another text processing device provided by an embodiment of this specification;

[0033] Figure 6 It is a structural block diagram of a computing device provided by an embodiment of this specification. DETAILED DESCRIPTION

[0034] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0035] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0036] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0037] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0038] First, the terms involved in one or more embodiments of this specification are explained.

[0039] RAG (Retrieval-Augmented Generation) is a model that combines information retrieval and text generation techniques. It retrieves external knowledge bases to obtain additional corpora and uses these corpora to improve the generation effect of large language models (LLMs).

[0040] In this specification, a text processing method is provided. This specification also relates to a text processing apparatus, a computing device, a computer-readable storage medium and a computer program product, which are described in detail one by one in the following embodiments.

[0041] See also Figure 1 As shown in the schematic diagram, the text processing method provided in this embodiment, in order to improve the accuracy of answering questions of the question-answering model and reduce the impact of problems in the intermediate links on the output answers, can first input the question text into the retrieval module in the question-answering model after receiving the question text; and in order to provide accurate candidate texts that match the question in the retrieval stage, the question text can be retrieved by a multi-way recall unit preset in the retrieval module, and the multi-way recall unit is composed of an intervention recall unit and a non-intervention recall unit, and the database of the intervention recall unit pre-stores the question and answer text data; therefore, it can be ensured that when the intervention recall unit performs text recall, it can complete the retrieval by calculating the vector distance between the question text and the candidate question text in the question and answer text data, thereby ensuring that the retrieval is more accurate. On this basis, the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit can be determined according to the text retrieval results; thereafter, the target text can be selected from the first candidate text and the second candidate text, so that when screening the candidate text matching the question text, the priority information of the intervention recall unit and the non-intervention recall unit can be combined, and finally the target text and the answer text can be input into the generation module for processing, so that the answer text corresponding to the question text can be obtained according to the processing results. In the question-answering stage, the recall ability of the retrieval module can be improved through multi-way recall, so as to ensure that the generation module can output a more accurate answer text, thereby improving the user experience.

[0042] See also Figure 2 , Figure 2 A flowchart of a text processing method provided according to an embodiment of the present specification is shown, wherein the method is applied to a question-answering model, wherein the question-answering model includes a retrieval module and a generation module, and specifically includes the following steps.

[0043] Step S202, receiving a question text submitted to the question-answering model, and inputting the question text into the retrieval module.

[0044] The text processing method provided in this embodiment can be applied to any text processing field. By deploying a question-and-answer model, it is possible to answer questions about knowledge in any field, such as education, finance, laws and regulations, games, etc. By deploying a question-and-answer model in each field, it is possible to answer more accurate answers in the field through the question-and-answer model, thereby meeting the user's question-and-answer needs.

[0045] Specifically, the question-answering model specifically refers to a large language model including a retrieval module and a generation module, whose input is a question text and whose output is an answer text. Among them, the retrieval module is used to retrieve documents or document fragments that match the question text from a large document library when answering questions, so as to support the subsequent input generation module to generate answer text for the question text. In practical applications, the retrieval module can be implemented by a pre-trained retrieval model. The retrieval module provided in this embodiment is encapsulated with a multi-way recall unit, which can realize the recall of multiple candidate texts for the question text, thereby improving the generation accuracy of the answer text. Correspondingly, the generation module specifically refers to a module in the question-answering model that combines the question text and the retrieved candidate text to generate the answer text, which can be implemented by a pre-trained language generation model (such as GPT or T5), which can use the retrieved information to generate answers, summaries or other forms of text output. Further, the question text specifically refers to the text corresponding to the target field and composed of questions raised by the user.

[0046] Based on this, in order to improve the accuracy of the question answering model and reduce the impact of problems in the intermediate links on the output answer, after receiving the question text, the question text can be first input into the retrieval module in the question answering model; and in order to provide accurate and matching candidate texts in the retrieval stage, the question text can be retrieved through the multi-way recall unit preset by the retrieval module, and the multi-way recall unit is composed of an intervention recall unit and a non-intervention recall unit, and the database of the intervention recall unit pre-stores the question and answer text data; therefore, it can be ensured that when the intervention recall unit performs text recall, it can complete the retrieval by calculating the vector distance between the question text and the candidate question text in the question and answer text data, ensuring that the retrieval is more accurate. On this basis, the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit can be determined according to the text retrieval results; thereafter, the target text can be selected from the first candidate text and the second candidate text, so that when screening the candidate text matching the question text, the priority information of the intervention recall unit and the non-intervention recall unit can be combined, and finally the target text and the answer text can be input into the generation module for processing, so that the answer text corresponding to the question text can be obtained according to the processing results.

[0047] Furthermore, when the question text is input into the retrieval module in the question-answering model for processing, in order to improve the retrieval accuracy and reduce the retrieval scope, thereby improving the retrieval efficiency, semantic entity extraction can be performed on the question text, and the semantic entity can be mapped, so as to obtain the standard semantic entity and then narrow the retrieval scope. In this embodiment, the specific implementation method is as follows:

[0048] The question text is input into the entity extraction model for processing to obtain the semantic entity of the question text; the semantic entity is entity mapped according to a preset vocabulary to obtain the target semantic entity, and the target semantic entity is input into the retrieval module. Further, the question text is subjected to text retrieval by the multi-way recall unit preset in the retrieval module, including: the initial candidate text associated with the target semantic entity is screened in the retrieval database corresponding to the retrieval module by the retrieval unit in the retrieval module, and a candidate text set is constructed according to the screening result; the question text is subjected to text retrieval in the candidate text set by the multi-way recall unit preset in the retrieval module.

[0049] Specifically, semantic entities specifically refer to entities extracted after entity recognition is performed on the question text. Accordingly, the entity advance model specifically refers to a model that can extract named entities after semantic recognition of the question text, which can adopt any type of named entity recognition model, and this embodiment does not make any limitation here. Accordingly, the vocabulary specifically refers to a vocabulary that is pre-configured for the current field and records the entity mapping relationship. The vocabulary records the correspondence between non-standard entities and standard entities, such as the company abbreviation corresponding to the company full name, the actor's stage name corresponding to the actor's real name, etc., which is used to realize the mapping of semantic entities to target semantic entities, wherein the target semantic entity is the standard entity name corresponding to the semantic entity. Accordingly, the initial candidate text specifically refers to the text corresponding to the document obtained after preliminary screening of the document for the question text in combination with the target semantic entity, which is used to reduce the scope of knowledge retrieval and improve the efficiency of question and answer. Accordingly, the candidate text set specifically refers to the text set composed of the initial candidate texts obtained after preliminary screening.

[0050] Based on this, after obtaining the question text submitted by the user, in order to enable the question-answering model to quickly and accurately feedback the answer text, the question text can be input into the entity extraction model for processing before the retrieval module is executed, so as to extract the semantic entity in the question text; and then, in order to reduce the knowledge retrieval scope, the semantic entity can be entity mapped according to the preset vocabulary, the target semantic entity can be obtained, and the target semantic entity can be input into the retrieval module. After that, the initial candidate text associated with the target semantic entity can be screened in the retrieval database corresponding to the retrieval module through the retrieval unit in the retrieval module, so as to reduce the retrieval scope by preliminary screening of the text scope, so as to construct a candidate text set according to the screening results; and then, the question text can be retrieved in the candidate text set through the multi-way recall unit preset in the retrieval module, so as to improve the retrieval efficiency.

[0051] That is to say, considering that after the user submits the question text, the semantic entities contained in the question text may not be named as standard entities, it is difficult to perform effective retrieval in the knowledge base at this time; therefore, in order to avoid this problem, semantic entity enhancement processing can be performed. Semantic entities can be extracted from the question text through the LLM large model first, and then it can be determined whether the extracted semantic entities are valid semantic entities, which can be understood as whether they are semantic entities strongly related to the question text. If not, no extraction is required, and text retrieval can be performed directly on the basis of the full amount of documents in the knowledge base. If it exists, the preset vocabulary can be used to map the semantic entity to a standard semantic entity, and then the standard semantic entity can be used to preliminarily filter the documents in the knowledge base. After obtaining the documents that match the standard semantic entity, it can be used as the basis for subsequent retrieval, thereby effectively reducing the scope of retrieval.

[0052] For example, user A submits the question text {What are the regulations of H Company?} through the client. At this time, the LLM large language model can be used to perform semantic entity recognition on the question text, and the semantic entity can be determined to be {H Company} based on the recognition result. Then, the semantic entity {H Company} can be mapped using the pre-established vocabulary, and the standard semantic entity corresponding to the semantic entity {H Company} can be determined to be {ABCH Co., Ltd.} based on the mapping result. On this basis, the standard semantic entity {ABCH Co., Ltd.} can be used to perform a preliminary search in the knowledge base corresponding to the question-answering model, and relevant documents matching {ABCH Co., Ltd.} can be screened out and used as the retrieval basis for subsequent responses to the question text.

[0053] In summary, by mapping the semantic entities in the question text, the retrieval efficiency can be improved in subsequent retrieval, and by narrowing the retrieval scope through the target semantic entities, the efficiency of subsequent answer text generation can be effectively improved, so as to quickly feedback the answer text to the user and improve the user experience.

[0054] Step S204: performing text retrieval on the question text through a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, a database of the intervention recall unit pre-stores question and answer text data, and the intervention recall unit performs text retrieval on the question text by calculating a vector distance between the question text and candidate question texts in the question and answer text data.

[0055] Step S206, selecting a target text from the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result, inputting the target text and the question text into the generation module to obtain an answer text to the question text.

[0056] Specifically, after the question text is input into the retrieval module in the question and answer model as mentioned above, further, in order to improve the retrieval accuracy and reduce the impact of the retrieval results on the accuracy of answer text generation, a multi-way recall unit can be pre-set in the retrieval module to improve the retrieval accuracy by multi-way recall of candidate texts; on this basis, in order to further improve the retrieval accuracy, an intervention recall unit and a non-intervention recall unit can be set in the multi-way recall unit, and the database of the intervention recall unit will pre-store the question and answer text data, so that when performing text recall, it can be achieved that the recall result of the intervention recall unit can be considered to be searched from the question and answer text data, thereby improving the matching degree between the candidate text recalled by the intervention recall unit and the question text, so as to improve the subsequent answer text generation accuracy.

[0057] That is to say, after obtaining the question text, the question text can be first input into the retrieval module. At this time, the text retrieval can be performed through the multi-way recall unit pre-installed with the intervention recall unit and the non-intervention recall unit. Moreover, since the database corresponding to the intervention recall unit pre-stores the question and answer text data, and the question and answer text data consists of questions and answers, the text retrieval of the question text can be realized by calculating the vector distance between the question text and the candidate question text in the question and answer text data. Thus, the first candidate text can be recalled for the question text through the intervention recall unit, and the second candidate text can be recalled through the non-intervention recall unit.

[0058] After obtaining the first candidate text and the second candidate text, in order to improve the generation accuracy when generating the answer text, you can first select the target text from the first candidate text and the second candidate text. After obtaining the target text, you can input the target text and the answer text into the generation module for processing, so that the answer text corresponding to the question text can be obtained according to the processing results.

[0059] Among them, the multi-way recall unit specifically refers to a recall unit composed of an intervention recall unit and a non-intervention recall unit, which can perform recall processing of candidate texts through at least two-way recall modes, thereby improving the accuracy of text retrieval. Correspondingly, the intervention recall unit specifically refers to a recall unit that can perform vector calculation on the question text and the candidate question text in the question and answer text data pre-stored in the database corresponding to the intervention recall unit when performing text recall processing. The recall unit pre-stores a data structure composed of the texts belonging to the question and the answer in its corresponding database. When performing text recall, the question text and the question in the data structure can be vector calculated to determine whether the question text already exists in the corresponding answer text, thereby improving the accuracy of subsequent answer text generation. Correspondingly, the question and answer text data is a data structure composed of the text belonging to the question and the answer. Correspondingly, the non-intervention recall unit specifically refers to a recall unit that performs text recall in combination with the question text and / or the embedding vector of the question text, which can be an Embedding recall unit and / or an MB25 recall unit, realizing text recall through Embedding calculation or BM25 calculation. The target text specifically refers to a set number of candidate texts selected from the first candidate text and the second candidate text that can be used to generate the answer text corresponding to the question text. When selecting the target text, the target text can be selected according to the arrangement order of the priority information corresponding to the first candidate text and the second candidate text. This embodiment does not make any limitation here. Correspondingly, the answer text specifically refers to the answer text generated for the question text.

[0060] Furthermore, the first candidate text specifically refers to a plurality of candidate texts determined by the intervention recall unit after performing text retrieval on the question text set, and correspondingly, the second candidate text specifically refers to a plurality of candidate texts determined by the non-intervention recall unit after performing text retrieval on the question text set.

[0061] In specific implementation, when performing text set retrieval, different recall units will perform text recall in different ways. In this embodiment, when the intervention recall unit is an automatic recall unit, and the non-intervention recall unit is an embedded recall unit and a word frequency recall unit, the multi-channel recall unit preset by the retrieval module performs text retrieval on the question text, including:

[0062] Step S2042, performing a text search on the question text in the database of the embedded recall unit through the embedded recall unit preset in the retrieval module, and performing a text search on the question text in the database of the word frequency recall unit through the word frequency recall unit preset in the retrieval module.

[0063] Specifically, the embedding recall unit refers to a recall unit that performs text recall by combining the embedding vector of the question text with the embedding vector corresponding to the candidate text in the database corresponding to the embedding recall unit. Correspondingly, the word frequency recall unit refers to a recall unit that performs text recall by combining the question text with the candidate text in the database corresponding to the word frequency recall unit and performing word frequency value calculation.

[0064] Based on this, after the question text is input into the retrieval module, the embedded recall unit preset in the retrieval module can be used to perform a text search on the question text in the database of the embedded recall unit, and the word frequency recall unit preset in the retrieval module can be used to perform a text search on the question text in the database of the word frequency recall unit. In addition, considering that the text search scope has been narrowed down before, the text used as the search scope in the databases corresponding to the embedded recall unit and the word frequency recall unit can be the narrowed text scope, thereby improving the retrieval efficiency. In actual applications, the databases corresponding to each recall unit can be the same knowledge base or different knowledge bases, and this embodiment does not make any limitation here.

[0065] Furthermore, when the non-interference recall unit performs text retrieval, since the non-interference recall unit includes a word frequency recall unit and an embedded recall unit, the two units need to use different recall methods to determine the second candidate text. In this embodiment, the specific implementation method is as follows:

[0066] Determine the candidate text embedding vector corresponding to the candidate text in the database corresponding to the embedding recall unit; calculate the second vector distance between the question text embedding vector and the candidate text embedding vector through the embedding recall unit preset by the retrieval module; calculate the word frequency value between the question text and the candidate text contained in the database corresponding to the word frequency recall unit through the word frequency recall unit preset by the retrieval module;

[0067] On this basis, the second vector distance and word frequency value calculated in combination with the two can be used for the subsequent determination of the second candidate text, specifically including: extracting the embedded recall candidate text corresponding to the embedded recall unit from the database corresponding to the embedded recall unit according to the second vector distance; extracting the word frequency recall candidate text corresponding to the word frequency recall unit from the database corresponding to the word frequency recall unit according to the word frequency value; and using the embedded recall candidate text and the word frequency recall candidate text as the second candidate text.

[0068] Specifically, the candidate text embedding vector specifically refers to the embedding vector corresponding to the candidate text in the database corresponding to the embedding recall unit. Correspondingly, the question text embedding vector specifically refers to the embedding vector corresponding to the question text; Correspondingly, the first vector distance specifically refers to the vector distance obtained after the question text embedding vector and each candidate text embedding vector are calculated. Correspondingly, the word frequency value specifically refers to the value obtained after the BM25 value calculation is performed on the question text and the candidate text. Correspondingly, the embedded recall candidate text specifically refers to the candidate text recalled after the embedded recall unit performs text retrieval on the question text. Correspondingly, the word frequency recall candidate text specifically refers to the candidate text recalled after the word frequency recall unit performs text retrieval on the question text. It should be noted that when each recall unit performs text recall, the number of text recalls can be set according to actual needs, such as 3, 5, etc., and this embodiment does not make any limitation here.

[0069] Based on this, when the non-intervention recall unit performs text recall, it can first determine the candidate text embedding vector corresponding to the candidate text in the database corresponding to the embedded recall unit; thereafter, the second vector distance between the question text embedding vector and each candidate text embedding vector can be calculated through the embedded recall unit preset by the retrieval module; at the same time, for the word frequency recall unit, the word frequency value between the question text and the candidate text contained in the database corresponding to the word frequency recall unit can be calculated through the word frequency recall unit preset by the retrieval module. After obtaining the second vector distance and the word frequency value, the embedded recall candidate text corresponding to the embedded recall unit can be extracted from the database corresponding to the embedded recall unit according to the second vector distance; the word frequency recall candidate text corresponding to the word frequency recall unit can be extracted from the database corresponding to the word frequency recall unit according to the word frequency value; thereafter, the embedded recall candidate text and the word frequency recall candidate text can be combined as the second candidate text.

[0070] In practical applications, when recall candidate texts are extracted from the database corresponding to each recall unit according to the word frequency value and the second vector distance, the candidate texts can be sorted in descending order according to the word frequency value and the second vector distance. According to the sorting result, a set number of candidate texts can be selected from large to small as recall candidate texts to form the second candidate text for subsequent text generation.

[0071] In summary, by calculating the word frequency value and the second vector distance to recall the candidate text for each recall unit, it can ensure that the accuracy of answer text generation is improved when the answer text is generated subsequently, and it can avoid the situation where any recall unit fails, and still provide question and answer services through the question and answer model.

[0072] Step S2044: perform text search on the question text in the database of the automatic recall unit through the automatic recall unit preset in the retrieval module.

[0073] Specifically, the automatic recall unit specifically refers to a recall unit that performs vector calculation to complete text recall by combining the embedding vector of the question text with the embedding vector corresponding to the candidate question text in the question and answer text data pre-stored in the database corresponding to the automatic recall unit.

[0074] Furthermore, when the intervention recall unit is an automatic recall unit, when performing text retrieval, it actually calculates the first vector distance between the question text embedding vector of the question text and the embedding vector corresponding to the question in the question and answer text data pre-stored by the automatic recall unit. In this embodiment, the specific implementation method is as follows:

[0075] Determine a question text embedding vector corresponding to the question text, and a candidate question text embedding vector corresponding to the candidate question text in the question and answer text data pre-stored in a database corresponding to the automatic recall unit; calculate a first vector distance between the question text embedding vector and the candidate question text embedding vector through the automatic recall unit preset in the retrieval module;

[0076] On this basis, the specific implementation of determining the first candidate text in combination with the first vector distance includes: determining the target question and answer text data in the database corresponding to the automatic recall unit according to the first vector distance, and extracting the first candidate text corresponding to the automatic recall unit from the target question and answer text data.

[0077] Specifically, the candidate question text embedding vector specifically refers to the embedding vector corresponding to the candidate question text in the question and answer text data in the database corresponding to the automatic recall unit. Since the automatic recall unit has been modified, the question and answer text data is stored in its database, and the data is composed of the texts belonging to the questions and answers. Therefore, when performing text recall, the distance between the embedding vectors of the question text and the question corresponding to the text belonging to the answer can be calculated, thereby improving the text recall rate. Correspondingly, the first vector distance is the vector distance calculated between the embedding vector of the question text and the embedding vector of the candidate question text. Correspondingly, the target question and answer text data specifically refers to the question and answer text data that can be screened out from the question and answer text data pre-stored in the database after the first vector distance calculation to extract the first candidate text.

[0078] Based on this, when the intervention recall unit performs text recall, it can first determine the question text embedding vector corresponding to the question text, as well as the candidate question text embedding vectors corresponding to the candidate question text in the question and answer text data pre-stored in the database corresponding to the automatic recall unit; at this time, the automatic recall unit preset by the retrieval module can be used to calculate the first vector distance between the question text embedding vector and each candidate question text embedding vector; on this basis, the target question and answer text data can be screened out in the database corresponding to the automatic recall unit according to the first vector distance; since the question and answer text data is composed of the text belonging to the question and the answer, the text belonging to the answer can be selected from the target question and answer text data as the first candidate text.

[0079] In actual applications, when filtering target question and answer text data from a database according to the first vector distance, the question and answer text data can be sorted in descending order according to the first distance, and a set number of question and answer text data can be selected from large to small according to the sorting result as target question and answer text data to achieve subsequent extraction of the first candidate text for text generation.

[0080] Using the above example, after the question text {What are the regulations of H Company?} is input into the retrieval module in the question-answering model, since the retrieval module includes the embedding recall unit, the bm25 recall unit, and the FAQ (Frequently Asked Questions) recall unit, the three recall units can be used to perform text recall processing. Specifically, the number of recalls N is set to 3. At this time, the embedding recall unit can calculate the vector distance between the embedding vector E corresponding to the question text and the embedding vectors corresponding to the relevant documents matching {ABCH Co., Ltd.}. According to the calculation results, the recalled documents can be obtained as candidate documents E1, E2, and E3.

[0081] The BM25 recall unit can be used to calculate the BM25 values ​​corresponding to the question text and the relevant documents matching {ABCH Co., Ltd.}. According to the calculation results, the recalled documents can be obtained as candidate documents B1, B2 and B3 respectively.

[0082] The FAQ recall unit can be used to calculate the vector distance between the embedding vector E corresponding to the question text and the embedding vector corresponding to the document association question in the relevant documents matching {ABCH Co., Ltd.}. Based on the calculation results, the recalled documents corresponding to the document association question can be obtained. The recalled documents are candidate documents F1, F2 and F3 respectively.

[0083] After obtaining the candidate documents E1, E2 and E3; B1, B2 and B3; F1, F2 and F3, the candidate documents may be sorted so that the target document may be subsequently screened from the sorted documents for generating the answer text for the question text.

[0084] In addition, in order to support the candidate texts recalled by the automatic recall unit with a higher priority, so as to avoid being filtered out when selecting the target text, the question and answer text data can be stored in the database of the automatic recall unit in advance. The construction of the question and answer text data is implemented as follows:

[0085] Determine the question text to be stored that meets the pre-storage conditions corresponding to the automatic recall unit, and determine the answer text to be stored to which the answer text corresponding to the question text to be stored belongs; construct question and answer text data according to the question text to be stored and the answer text to be stored, and store the question and answer text data in the database corresponding to the automatic recall unit.

[0086] Specifically, the pre-stored conditions specifically refer to the conditions set for the automatic recall unit, which are used to screen out complex question texts and / or difficult question texts; accordingly, the question text to be stored is the complex question text and / or the difficult question text; accordingly, the answer text to be stored specifically refers to the text to which the answer text corresponding to the question text to be stored belongs.

[0087] Based on this, in order to support the automatic recall unit to improve the recall rate when screening the first candidate text, the question and answer text data can be stored in the database corresponding to the automatic recall unit. In this process, the question text to be stored that meets the pre-storage conditions corresponding to the automatic recall unit can be determined, and the answer text to be stored to which the answer text corresponding to the question text to be stored belongs can be determined; on this basis, the question and answer text data can be constructed according to the question text to be stored and the answer text to be stored, and the question and answer text data can be stored in the database corresponding to the automatic recall unit, so that when the automatic recall unit is used, the candidate text can be recalled from the question and answer text data by matching the question first, thereby improving the text retrieval accuracy.

[0088] In summary, text retrieval is performed in parallel by multiple recall units, and the intervention recall unit can improve the priority of candidate texts, so as to ensure that the accuracy of answer generation is further improved when generating answers, thereby improving the user experience.

[0089] Furthermore, when selecting the target text from the first candidate text and the second candidate text, considering that the target text is the basis for generating the answer text later, in order to ensure the selection accuracy of the target text, it can be achieved by comparing the priorities. In this embodiment, the specific implementation is as follows:

[0090] Determine the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result; determine the first priority information corresponding to the first candidate text and the second priority information corresponding to the second candidate text; sort the first candidate text and the second candidate text according to the first priority information and the second priority information; select the target text from the first candidate text and the second candidate text according to the sorting result.

[0091] Specifically, the first priority information specifically refers to the priority level information corresponding to the first candidate text, and correspondingly, the second priority information specifically refers to the priority level information corresponding to the second candidate text, wherein the higher the priority level information, the higher the priority of the candidate text, and the higher its matching degree with the question text.

[0092] Based on this, when selecting the target text, in order to be able to use the candidate text that matches the question text more highly to generate the answer text, the first candidate text recalled by the intervening recall unit and the second candidate text recalled by the non-intervening recall unit can be determined based on the text retrieval results; then, the first priority information corresponding to the first candidate text and the second priority information corresponding to the second candidate text are determined; according to the first priority information and the second priority information, the first candidate text and the second candidate text are sorted; thereafter, the target text can be selected from the first candidate text and the second candidate text according to the sorting result for subsequent use.

[0093] In practical applications, priority information can be set according to the priority of the recall unit. For example, the retrieval module includes an embedding recall unit, a bm25 recall unit and a FAQ recall unit, among which the priority of the embedding recall unit is greater than the priority of the bm25 recall unit and greater than the priority of the FAQ recall unit. When sorting the candidate texts recalled by each recall unit, the priority of the candidate text corresponding to the embedding recall unit is higher than the priority of the candidate text corresponding to the bm25 recall unit, and the priority of the candidate text corresponding to the bm25 recall unit is higher than the priority of the candidate text corresponding to the FAQ recall unit.

[0094] Using the above example, after obtaining candidate documents E1, E2 and E3; B1, B2 and B3; F1, F2 and F3, since the priority of embedding recall unit> the priority of bm25 recall unit> the priority of FAQ recall unit; therefore, after sorting the candidate documents, the sorting result is {E1, B1, F1, E2, B2, F2, E3, B3, F3}. When K is equal to 5, the TOP5 candidate documents can be selected as the target documents in the sorting results, and the target documents are determined to include {E1, B1, F1, E2, B2}; finally, each target document and question text are input into the generation module of the question-answering model for processing, and the answer text {abcdef...} corresponding to the question text {What are the regulations of H Company?} can be obtained according to the processing results, and it can be fed back to the user for viewing.

[0095] In addition, considering that the intervention recall unit is affected by the intervention, the probability of the text recalled by it being the target text is higher. Therefore, in order to ensure that the candidate text recalled by the intervention recall unit is not filtered out, the priority information of the first candidate text can be adjusted before selecting the target text. In this embodiment, the specific implementation method is as follows:

[0096] Determine the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result; determine the third priority information corresponding to the first candidate text and the fourth priority information corresponding to the second candidate text; if the third priority information is greater than a preset priority threshold, adjust the third priority information to the fifth priority information; sort the first candidate text and the second candidate text according to the fourth priority information and the fifth priority information; and select a target text from the first candidate text and the second candidate text according to the sorting result.

[0097] Specifically, the description of the third priority information and the fourth priority information can refer to the description of the first priority information and the second priority information in the above embodiment, and this embodiment will not be repeated here. Correspondingly, the fifth priority information specifically refers to the priority information obtained after adjusting the third priority information.

[0098] Based on this, since the intervention recall unit stores question and answer text data, its recall result accuracy is higher than that of other units, that is, when the distance after the question embedding calculation is high, the probability that the document corresponding to the question contains the answer is higher, so the third priority information can be adjusted to ensure that the text corresponding to the priority is not filtered out, so that the target text containing the first candidate text can be screened out for generating the answer. Therefore, the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit can be determined according to the text retrieval results; then the third priority information corresponding to the first candidate text and the fourth priority information corresponding to the second candidate text are determined; when the third priority information is greater than the preset priority threshold, it means that the question text matches the question related to the question and answer text data in the intervention recall unit, and the third priority information can be adjusted to the fifth priority information; on this basis, the first candidate text and the second candidate text can be sorted according to the fourth priority information and the fifth priority information; thus, the target text can be selected from the first candidate text and the second candidate text according to the sorting result for subsequent use.

[0099] Continuing with the above example, after obtaining candidate documents E1, E2 and E3; B1, B2 and B3; F1, F2 and F3, the FAQ recall unit calculates the vector distance between the embedding vector corresponding to the question text and the embedding vector corresponding to the question in the FAQ when recalling the candidate text, and then screens out the candidate documents, and compares the candidate documents F1, F2 and F3 with the preset threshold to determine that the priority information of each candidate document is greater than the threshold. At this time, after sorting the candidate documents, the sorting result will be {F1, F2, F3, E1, B1, E2, B2, E3, B3}. When K is equal to 5, the TOP5 candidate documents can be selected as the target documents in the sorting result, and the target documents are determined to include {F1, F2, F3, E1, B1}. Finally, each target document and question text are input into the generation module in the question-answering model for processing. According to the processing result, the answer text {hijkl…} corresponding to the question text {What are the regulations of H Company?} can be obtained, and it can be fed back to the user for viewing.

[0100] The text processing method provided in this embodiment can improve the accuracy of the question answering model and reduce the impact of problems in the intermediate links on the output answer. After receiving the question text, the question text can be first input into the retrieval module in the question answering model; and in order to provide accurate candidate texts that match the question in the retrieval stage, the question text can be retrieved by the multi-way recall unit preset by the retrieval module, and the multi-way recall unit is composed of an intervention recall unit and a non-intervention recall unit, and the database of the intervention recall unit pre-stores the question and answer text data; therefore, it can be ensured that when the intervention recall unit performs text recall, the retrieval can be completed by calculating the vector distance between the question text and the candidate question text in the question and answer text data, ensuring that the retrieval is more accurate. On this basis, the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit can be determined according to the text retrieval results; thereafter, the target text can be selected from the first candidate text and the second candidate text, so that when screening the candidate text matching the question text, the priority information of the intervention recall unit and the non-intervention recall unit can be combined, and finally the target text and the answer text can be input into the generation module for processing, so that the answer text corresponding to the question text can be obtained according to the processing results. In the question-answering stage, the recall capability of the retrieval module can be improved through multi-way recall, thereby ensuring that the generation module can output answer text with higher accuracy, thereby improving the user experience.

[0101] See also Figure 3 , Figure 3 A flowchart of another text processing method provided according to an embodiment of the present specification is shown. The method is applied to a server, and the server deploys a question-answering model, wherein the question-answering model includes a retrieval module and a generation module, and specifically includes the following steps.

[0102] Step S302, receiving a question text submitted by a client, and inputting the question text into the search module;

[0103] Step S304, performing text retrieval on the question text by using a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, wherein a database of the intervention recall unit stores question-answer text data in advance, and the intervention recall unit performs text retrieval on the question text by calculating a vector distance between the question text and a candidate question text in the question-answer text data;

[0104] Step S306, selecting a target text from the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text search result;

[0105] Step S308, inputting the target text and the question text into the generation module, obtaining the answer text of the question text, and feeding back the answer text to the client.

[0106] For a detailed description of another text processing method provided in this embodiment, please refer to the corresponding description in the above embodiment, and this embodiment will not be described in detail here.

[0107] Specifically, the server refers to the server that provides question-and-answer services, and the server deploys a question-and-answer model, which is affected by the scope of the question-and-answer services provided by the server, that is, different service types correspond to question-and-answer models with question-and-answer capabilities in different fields. Correspondingly, the client refers to a terminal device that can access the question-and-answer services provided by the server. The client can submit questions through the interface provided by the server, so that the server can answer the question and feedback the answer text to the client.

[0108] Corresponding to the above method embodiment, this specification provides an application of the text processing method in the financial field as an example to further illustrate the text processing method. Specifically, the following steps are included.

[0109] Step 1: receive a question text submitted to the question-answering model, input the question text into an entity extraction model for processing, and obtain semantic entities of the question text.

[0110] Step 2: perform entity mapping on the semantic entity according to a preset vocabulary to obtain a target semantic entity, and input the target semantic entity into the retrieval module.

[0111] Step 3: screening the initial candidate texts associated with the target semantic entity in the retrieval database corresponding to the retrieval module through the retrieval unit in the retrieval module, and constructing a candidate text set according to the screening result.

[0112] Step 4: Perform text retrieval on the question text in the candidate text set through the embedded recall unit preset in the retrieval module.

[0113] Step 5: Perform text retrieval on the question text in the candidate text set through the word frequency recall unit preset in the retrieval module.

[0114] Step 6: Perform text retrieval on the question text in the candidate text set through the automatic recall unit preset in the retrieval module.

[0115] Step 7: Determine the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result.

[0116] It should be noted that step 3, step 4 and step 5 may be performed simultaneously or sequentially, and this embodiment does not impose any limitation thereto.

[0117] Specifically, in the first aspect, the candidate text embedding vector corresponding to the candidate text in the candidate text set is determined; and the second vector distance between the question text embedding vector and the candidate text embedding vector is calculated through the embedding recall unit preset by the retrieval module.

[0118] In a second aspect, the word frequency value between the question text and the candidate texts included in the database corresponding to the word frequency recall unit is calculated through the word frequency recall unit preset in the retrieval module.

[0119] Thirdly, determine the question text embedding vector corresponding to the question text, and the candidate question text embedding vector corresponding to the candidate question text in the question and answer text data pre-stored in the database corresponding to the automatic recall unit; calculate the first vector distance between the question text embedding vector and the candidate question text embedding vector through the automatic recall unit preset by the retrieval module.

[0120] On this basis, the embedded recall candidate text corresponding to the embedded recall unit can be extracted from the candidate text set according to the second vector distance; the word frequency recall candidate text corresponding to the word frequency recall unit can be extracted from the candidate text set according to the word frequency value; the embedded recall candidate text and the word frequency recall candidate text can be used as the second candidate text. At the same time, the target question and answer text data can be determined in the database corresponding to the automatic recall unit according to the first vector distance, and the first candidate text corresponding to the automatic recall unit can be extracted from the target question and answer text data.

[0121] Step 8: Determine the third priority information corresponding to the first candidate text and the fourth priority information corresponding to the second candidate text.

[0122] Step 9: When the third priority information is greater than a preset priority threshold, adjust the third priority information to fifth priority information.

[0123] Step 10: Sort the first candidate text and the second candidate text according to the fourth priority information and the fifth priority information.

[0124] Step 11: Select a target text from the first candidate texts and the second candidate texts according to the sorting result.

[0125] Step 12: input the target text and the question text into the generation module for processing to obtain the answer text corresponding to the question text.

[0126] The text processing method provided in this embodiment can improve the accuracy of the question answering model and reduce the impact of problems in the intermediate links on the output answer. After receiving the question text, the question text can be first input into the retrieval module in the question answering model; and in order to provide accurate candidate texts that match the question in the retrieval stage, the question text can be retrieved by the multi-way recall unit preset by the retrieval module, and the multi-way recall unit is composed of an intervention recall unit and a non-intervention recall unit, and the database of the intervention recall unit pre-stores the question and answer text data; therefore, it can be ensured that when the intervention recall unit performs text recall, the retrieval can be completed by calculating the vector distance between the question text and the candidate question text in the question and answer text data, ensuring that the retrieval is more accurate. On this basis, the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit can be determined according to the text retrieval results; thereafter, the target text can be selected from the first candidate text and the second candidate text, so that when screening the candidate text matching the question text, the priority information of the intervention recall unit and the non-intervention recall unit can be combined, and finally the target text and the answer text can be input into the generation module for processing, so that the answer text corresponding to the question text can be obtained according to the processing results. In the question-answering stage, the recall capability of the retrieval module can be improved through multi-way recall, thereby ensuring that the generation module can output answer text with higher accuracy, thereby improving the user experience.

[0127] Corresponding to the above method embodiment, this specification also provides a text processing device embodiment, Figure 4 FIG. 1 shows a schematic diagram of the structure of a text processing device provided by an embodiment of the present specification. Figure 4 As shown, the device is applied to a question-answering model, wherein the question-answering model includes a retrieval module and a generation module, including:

[0128] A receiving module 402 is configured to receive a question text submitted to the question-answering model and input the question text into the retrieval module;

[0129] A retrieval module 404 is configured to perform text retrieval on the question text through a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, wherein a database of the intervention recall unit stores question-answer text data in advance, and the intervention recall unit performs text retrieval on the question text by calculating a vector distance between the question text and a candidate question text in the question-answer text data;

[0130] The determination module 406 is configured to select a target text from the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result, input the target text and the question text into the generation module, and obtain the answer text of the question text.

[0131] In an optional embodiment, the receiving module 402 is further configured to:

[0132] Input the question text into the entity extraction model for processing to obtain the semantic entity of the question text; perform entity mapping on the semantic entity according to a preset vocabulary to obtain the target semantic entity, and input the target semantic entity into the retrieval module;

[0133] Wherein, the retrieval module 404 is further configured to:

[0134] The retrieval unit in the retrieval module screens the initial candidate texts associated with the target semantic entity in the retrieval database corresponding to the retrieval module, and constructs a candidate text set based on the screening results; and the multi-way recall unit preset in the retrieval module performs text retrieval on the question text in the candidate text set.

[0135] In an optional embodiment, when the intervention recall unit is an automatic recall unit and the non-intervention recall unit is an embedded recall unit and a word frequency recall unit, the retrieval module 404 is further configured as follows:

[0136] Through the embedded recall unit preset in the retrieval module, the question text is searched in the database of the embedded recall unit, and through the word frequency recall unit preset in the retrieval module, the question text is searched in the database of the word frequency recall unit; through the automatic recall unit preset in the retrieval module, the question text is searched in the database of the automatic recall unit.

[0137] In an optional embodiment, the retrieval module 404 is further configured to:

[0138] Determine a question text embedding vector corresponding to the question text, and a candidate question text embedding vector corresponding to the candidate question text in the question and answer text data pre-stored in a database corresponding to the automatic recall unit; calculate a first vector distance between the question text embedding vector and the candidate question text embedding vector through the automatic recall unit preset in the retrieval module;

[0139] Wherein, the determining module 406 is further configured to:

[0140] The target question and answer text data is determined in the database corresponding to the automatic recall unit according to the first vector distance, and the first candidate text corresponding to the automatic recall unit is extracted from the target question and answer text data.

[0141] In an optional embodiment, the retrieval module 404 is further configured to:

[0142] Determine the candidate text embedding vector corresponding to the candidate text in the database corresponding to the embedding recall unit; calculate the second vector distance between the question text embedding vector and the candidate text embedding vector through the embedding recall unit preset by the retrieval module; calculate the word frequency value between the question text and the candidate text contained in the database corresponding to the word frequency recall unit through the word frequency recall unit preset by the retrieval module;

[0143] Wherein, the determining module 406 is further configured to:

[0144] According to the second vector distance, the embedded recall candidate text corresponding to the embedded recall unit is extracted from the database corresponding to the embedded recall unit; according to the word frequency value, the word frequency recall candidate text corresponding to the word frequency recall unit is extracted from the database corresponding to the word frequency recall unit; and the embedded recall candidate text and the word frequency recall candidate text are used as the second candidate text.

[0145] In an optional embodiment, the selection module 408 is further configured to:

[0146] Determine the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result; determine the first priority information corresponding to the first candidate text and the second priority information corresponding to the second candidate text; sort the first candidate text and the second candidate text according to the first priority information and the second priority information; select the target text from the first candidate text and the second candidate text according to the sorting result.

[0147] In an optional embodiment, the selection module 408 is further configured to:

[0148] Determine the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result; determine the third priority information corresponding to the first candidate text and the fourth priority information corresponding to the second candidate text; if the third priority information is greater than a preset priority threshold, adjust the third priority information to the fifth priority information; sort the first candidate text and the second candidate text according to the fourth priority information and the fifth priority information; and select a target text from the first candidate text and the second candidate text according to the sorting result.

[0149] In an optional embodiment, the device further comprises:

[0150] The storage module is configured to determine the question text to be stored that meets the pre-storage conditions corresponding to the automatic recall unit, and determine the answer text to be stored to which the answer text corresponding to the question text to be stored belongs; construct question and answer text data according to the question text to be stored and the answer text to be stored, and store the question and answer text data in the database corresponding to the automatic recall unit.

[0151] The text processing device provided in this embodiment can improve the accuracy of answering questions in the question-answering model and reduce the impact of problems in the intermediate links on the output answer. After receiving the question text, the question text can be first input into the retrieval module in the question-answering model; and in order to provide accurate candidate texts that match the question in the retrieval stage, the question text can be retrieved by the multi-way recall unit preset by the retrieval module, and the multi-way recall unit is composed of an intervention recall unit and a non-intervention recall unit, and the database of the intervention recall unit pre-stores the question-answering text data; therefore, it can be ensured that when the intervention recall unit performs text recall, the retrieval can be completed by calculating the vector distance between the question text and the candidate question text in the question-answering text data, ensuring that the retrieval is more accurate. On this basis, the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit can be determined according to the text retrieval result; thereafter, the target text can be selected from the first candidate text and the second candidate text, so that when screening the candidate text matching the question text, the priority information of the intervention recall unit and the non-intervention recall unit can be combined, and finally the target text and the answer text can be input into the generation module for processing, so that the answer text corresponding to the question text can be obtained according to the processing result. In the question-answering stage, the recall capability of the retrieval module can be improved through multi-way recall, thereby ensuring that the generation module can output answer text with higher accuracy, thereby improving the user experience.

[0152] The above is a schematic scheme of a text processing device of this embodiment. It should be noted that the technical scheme of the text processing device and the technical scheme of the above text processing method belong to the same concept, and the details not described in detail in the technical scheme of the text processing device can be referred to the description of the technical scheme of the above text processing method.

[0153] Corresponding to the above method embodiment, this specification also provides another text processing device embodiment, Figure 5 FIG. 2 shows a schematic diagram of the structure of another text processing device provided by an embodiment of the present specification. Figure 5 As shown, the device is applied to a server, and the server deploys a question-answering model, wherein the question-answering model includes a retrieval module and a generation module, including:

[0154] A text receiving module 502 is configured to receive a question text submitted by a client and input the question text into the search module;

[0155] A text retrieval module 504 is configured to perform text retrieval on the question text through a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, wherein the database of the intervention recall unit stores question-answer text data in advance, and the intervention recall unit performs text retrieval on the question text by calculating a vector distance between the question text and a candidate question text in the question-answer text data;

[0156] A text selection module 506 is configured to select a target text from the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result;

[0157] The feedback text module 508 is configured to input the target text and the question text into the generation module, obtain the answer text of the question text, and feed back the answer text to the client.

[0158] The above is a schematic scheme of another text processing device of this embodiment. It should be noted that the technical scheme of the text processing device and the technical scheme of the above text processing method belong to the same concept, and the details not described in detail in the technical scheme of the text processing device can be referred to the description of the technical scheme of the above text processing method.

[0159] Figure 6 The block diagram of a computing device 600 according to an embodiment of the present specification is shown. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0160] The computing device 600 also includes an access device 640 that enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) that is wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0161] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0162] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 600 may also be a mobile or stationary server.

[0163] The processor 620 is used to execute the following computer executable instructions, which implement the steps of the above-mentioned text processing method when executed by the processor.

[0164] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above text processing method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above text processing method.

[0165] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement the steps of the above-mentioned text processing method when executed by a processor.

[0166] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above text processing method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above text processing method.

[0167] An embodiment of the present specification also provides a computer program product, including a computer program or instructions, which implement the steps of the above-mentioned text processing method when executed by a processor.

[0168] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above text processing method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above text processing method.

[0169] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0170] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0171] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0172] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0173] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can understand and use this specification well.

Claims

1. A text processing method, characterized in that: Applied to a question-answering model, wherein the question-answering model comprises a retrieval module and a generation module, including: Receiving a question text submitted to the question-answering model, and inputting the question text into the retrieval module; Performing a text search on the question text by using a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, wherein a database of the intervention recall unit stores question-answer text data in advance, and the intervention recall unit performs a text search on the question text by calculating a vector distance between the question text and a candidate question text in the question-answer text data; Determine the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result; determine the third priority information corresponding to the first candidate text and the fourth priority information corresponding to the second candidate text; when the third priority information is greater than a preset priority threshold, adjust the third priority information to the fifth priority information, wherein the fifth priority information is used to control the first candidate text not to be filtered; sort the first candidate text and the second candidate text according to the fourth priority information and the fifth priority information; select a target text from the first candidate text and the second candidate text according to the sorting result, input the target text and the question text into the generation module, and obtain an answer text to the question text.

2. The text processing method according to claim 1, characterized in that: The step of inputting the question text into the search module comprises: Inputting the question text into an entity extraction model for processing to obtain semantic entities of the question text; Performing entity mapping on the semantic entity according to a preset vocabulary to obtain a target semantic entity, and inputting the target semantic entity into the retrieval module; The step of performing text retrieval on the question text through the multi-channel recall unit preset in the retrieval module includes: Screening the initial candidate texts associated with the target semantic entity in the retrieval database corresponding to the retrieval module by the retrieval unit in the retrieval module, and constructing a candidate text set according to the screening results; The question text is searched in the candidate text set by a multi-way recall unit preset in the search module.

3. The text processing method according to claim 1, characterized in that: In the case where the intervention recall unit is an automatic recall unit and the non-intervention recall unit is an embedded recall unit and a word frequency recall unit, the text retrieval of the question text by the multi-channel recall unit preset by the retrieval module includes: Performing a text search on the question text in a database of the embedded recall unit through the embedded recall unit preset in the retrieval module, and performing a text search on the question text in a database of the word frequency recall unit through the word frequency recall unit preset in the retrieval module; Through the automatic recall unit preset in the retrieval module, a text search is performed on the question text in the database of the automatic recall unit.

4. The text processing method according to claim 3, characterized in that: The automatic recall unit preset by the retrieval module performs a text search on the question text in the database of the automatic recall unit, including: Determine a question text embedding vector corresponding to the question text, and a candidate question text embedding vector corresponding to a candidate question text in the question-answer text data pre-stored in a database corresponding to the automatic recall unit; Calculating a first vector distance between the question text embedding vector and the candidate question text embedding vector by an automatic recall unit preset in the retrieval module; The determination of the first candidate text recalled by the intervention recall unit includes: The target question and answer text data is determined in the database corresponding to the automatic recall unit according to the first vector distance, and the first candidate text corresponding to the automatic recall unit is extracted from the target question and answer text data.

5. The text processing method according to claim 4, characterized in that: The step of performing a text search on the question text in a database of the embedded recall unit through the embedded recall unit preset by the retrieval module, and performing a text search on the question text in a database of the word frequency recall unit through the word frequency recall unit preset by the retrieval module, comprises: Determine a candidate text embedding vector corresponding to the candidate text in the database corresponding to the embedding recall unit; Calculating a second vector distance between the embedding vector of the question text and the embedding vector of the candidate text by using an embedding recall unit preset in the retrieval module; Calculate the word frequency value between the question text and the candidate texts contained in the database corresponding to the word frequency recall unit through the word frequency recall unit preset in the retrieval module; The determination of the second candidate text recalled by the non-intervention recall unit includes: Extracting, from the database corresponding to the embedding recall unit, embedding recall candidate text corresponding to the embedding recall unit according to the second vector distance; Extracting a word frequency recall candidate text corresponding to the word frequency recall unit from a database corresponding to the word frequency recall unit according to the word frequency value; The embedded recall candidate text and the word frequency recall candidate text are used as the second candidate text.

6. The text processing method according to claim 1, characterized in that: The step of selecting a target text from the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result includes: Determine, according to the text retrieval result, a first candidate text to be recalled by the intervention recall unit and a second candidate text to be recalled by the non-intervention recall unit; Determining first priority information corresponding to the first candidate text, and second priority information corresponding to the second candidate text; sorting the first candidate text and the second candidate text according to the first priority information and the second priority information; A target text is selected from the first candidate texts and the second candidate texts according to the sorting result.

7. The text processing method according to claim 4, characterized in that: Before the step of performing text retrieval on the question text by the multi-way recall unit preset by the retrieval module is performed, the method further includes: Determine the question text to be stored that meets the pre-storage condition corresponding to the automatic recall unit, and determine the answer text to be stored to which the answer text corresponding to the question text to be stored belongs; Question and answer text data are constructed according to the question text to be stored and the answer text to be stored, and the question and answer text data are stored in a database corresponding to the automatic recall unit.

8. A text processing method, characterized in that: Applied to a server, the server deploys a question-answering model, wherein the question-answering model includes a retrieval module and a generation module, including: Receive a question text submitted by a client, and input the question text into the search module; Performing a text search on the question text by using a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, wherein a database of the intervention recall unit stores question-answer text data in advance, and the intervention recall unit performs a text search on the question text by calculating a vector distance between the question text and a candidate question text in the question-answer text data; Determine the first candidate text recalled by the intervention recall unit and the second candidate text recalled by the non-intervention recall unit according to the text retrieval result; determine the third priority information corresponding to the first candidate text and the fourth priority information corresponding to the second candidate text; if the third priority information is greater than a preset priority threshold, adjust the third priority information to the fifth priority information, wherein the fifth priority information is used to control the first candidate text not to be filtered; sort the first candidate text and the second candidate text according to the fourth priority information and the fifth priority information; select the target text from the first candidate text and the second candidate text according to the sorting result; The target text and the question text are input into the generation module to obtain an answer text for the question text, and the answer text is fed back to the client.

9. A text processing device, characterized in that: Applied to a question-answering model, wherein the question-answering model comprises a retrieval module and a generation module, including: A receiving module, configured to receive a question text submitted to the question-answering model, and input the question text into the retrieval module; A retrieval module is configured to perform text retrieval on the question text through a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, wherein a database of the intervention recall unit stores question-answer text data in advance, and the intervention recall unit performs text retrieval on the question text by calculating a vector distance between the question text and a candidate question text in the question-answer text data; A determination module is configured to determine, based on the text retrieval result, a first candidate text recalled by the intervention recall unit and a second candidate text recalled by the non-intervention recall unit; determine third priority information corresponding to the first candidate text and fourth priority information corresponding to the second candidate text; in the case where the third priority information is greater than a preset priority threshold, adjust the third priority information to fifth priority information, wherein the fifth priority information is used to control the first candidate text from being filtered; sort the first candidate text and the second candidate text according to the fourth priority information and the fifth priority information; select a target text from the first candidate text and the second candidate text according to the sorting result, input the target text and the question text into the generation module, and obtain an answer text to the question text.

10. A text processing device, characterized in that: Applied to a server, the server deploys a question-answering model, wherein the question-answering model includes a retrieval module and a generation module, including: A text receiving module is configured to receive a question text submitted by a client and input the question text into the search module; A text retrieval module is configured to perform text retrieval on the question text through a multi-way recall unit preset in the retrieval module, wherein the multi-way recall unit includes an intervention recall unit and a non-intervention recall unit, wherein a database of the intervention recall unit stores question-answer text data in advance, and the intervention recall unit performs text retrieval on the question text by calculating a vector distance between the question text and a candidate question text in the question-answer text data; A text selection module is configured to determine, according to the text retrieval result, a first candidate text recalled by the intervention recall unit and a second candidate text recalled by the non-intervention recall unit; determine third priority information corresponding to the first candidate text and fourth priority information corresponding to the second candidate text; if the third priority information is greater than a preset priority threshold, adjust the third priority information to fifth priority information, wherein the fifth priority information is used to control the first candidate text not to be filtered; sort the first candidate text and the second candidate text according to the fourth priority information and the fifth priority information; and select a target text from the first candidate text and the second candidate text according to the sorting result; The feedback text module is configured to input the target text and the question text into the generation module, obtain the answer text of the question text, and feed back the answer text to the client.

11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method described in any one of claims 1 to 8 are implemented.

12. A computer-readable storage medium, characterized in that: It stores computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of claims 1 to 8.

13. A computer program product, characterized in that The method comprises a computer program or an instruction, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 8.