Text data processing method and device and related equipment

Abstract

The invention provides a text data processing method and device and related equipment, and relates to the technical field of natural language processing, and the method comprises the steps: obtaining a to-be-processed question text; based on a pre-established knowledge document library, querying an evidence text set having an association relationship with the question text; preprocessing the evidence text set, and determining an evidence text corresponding to the question text; and splicing the question text and the evidence text, inputting the spliced question text and evidence text into a pre-trained answer text generation model, and outputting an answer text of the question text. The retrieval efficiency of the evidence text can be improved, and the accuracy of the answer text can be improved.

Description

Background Art

[0002] In the retrieval enhancement generation task, the system needs to retrieve information related to the query question from the knowledge base, called evidence text, and concatenate it with the query question for answer generation. The traditional method divides the knowledge base documents into text blocks of fixed length, and retrieves the Topk (referring to finding the largest top K elements in a large amount of data) text blocks most relevant to the query question as evidence text based on the similarity model. It can be seen that this method relies on the accuracy of the similarity model and the rationality of the boundaries of the text block division.

[0003] Although the above-mentioned retrieval enhancement generation method based on block segmentation and similarity matching can effectively obtain evidence text related to the query question, due to the existence of the text block segmentation operation, the semantic integrity of the retrieved evidence text cannot be guaranteed, and the relatively large amount of redundant content in the evidence text can easily mislead the model to generate biased answers.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The present disclosure provides a text data processing method, apparatus and related equipment, which at least to a certain extent overcome the problems in related technologies that the semantic integrity of retrieved evidence text cannot be guaranteed, and that more redundant content in the evidence text can easily mislead the model to generate biased answers.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a text data processing method is provided, including: obtaining a question text to be processed; based on a pre-established knowledge document library, querying a collection of evidence texts that are associated with the question text; pre-processing the collection of evidence texts to determine the evidence text corresponding to the question text; splicing the question text with the evidence text, inputting them into a pre-trained answer text generation model, and outputting an answer text to the question text.

[0008] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, the evidence text set is preprocessed to determine the evidence text corresponding to the question text, including: performing sentence segmentation on each text in the evidence text set to obtain a sentence set of each text; intercepting the sentences in the sentence set of each text to determine the sentence prefix of each sentence; determining the starting position of each sentence in the evidence text set based on the sentence prefix of each sentence; determining the ending position of each sentence in the evidence text set based on the starting position of each sentence in the evidence text set; and determining the evidence text corresponding to the question text based on the starting position and ending position of each sentence in the evidence text set.

[0009] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, based on the starting position of each sentence in the evidence text set, the ending position of each sentence in the evidence text set is determined, including: intercepting each sentence in the sentence set of each text to determine the sentence prefix corresponding to each sentence in the sentence set of each text; forming a prefix vocabulary of each text according to the sentence prefix corresponding to each sentence in the sentence set of each text; splicing the question text with each text in the evidence text set to obtain the splicing information corresponding to each text; using the prefix vocabulary as a generation dictionary, based on the target prefix generation algorithm, generating a sentence prefix set according to the splicing information; and determining the starting position of each sentence in the evidence text set in the sentence prefix set.

[0010] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, the evidence text corresponding to the question text is determined based on the starting position and the ending position of each sentence in the evidence text set, including: performing de-duplication and merging processing on the starting position and the ending position of each sentence in the evidence text set to determine the evidence text corresponding to the question text.

[0011] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme and based on a pre-established knowledge document library, a set of evidence texts that are associated with the question text is queried, including: performing word segmentation processing on the question text to generate question keywords corresponding to the question text; performing vectorization processing on the question keywords to obtain corresponding question vectors; calculating the similarity between the question vector and each document vector in the pre-established knowledge document library; and determining a set of evidence texts that are associated with the question text based on the similarity between the question vector and each document vector.

[0012] In some exemplary embodiments of the present disclosure, based on the aforementioned scheme, before determining the starting position of each sentence in the evidence text set in the sentence prefix set, the method further includes: encoding the splicing information corresponding to each text.

[0013] According to another aspect of the present disclosure, a text data processing device is also provided, including: a question text acquisition module, used to acquire the question text to be processed; an evidence text query module, used to query a collection of evidence texts that have an association relationship with the question text based on a pre-established knowledge document library; an evidence text determination module, used to pre-process the evidence text collection and determine the evidence text corresponding to the question text; and an answer text output module, used to splice the question text with the evidence text, input them into a pre-trained answer text generation model, and output the answer text of the question text.

[0014] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned text data processing methods by executing the executable instructions.

[0015] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the above-mentioned text data processing methods is implemented.

[0016] According to another aspect of the present disclosure, a computer program product is provided, including: a computer program or instructions, wherein when the computer program or instructions are executed by a processor, any one of the above-mentioned text data processing methods is implemented.

[0017] A text data processing method, apparatus and related equipment provided in the embodiments of the present disclosure use a pre-established knowledge document library to query a collection of evidence texts that are associated with a question text, and then pre-process the collection of evidence texts, thereby getting rid of the traditional operation of dividing the evidence texts in the knowledge document library into blocks, locating the evidence text corresponding to the question text in the complete evidence text collection, avoiding the hallucination problem, and effectively improving the retrieval efficiency of the evidence text, ultimately achieving an improvement in the effect of the retrieval enhancement generation system.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0020] Figure 1 A schematic diagram showing an exemplary application system architecture of a text data processing method in an embodiment of the present disclosure;

[0021] Figure 2 A schematic diagram of a text data processing method in an embodiment of the present disclosure is shown;

[0022] Figure 3 A schematic diagram showing a starting position determination corresponding to an evidence text in an embodiment of the present disclosure is shown;

[0023] Figure 4 A schematic diagram showing a method for determining an end position corresponding to an evidence text in an embodiment of the present disclosure is shown;

[0024] Figure 5 A schematic diagram of a process flow of an application text data processing method in an embodiment of the present disclosure is shown;

[0025] Figure 6 A schematic diagram of a text data processing device in an embodiment of the present disclosure is shown;

[0026] Figure 7 A schematic diagram of an electronic device to which a text data processing method is applied in an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0029] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0030] Figure 1FIG. 2 shows an exemplary application system architecture diagram to which the text data processing method in the embodiment of the present disclosure can be applied. Figure 1 As shown, the system architecture may include a terminal device 101 , a network 102 and a server 103 .

[0031] The network 102 is a medium for providing a communication link between the terminal device 101 and the server 103, and can be a wired network or a wireless network.

[0032] Optionally, the wireless network or wired network described above uses standard communication technology and / or protocol. The network is usually the Internet, but it can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network or any combination of a virtual private network). In some embodiments, technologies and / or formats including Hyper Text Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec) can also be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can also be used to replace or supplement the above data communication technologies.

[0033] The terminal device 101 can be various electronic devices, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, wearable devices, augmented reality devices, virtual reality devices, etc.

[0034] Optionally, the client of the application installed in different terminal devices 101 is the same, or the client of the same type of application based on different operating systems. Based on the different terminal platforms, the specific form of the client of the application can also be different, for example, the application client can be a mobile client, a PC client, etc.

[0035] The server 103 may be a server that provides various services, such as a background management server that provides support for the device operated by the user using the terminal device 301. The background management server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0036] Optionally, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0037] Those skilled in the art will know that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration, and any number of terminal devices, networks and servers may be provided according to actual needs, and the embodiments of the present disclosure do not limit this.

[0038] Under the above system architecture, a text data processing method is provided in an embodiment of the present disclosure, and the method can be executed by any electronic device with computing and processing capabilities.

[0039] In some embodiments, the text data processing method provided in the embodiments of the present disclosure can be executed by a terminal device of the above-mentioned system architecture; in other embodiments, the text data processing method provided in the embodiments of the present disclosure can be executed by a server in the above-mentioned system architecture; in other embodiments, the text data processing method provided in the embodiments of the present disclosure can be implemented by the terminal device and the server in the above-mentioned system architecture through interaction.

[0040] First, in response to the above-mentioned problems, a control method for text data processing is provided in an embodiment of the present disclosure, which can be applied to any question-answering system. Compared with the retrieval enhancement generation method based on segmentation and similarity matching in the related art, the semantic integrity of the retrieved evidence text cannot be guaranteed, and the relatively large amount of redundant content in the evidence text can easily mislead the model to generate biased answers. The embodiment of the present disclosure uses a pre-established knowledge document library to query a collection of evidence texts that has an association with the question text, and then pre-processes the collection of evidence texts, thereby getting rid of the traditional operation of segmenting the evidence texts in the knowledge document library, locating the evidence text corresponding to the question text in the complete evidence text collection, avoiding the hallucination problem, and effectively improving the retrieval efficiency of the evidence text, thereby ultimately achieving an improvement in the effect of the retrieval enhancement generation system.

[0041] Figure 2 A schematic diagram of a text data processing method in an embodiment of the present disclosure is shown, and the method includes the following steps:

[0042] S202, obtaining the question text to be processed.

[0043] It should be noted that the question text to be processed in the embodiments of the present disclosure may be any question text that needs to be processed. In some embodiments, it may be a question asked by the target user on various search platforms. For example, the question text may be a question asked by the target user, such as “Who is the winner of the Nobel Prize in Physics in 2023?”

[0044] S204, based on the pre-established knowledge document library, query the evidence text collection that has a correlation relationship with the question text.

[0045] It should be noted that the knowledge document library in the disclosed embodiment includes evidence text that may answer the question text; the evidence text collection contains multiple evidence texts that answer the reference information of the question text; the evidence text is a text fragment extracted from a large number of documents or knowledge document libraries, which can provide specific information to answer the question text. In the question-and-answer system, the evidence text is used to support the system's answer and ensure the accuracy and reliability of the answer. For example, when the target user asks "Who is the winner of the Nobel Prize in Physics in 2023", the question-and-answer system finds the relevant paragraphs from the knowledge document library and extracts the evidence text "The winners of the Nobel Prize in Physics in 2023 are A, B and C".

[0046] S206, pre-processing the evidence text set to determine the evidence text corresponding to the question text.

[0047] It should be noted that the embodiment of the present disclosure preprocesses the evidence text set to locate the evidence text corresponding to the question text in the complete evidence text set. More specifically, the embodiment of the present disclosure uses a target prefix generation algorithm to determine the starting position of the evidence text, and uses a jump decoding algorithm to determine the ending position of the evidence text. The target prefix generation algorithm and the jump decoding algorithm can flexibly select the starting position and the ending position of the evidence text corresponding to the question text in a long evidence text set, thereby realizing the extraction of evidence text without text block division, so that the extracted evidence text is more relevant to the question text and has less redundant information.

[0048] S208, concatenating the question text and the evidence text, inputting them into a pre-trained answer text generation model, and outputting the answer text of the question text.

[0049] The text data processing method provided in the embodiment of the present disclosure first obtains the question text to be processed; secondly, based on the pre-established knowledge document library, queries the evidence text set that has an association relationship with the question text; then, pre-processes the evidence text set to determine the evidence text corresponding to the question text; finally, splices the question text with the evidence text, inputs it into the pre-trained answer text generation model, and outputs the answer text of the question text. Compared with the retrieval enhancement generation method based on block division and similarity matching in the related art, due to the existence of the text block division operation, the semantic integrity of the evidence text retrieved cannot be guaranteed, and the more redundant content in the evidence text is easy to mislead the model to generate biased answers. The embodiment of the present disclosure uses the pre-established knowledge document library to query the evidence text set that has an association relationship with the question text, and then pre-processes the evidence text set, getting rid of the traditional operation of dividing the evidence text in the knowledge document library, locating the evidence text corresponding to the question text in the complete evidence text set, avoiding the hallucination problem, and effectively improving the retrieval efficiency of the evidence text, and finally achieving the improvement of the effect of the retrieval enhancement generation system.

[0050] In some embodiments, the disclosed embodiments preprocess the evidence text set to determine the evidence text corresponding to the question text, including: performing sentence processing on each text in the evidence text set to obtain a sentence set of each text; intercepting the sentences in the sentence set of each text to determine the sentence prefix of each sentence; determining the starting position of each sentence in the evidence text set based on the sentence prefix of each sentence; determining the ending position of each sentence in the evidence text set based on the starting position of each sentence in the evidence text set; determining the evidence text corresponding to the question text based on the starting position and ending position of each sentence in the evidence text set. Specifically, the disclosed embodiments determine the starting position by the sentence prefix of each sentence, which can accurately find the text paragraph containing the required information and avoid the interference of redundant information, and determine the ending position of each sentence in the evidence text set based on the starting position of each sentence in the evidence text set, which can ensure that the extracted evidence text is complete and relevant, avoid missing key information or including irrelevant content, and further, in large documents or knowledge document libraries, directly locating relevant paragraphs can significantly reduce the consumption of search time and computing resources.

[0051] In some embodiments, the disclosed embodiments screen a set of unsegmented evidence texts from a pre-established knowledge document library based on the question text, concatenate the question text to be queried and the screened unsegmented evidence text set into the input text of a pre-trained encoding large model, and input the input question text into the encoding large model to obtain the hidden state encoding of the input text.

[0052] In some embodiments, the disclosed embodiments perform sentence processing on each evidence text in the evidence text set, and determine a sentence prefix of each sentence that can uniquely identify the sentence, and construct a prefix vocabulary based on the determined sentence prefix content.

[0053] In some embodiments, Figure 3 As shown, the embodiment of the present disclosure starts to search for the starting position of the evidence text in the evidence text set through the target prefix generation algorithm. The specific process includes:

[0054] S302, initialize an empty candidate generation sequence.

[0055] S304, based on the hidden state encoding corresponding to the input text of the encoding large model, for each current candidate generation sequence, select the top k words ranked from the prefix word library in descending probability as the next word to generate, and obtain new top k candidate generation sequences.

[0056] S306, looping through S304. During the process, when a candidate generated sequence can uniquely identify a sentence in the document, the generation of the sequence is stopped, and the sentence is defined as the starting position of the evidence text. When all candidate sequences are stopped from being generated, the loop ends, and the starting position of the evidence text corresponding to the Topk candidate sequences with the highest average generation probability is selected.

[0057] In some embodiments, the embodiments of the present disclosure determine the ending position of each sentence in the evidence text set based on the starting position of each sentence in the evidence text set, including: intercepting each sentence in the sentence set of each text, and determining the sentence prefix corresponding to each sentence in the sentence set of each text; forming a prefix vocabulary of each text according to the sentence prefix corresponding to each sentence in the sentence set of each text; splicing the question text with each text in the evidence text set to obtain the splicing information corresponding to each text; using the prefix vocabulary as a generated dictionary, based on the target prefix generation algorithm, generating a sentence prefix set according to the splicing information; determining the starting position of each sentence in the evidence text set in the sentence prefix set. Specifically, the embodiments of the present disclosure use the target prefix generation algorithm to find the starting position corresponding to the evidence text, which can greatly improve the generation speed of the evidence text and enhance the system's retrieval ability for the evidence text.

[0058] In some embodiments, Figure 4 As shown, the embodiment of the present disclosure uses a jump decoding algorithm to find the corresponding end position of the evidence text at the starting position of each sentence in the evidence text set determined in the above technical solution. The specific process includes:

[0059] S402, concatenating the sentence at the starting position with multiple sentences with a subsequent preset threshold value as input data for a large decoding model.

[0060] S404: For each concatenated input data, record the probability that the next generated word-gram is the end word-gram.

[0061] S406, comparing the probabilities of different concatenated subsequent generated end word units, and selecting the end unit with the highest probability as the end position of the evidence text.

[0062] In some embodiments, the disclosed embodiments consider multiple sentences with a preset threshold backward from the starting position of each evidence text, calculate the probability that the next character after the decoding large model generates the text is the end word, and use it as the generation probability of the text. The end word with the largest probability is selected as the end position to determine the scope of the evidence text. It can be seen that the disclosed embodiments use a jump decoding algorithm to find the corresponding end position of the evidence text, which can greatly improve the generation speed of the evidence text and enhance the system's retrieval ability for the evidence text.

[0063] In some embodiments, the disclosed embodiments determine the evidence text corresponding to the question text based on the starting position and the ending position of each sentence in the evidence text set, including: performing deduplication and merging processing on the starting position and the ending position of each sentence in the evidence text set to determine the evidence text corresponding to the question text. Specifically, after obtaining multiple pieces of evidence text, if there is overlap between the evidence texts, they are deduplicated and spliced ​​to obtain the evidence text retrieval result for the question text queried by the target user, thereby realizing evidence text extraction without text block division, so that the extracted evidence text fragments are more relevant to the queried question text and have less redundant information.

[0064] In some embodiments, the disclosed embodiments segment each text in the evidence text set into sentences. For each text, the sentence set is segmented from the front end to the back until a sentence prefix that can uniquely identify the sentence in each text is obtained. The sentence prefixes corresponding to all sentences constitute a prefix set for each text. Each prefix in the prefix set is segmented to form a prefix vocabulary for each text. The prefix vocabulary is used as a generated dictionary, and a target prefix generation algorithm is used to generate the Topk sentence prefixes with the highest probability, which constitute the starting position set of the evidence text of the document.

[0065] In some embodiments, the disclosed embodiments use a jump decoding algorithm to determine an end position with the highest probability of an evidence text for each starting position in the starting position set, that is, a piece of evidence text uniquely determined in the evidence text set for each set of starting positions and end positions, and all evidence texts constitute a candidate evidence set retrieved from the document. The evidence texts in the candidate evidence set are de-duplicated and merged; each piece of evidence text in the evidence text set obtained by each text in the evidence text set is spliced ​​to obtain a total piece of evidence text.

[0066] In some embodiments, the disclosed embodiments query a collection of evidence texts that have an association relationship with the question text based on a pre-established knowledge document library, including: performing word segmentation processing on the question text to generate question keywords corresponding to the question text; performing vectorization processing on the question keywords to obtain corresponding question vectors; calculating the similarity between the question vector and each document vector in the pre-established knowledge document library; and determining a collection of evidence texts that have an association relationship with the question text based on the similarity between the question vector and each document vector. Specifically, the disclosed embodiments can efficiently find the text that is most relevant to the question text from the knowledge document library by performing operations such as word segmentation processing, vectorization processing, and similarity calculation on the question text, thereby providing an accurate and useful collection of evidence texts.

[0067] In some embodiments, the disclosed embodiment determines that each sentence in the sentence prefix set is before the starting position of the evidence text set, and the text data processing method in the disclosed embodiment further includes: encoding the splicing information corresponding to each text. Specifically, the disclosed embodiment splices the question text queried by the target user with each text in the evidence text set, passes the splicing information through an encoder to obtain the information encoding corresponding to each text, and provides the required input information encoding for the subsequent determination of the starting position of each sentence in the evidence text set.

[0068] In some embodiments, Figure 5 As shown, the text data processing method in the embodiment of the present disclosure is applied, and the specific process includes:

[0069] S502, receiving a question text input by a target user to be queried.

[0070] S504, input the question text to be queried into the retriever, and obtain a set of unblocked evidence texts containing possible answers to the question texts through similarity matching retrieval.

[0071] S506, using a target prefix generation algorithm and a jump decoding algorithm for each text in the evidence text set to determine the evidence text corresponding to each text in the evidence text set.

[0072] S508, removing duplicates and splicing the evidence text corresponding to each text to obtain the evidence text corresponding to the question text.

[0073] S510, the question text to be queried and the evidence text corresponding to the question text are spliced ​​and input into the big model for answer generation, thereby enhancing the answer generation capability of the big model.

[0074] In some embodiments, the disclosed embodiments use a target prefix generation algorithm and a jump decoding algorithm to flexibly select the starting and ending positions of the evidence text corresponding to the question text in a long piece of evidence text, thereby realizing evidence text extraction without the need for text block division, so that the extracted evidence text has a higher relevance to the question text to be queried and has less redundant information.

[0075] In some embodiments, the embodiments of the present disclosure avoid the semantic loss or redundancy problem of evidence text caused by block operation by converting the retrieval task into a positioning task based on generation probability. In addition, the embodiments of the present disclosure design a target prefix generation algorithm to determine the starting position of the evidence text, and quickly determine the ending position of the evidence text through a jump decoding algorithm, thereby ultimately enhancing the system's retrieval capability for the evidence text and improving the effect of the retrieval enhancement generation system.

[0076] Based on the same inventive concept, the present disclosure also provides a text data processing device in the following embodiments. Since the principle of solving the problem in the device embodiment is similar to that in the above method embodiment, the implementation of the device embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0077] Figure 6 A schematic diagram of a text data processing device in an embodiment of the present disclosure is shown, and the device includes:

[0078] The question text acquisition module 601 is used to acquire the question text to be processed;

[0079] An evidence text query module 602 is used to query a set of evidence texts associated with the question text based on a pre-established knowledge document library;

[0080] The evidence text determination module 603 is used to pre-process the evidence text set and determine the evidence text corresponding to the question text;

[0081] The answer text output module 604 is used to splice the question text with the evidence text, input them into a pre-trained answer text generation model, and output the answer text of the question text.

[0082] In the embodiment of the present disclosure, a text data processing device is provided, which obtains the question text to be processed through the question text acquisition module; through the evidence text query module, based on the pre-established knowledge document library, queries the evidence text set that has an association relationship with the question text; through the evidence text determination module, pre-processes the evidence text set to determine the evidence text corresponding to the question text; through the answer text output module, splices the question text with the evidence text, inputs it into the pre-trained answer text generation model, and outputs the answer text of the question text. Compared with the retrieval enhancement generation method based on block division and similarity matching in the related art, due to the existence of the text block division operation, the semantic integrity of the evidence text retrieved cannot be guaranteed, and the more redundant content in the evidence text is easy to mislead the model to generate biased answers. In the embodiment of the present disclosure, through the pre-established knowledge document library, queries the evidence text set that has an association relationship with the question text, and then pre-processes the evidence text set, gets rid of the traditional operation of dividing the evidence text in the knowledge document library, locates the evidence text corresponding to the question text in the complete evidence text set, avoids the hallucination problem, and effectively improves the retrieval efficiency of the evidence text, and finally achieves the improvement of the effect of the retrieval enhancement generation system.

[0083] In some embodiments, the evidence text determination module in the embodiments of the present disclosure is also used to perform sentence processing on each text in the evidence text set to obtain a sentence set for each text; to intercept the sentences in the sentence set of each text to determine the sentence prefix of each sentence; to determine the starting position of each sentence in the evidence text set based on the sentence prefix of each sentence; to determine the ending position of each sentence in the evidence text set based on the starting position of each sentence in the evidence text set; and to determine the evidence text corresponding to the question text based on the starting position and ending position of each sentence in the evidence text set.

[0084] In some embodiments, the evidence text determination module in the embodiments of the present disclosure is also used to intercept each sentence in the sentence set of each text, determine the sentence prefix corresponding to each sentence in the sentence set of each text; form a prefix vocabulary of each text according to the sentence prefix corresponding to each sentence in the sentence set of each text; splice the question text with each text in the evidence text set to obtain the splicing information corresponding to each text; use the prefix vocabulary as a generation dictionary, and generate a sentence prefix set based on the target prefix generation algorithm according to the splicing information; determine the starting position of each sentence in the evidence text set in the sentence prefix set.

[0085] In some embodiments, the evidence text determination module in the embodiments of the present disclosure is also used to perform de-duplication and merging processing on the starting position and the ending position of each sentence in the evidence text set to determine the evidence text corresponding to the question text.

[0086] In some embodiments, the evidence text query module in the embodiments of the present disclosure is also used to perform word segmentation on the question text to generate question keywords corresponding to the question text; vectorize the question keywords to obtain corresponding question vectors; calculate the similarity between the question vector and each document vector in a pre-established knowledge document library; and determine a set of evidence texts that have an association with the question text based on the similarity between the question vector and each document vector.

[0087] In some embodiments, the text data processing device in the embodiments of the present disclosure further includes: an encoding module, which is used to determine each sentence in the sentence prefix set before the starting position of the evidence text set in the evidence text set, and perform encoding operations on the splicing information corresponding to each text.

[0088] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0089] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present disclosure, the electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned text data processing methods by executing the executable instructions. Since the principle of solving the problem in the electronic device embodiment is similar to that in the above-mentioned method embodiment, the implementation of the electronic device embodiment can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0090] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present disclosure is described. Figure 7 The electronic device 700 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0091] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include but are not limited to: at least one processing unit 701, at least one storage unit 702, and a bus 703 connecting different system components (including the storage unit 702 and the processing unit 701).

[0092] The storage unit stores program codes, which can be executed by the processing unit 701, so that the processing unit 701 executes the steps described in the above “exemplary method” section of this specification according to various exemplary embodiments of the present disclosure.

[0093] In some embodiments, when the electronic device is used to control the text data processing method disclosed above, for example, the processing unit 701 may perform the following steps of the above method embodiment:

[0094] Obtain the question text to be processed; based on a pre-established knowledge document library, query the evidence text collection that is associated with the question text; pre-process the evidence text collection to determine the evidence text corresponding to the question text; concatenate the question text and the evidence text, input them into a pre-trained answer text generation model, and output the answer text to the question text.

[0095] The storage unit 702 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7021 and / or a cache storage unit 7022 , and may further include a read-only storage unit (ROM) 7023 .

[0096] The storage unit 702 may also include a program / utility 7024 having a set (at least one) of program modules 7025, such program modules 7025 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0097] Bus 703 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0098] The electronic device 700 may also communicate with one or more external devices 704 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or may communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). This communication may be performed through an input / output (I / O) interface 705. In addition, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter 706. As shown, the network adapter 706 communicates with other modules of the electronic device 700 through a bus 703. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0099] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0100] Based on the same inventive concept, the embodiment of the present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned text data processing methods is implemented. Since the principle of solving the problem in the embodiment of the computer-readable storage medium is similar to that in the above-mentioned method embodiment, the implementation of the embodiment of the computer-readable storage medium can refer to the implementation of the above-mentioned method embodiment, and the repeated parts will not be repeated.

[0101] More specific examples of computer-readable storage media in the present disclosure may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0102] In the present disclosure, a computer readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0103] Alternatively, the program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0104] In a specific implementation, the program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0105] Based on the same inventive concept, a computer program product is also provided in the embodiments of the present disclosure, including a computer program product, including: a computer program or an instruction, and when the computer program or the instruction is executed by a processor, the text data processing method of any one of the above method embodiments is implemented. Since the principle of solving the problem in the computer program product embodiment is similar to that in the above method embodiment, the implementation of the computer program product embodiment can refer to the implementation of the above method embodiment, and the repeated parts will not be repeated.

[0106] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0107] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0108] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0109] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the appended claims.

Claims

1. A text data processing method, characterized in that: include: Get the question text to be processed; Based on a pre-established knowledge document library, query a collection of evidence texts that are associated with the question text; Preprocessing the evidence text set to determine the evidence text corresponding to the question text; The question text and the evidence text are concatenated and input into a pre-trained answer text generation model to output the answer text of the question text.

2. The text data processing method according to claim 1, characterized in that: Preprocessing the evidence text set to determine the evidence text corresponding to the question text includes: Perform sentence processing on each text in the evidence text set to obtain a sentence set of each text; Sentences in the sentence set of each text are intercepted to determine the sentence prefix of each sentence; Determine the starting position of each sentence in the evidence text set according to the sentence prefix of each sentence; Determining the ending position of each sentence in the evidence text set based on the starting position of each sentence in the evidence text set; Based on the starting position and the ending position of each sentence in the evidence text set, the evidence text corresponding to the question text is determined.

3. The text data processing method according to claim 2, characterized in that: Determining the ending position of each sentence in the evidence text set based on the starting position of each sentence in the evidence text set includes: Each sentence in the sentence set of each text is intercepted to determine the sentence prefix corresponding to each sentence in the sentence set of each text; According to the sentence prefix corresponding to each sentence in the sentence set of each text, a prefix vocabulary of each text is formed; Splicing the question text with each text in the evidence text set to obtain splicing information corresponding to each text; Using the prefix word library as a generation dictionary, based on a target prefix generation algorithm, generating a sentence prefix set according to the splicing information; The starting position of each sentence in the evidence text set is determined in the sentence prefix set.

4. The text data processing method according to claim 2, characterized in that: Determining the evidence text corresponding to the question text based on the starting position and the ending position of each sentence in the evidence text set includes: The starting position and the ending position of each sentence in the evidence text set are de-duplicated and merged to determine the evidence text corresponding to the question text.

5. The text data processing method according to claim 1, characterized in that: Based on the pre-established knowledge document library, a collection of evidence texts associated with the question text is queried, including: Performing word segmentation processing on the question text to generate question keywords corresponding to the question text; Vectorize the question keywords to obtain corresponding question vectors; Calculating the similarity between the question vector and each document vector in a pre-established knowledge document library; According to the similarity between the question vector and each document vector, a set of evidence texts associated with the question text is determined.

6. The text data processing method according to claim 3, characterized in that: Determining in the sentence prefix set that each sentence is before the starting position of the evidence text set, the method further includes: Encode the splicing information corresponding to each text.

7. A text data processing device, characterized in that: include: A question text acquisition module is used to acquire the question text to be processed; An evidence text query module, used to query a collection of evidence texts associated with the question text based on a pre-established knowledge document library; An evidence text determination module, used to pre-process the evidence text set to determine the evidence text corresponding to the question text; The answer text output module is used to splice the question text with the evidence text, input them into a pre-trained answer text generation model, and output the answer text of the question text.

8. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to execute the text data processing method according to any one of claims 1 to 6 by executing the executable instructions.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the text data processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising: A computer program or instruction, characterized in that when the computer program or instruction is executed by a processor, it implements the text data processing method described in any one of claims 1 to 6.

Images