Question and answer pair generation method, electronic equipment, storage medium and computer program product

Abstract

The invention provides a question and answer pair generation method, electronic equipment, a storage medium and a computer program product, and the method comprises the steps: dividing a to-be-processed text in a target knowledge base to obtain a plurality of paragraphs; for each paragraph, extracting a plurality of keywords for the paragraph; constructing a first input parameter for each paragraph and the plurality of keywords corresponding to the paragraph, and processing the first input parameter through a specified large model to obtain a question corresponding to the paragraph; according to the plurality of keywords corresponding to each paragraph, recalling a plurality of related paragraphs corresponding to the plurality of keywords from the plurality of paragraphs through approximate nearest neighbor search; based on a plurality of related paragraphs corresponding to each paragraph, respectively constructing a second input parameter for each question of the paragraph, and processing the second input parameter through a specified large model to obtain an answer corresponding to the question; and constructing each question corresponding to the to-be-processed text and the answer corresponding to the question into a question-answer pair. According to the scheme, high-quality question and answer pairs can be generated.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a method and device for generating question-answer pairs, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the rapid development of artificial intelligence technology, building intelligent question-answering systems has become a hot topic in research and application. Intelligent question-answering systems can understand the user's query intent, retrieve or generate accurate answers from the knowledge base, and provide users with efficient and convenient information services. In order to build a highly accurate intelligent question-answering system, it is necessary to prepare high-quality question-answer pairs in several fields. How to generate high-quality question-answer pairs has become a problem that needs to be solved urgently. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a method for generating question-answer pairs and an electronic device, a storage medium, and a computer program product for generating high-quality question-answer pairs.

[0004] On the one hand, the present application provides a method for generating a question-answer pair, comprising:

[0005] For the text to be processed in the target knowledge base, divide it according to the preset division strategy to obtain multiple paragraphs;

[0006] For each paragraph, extract several keywords based on the preset extraction strategy;

[0007] Constructing a first input parameter for each paragraph and its corresponding keywords, and processing the first input parameter by specifying a large model to obtain questions corresponding to the paragraph; wherein the first input parameter indicates generating questions related to the keywords for the paragraph;

[0008] According to the keywords corresponding to each paragraph, recalling the relevant paragraphs corresponding to the keywords from the multiple paragraphs through approximate nearest neighbor search;

[0009] Based on a number of related paragraphs corresponding to each paragraph, construct a second input parameter for each question of the paragraph, and process the second input parameter through the specified large model to obtain an answer corresponding to the question; wherein the second input parameter indicates generating an answer for the question;

[0010] Each question corresponding to the text to be processed and the answer corresponding to the question are constructed as a question-answer pair to obtain multiple question-answer pairs.

[0011] In one embodiment, the text to be processed in the target knowledge base is divided according to a preset division strategy to obtain multiple paragraphs, including:

[0012] Segmenting the text to be processed according to text symbols to obtain multiple text units;

[0013] Perform vectorization on each text unit to obtain a unit vector;

[0014] Perform an approximate nearest neighbor search on the unit vector of each text unit in the logical word vector library, and determine a similarity score based on the similarity between the unit vector and the most similar logical word vector;

[0015] The similarity score of each text unit is compared with the similarity score of the adjacent text unit. If the difference in the similarity scores is less than a preset difference threshold, the text unit is merged with the text unit of the vector, and the multiple text units are merged to obtain multiple paragraphs.

[0016] In one embodiment, before performing an approximate nearest neighbor search on the unit vector of each text unit in the logical word vector library, the method further includes:

[0017] Perform vectorization processing on each logical word in the preset logical word list to obtain the logical word vector corresponding to the logical word;

[0018] Build a logical word vector library based on multiple logical word vectors.

[0019] In one embodiment, for each paragraph, a number of keywords are extracted based on a preset extraction strategy, including:

[0020] For each paragraph, perform word segmentation processing on the paragraph to obtain multiple word segmentation results;

[0021] Specific characters and stop words are filtered out from the multiple word segmentation results, and the remaining word segmentation results are used as keywords for the paragraph; wherein the specific characters are characters that are not Chinese characters, English letters, numbers, or blank characters.

[0022] In one embodiment, for each paragraph, a number of keywords are extracted based on a preset extraction strategy, including:

[0023] For each paragraph, each logical word in the preset logical word list is matched in the paragraph. If any logical word is matched in the paragraph, it is expanded according to the position index of the logical word in the paragraph to obtain the text interval corresponding to the logical word;

[0024] Perform word segmentation processing on the text in each text interval to obtain multiple word segmentation results corresponding to the text interval;

[0025] Filter out specific characters and stop words from the multiple word segmentation results of each text interval, and use the remaining word segmentation results as keywords of the text interval; wherein the specific characters are characters that are not Chinese characters, English letters, numbers, or blank characters;

[0026] The keywords in several text intervals in each paragraph are deduplicated to obtain the keywords of the paragraph.

[0027] In one embodiment, constructing the first input parameter for each paragraph and its corresponding keywords includes:

[0028] Each paragraph and its corresponding keywords are respectively filled into a preset first prompt word template to obtain a first input parameter; wherein the first prompt word template indicates a generated question.

[0029] In one embodiment, the second input parameter is constructed for each question of each paragraph based on a number of related paragraphs corresponding to each paragraph, including:

[0030] For each question in the paragraph, the question, several keywords corresponding to the paragraph, several related paragraphs corresponding to the keywords, and the full-text summary of the text to be processed are filled into a preset second prompt word template to obtain a second input parameter; wherein the second prompt word template indicates the generation of an answer.

[0031] In one embodiment, the step of obtaining the full-text abstract of the text to be processed includes:

[0032] The text to be processed and the summary prompt word are submitted to the designated large model to obtain a full-text summary output by the designated large model; wherein the summary prompt word indicates the generation of a summary.

[0033] On the other hand, the present application provides an electronic device, the electronic device comprising:

[0034] processor;

[0035] a memory for storing processor-executable instructions;

[0036] Wherein, the processor is configured to execute the above-mentioned question-answer pair generation method.

[0037] Furthermore, the present application provides a computer-readable storage medium, which stores a computer program, and the computer program can be executed by a processor to complete the above-mentioned method for generating question-answer pairs.

[0038] In addition, the present application provides a computer program product, including a computer program / instruction, which implements the above-mentioned method for generating question-answer pairs when executed by a processor.

[0039] This application solution can effectively pre-process multiple texts in the target knowledge base, and then use the specified large model to effectively generate questions and answers, thereby constructing high-quality question-answer pairs for the target knowledge field. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solution of the embodiments of the present application, the drawings required for use in the embodiments of the present application are briefly introduced below.

[0041] Figure 1 A schematic diagram of an application scenario of a method for generating question-answer pairs provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application;

[0043] Figure 3 A flowchart of a method for generating question-answer pairs provided in an embodiment of the present application;

[0044] Figure 4 An embodiment of the present application provides Figure 3 Detailed flow chart of step 320;

[0045] Figure 5 A schematic diagram of a flow chart of a paragraph division method provided in an embodiment of the present application;

[0046] Figure 6 A block diagram of a device for generating question and answer pairs provided in one embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0048] Similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0049] Figure 1 Schematic diagram of an application scenario of the method for generating question-answer pairs provided in an embodiment of the present application. Figure 1 As shown, the application scenario includes a client 20 and a server 30; the client 20 can be a host, a server, a server cluster or a cloud computing center, which is used to execute the task of generating question and answer pairs by calling a specified large model carried on the server 30; the server 30 can be a server, a server cluster or a cloud computing center, which can respond to the call of the client 20 and assist in executing the task of generating question and answer pairs.

[0050] like Figure 2 As shown, this embodiment provides an electronic device 1, including: at least one processor 11 and a memory 12, Figure 2 A processor 11 is taken as an example. The processor 11 and the memory 12 are connected via a bus 10. The memory 12 stores instructions that can be executed by the processor 11. The instructions are executed by the processor 11 so that the electronic device 1 can execute all or part of the process of the method in the following embodiment. In one embodiment, the electronic device 1 can be the above-mentioned client 20, which is used to execute the method for generating question-answer pairs.

[0051] The memory 12 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, disk or optical disk.

[0052] The present application also provides a computer-readable storage medium, which stores a computer program. The computer program can be executed by the processor 11 to complete the method for generating question-answer pairs provided in the present application.

[0053] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method for generating question-answer pairs provided in the present application.

[0054] See also Figure 3 , is a flow chart of a method for generating question-answer pairs provided in an embodiment of the present application, such as Figure 3 As shown, the method may include the following steps 310 to 360.

[0055] Step 310: Divide the text to be processed in the target knowledge base according to a preset division strategy to obtain multiple paragraphs.

[0056] The target knowledge base is a knowledge base of the target domain, and the target domain is the domain where question-answer pairs need to be generated. For example, when generating question-answer pairs for employee training in the insurance industry, the target knowledge base can be constructed through multiple articles and documents in the insurance field. For example, when constructing question-answer pairs in the transportation service field for the intelligent question-answering system, the target knowledge base can be constructed through multiple articles and documents in the transportation service field.

[0057] The division strategy is used to indicate the division method of the text. For example, the division strategy may indicate that each natural paragraph of the text is divided, so that each natural paragraph is a paragraph, and multiple paragraphs are obtained. Alternatively, the division strategy may indicate that each group of 300 consecutive characters is divided into a paragraph, so that multiple paragraphs are obtained.

[0058] Each article and each document in the target knowledge base can be considered as a text to be processed. For the text to be processed, the text to be processed can be divided by a division strategy to obtain multiple paragraphs.

[0059] Step 320: For each paragraph, extract a number of keywords based on a preset extraction strategy.

[0060] Among them, the extraction strategy indicates the way to extract keywords from the paragraph.

[0061] After obtaining multiple paragraphs, several keywords can be extracted from each paragraph through an extraction strategy.

[0062] In one embodiment, when extracting keywords, for each paragraph, the paragraph may be segmented to obtain multiple segmentation results. Here, the segmentation process may be performed by a segmentation tool (eg, @ / node-rs / jieba).

[0063] After obtaining multiple word segmentation results, specific characters and stop words can be filtered out, and the remaining word segmentation results can be used as keywords for the paragraph. Here, specific characters and stop words are words that have nothing to do with the actual content of the paragraph, and specific characters can be non-Chinese characters, non-English letters, non-numbers, and non-blank characters. The filtering rules can be configured through the regular algorithm to filter out the keywords of the paragraph from the multiple word segmentation results corresponding to each paragraph.

[0064] In one embodiment, considering that a paragraph may be long, if the paragraph is directly segmented and filtered, a large number of words can be obtained. However, directly using a large number of words as keywords will cause the subsequent generated question-answer pairs to be too vague, low-value and inaccurate. To solve this problem, see Figure 4 , provided in one embodiment of the present application Figure 3 The detailed flow chart of step 320 is as follows: Figure 4As shown, the following steps 321 to 324 may be performed when extracting keywords.

[0065] Step 321: For each paragraph, each logical word in the preset logical word list is matched in the paragraph. If any logical word is matched in the paragraph, it is expanded according to the position index of the logical word in the paragraph to obtain the text interval corresponding to the logical word.

[0066] The logical word list includes multiple logical words, each of which can represent a logical relationship in the text description. For example, the logical words can be "and", "but", "however", "therefore", "because...so...", "due to...therefore...", "lead to", "result", etc.

[0067] For each paragraph, each logical word can be matched in the paragraph. On the one hand, if the logical word cannot be matched, no processing can be performed. On the other hand, if the logical word is matched in the paragraph, the position index of the logical word in the paragraph can be determined. Here, the position index indicates the position of the logical word in the paragraph. Further, expansion can be performed based on the position index of the logical word in the paragraph to obtain the text interval corresponding to the logical word.

[0068] For example, if the position index of any logical word in a paragraph is n (the logical word starts at the nth word in the paragraph), the position index n can be expanded to [nk, n+k] as a text interval, and the text interval can intercept the words with position indexes nk to n+k from the paragraph as the text in the text interval. Here, the expansion is the number of positions k expanded forward and backward, which can be configured based on experience.

[0069] In addition, if the preset extension method exceeds the paragraph, it can be adaptively adjusted. For example, if nk is less than the position index of the first word in the paragraph, the endpoint nk of the text interval is corrected to 0 (the position index of the first word); if n+k is greater than the position index of the last word in the paragraph, the endpoint n+k of the text interval is corrected to the position index of the last word in the paragraph.

[0070] Step 322: Perform word segmentation processing on the text in each text interval to obtain multiple word segmentation results corresponding to the text interval.

[0071] After determining a number of text intervals, the text in the text intervals can be extracted from the paragraphs, and then word segmentation processing is performed on the text in each text interval, thereby obtaining a plurality of word segmentation results corresponding to the text interval.

[0072] Step 323: Filter out specific characters and stop words from the multiple word segmentation results of each text interval, and use the remaining word segmentation results as keywords for the text interval; wherein the specific characters are characters that are not Chinese characters, English letters, numbers, or blank characters.

[0073] After obtaining multiple word segmentation results for each text interval, specific characters and stop words can be filtered out, and the remaining word segmentation results can be used as keywords for the paragraph. Here, specific characters and stop words are words that have nothing to do with the actual content of the paragraph, and specific characters can be non-Chinese characters, non-English letters, non-numbers, and non-blank characters. Filtering rules can be configured through regular algorithms to filter out keywords for the text interval from multiple word segmentation results for the text interval.

[0074] Step 324: De-duplicate keywords in several text intervals in each paragraph to obtain keywords for the paragraph.

[0075] Keywords in different text intervals of the same paragraph may be repeated. Therefore, the keywords of several text intervals of the paragraph may be deduplicated, so that the keywords of all the deduplicated text intervals are used as the keywords of the paragraph.

[0076] Through this measure, when the paragraph is long, the text around the logical words can be selected to extract effective and short keywords.

[0077] Step 330: construct a first input parameter for each paragraph and its corresponding keywords, and process the first input parameter by specifying a large model to obtain questions corresponding to the paragraph; wherein the first input parameter indicates questions related to the keywords generated for the paragraph.

[0078] Here, the designated large model is a large model that can process a natural language model. Exemplarily, the designated large model can be any one or more of Doubao, Tongyi Qianwen, Wenxin Large Model, ChatGPT (Chat Generative Pre-trained Transformer), etc.

[0079] After extracting several keywords for any paragraph, the first input parameter of the specified large model can be constructed based on the paragraph and several keywords of the paragraph, and the first input parameter can be handed over to the specified large model so that the specified large model can generate questions corresponding to the paragraph based on the first input parameter.

[0080] Step 340: Based on the keywords corresponding to each paragraph, a number of related paragraphs corresponding to the keywords are retrieved from multiple paragraphs through approximate nearest neighbor search.

[0081] After extracting a number of keywords for any paragraph, for the keywords of the paragraph, the group of keywords can be searched for approximate nearest neighbors in multiple paragraphs, so as to recall a number of paragraphs corresponding to the group of keywords, which can be called relevant paragraphs of the group of keywords. It can be understood that the relevant paragraphs of the group of keywords include the paragraph from which the group of keywords is extracted.

[0082] Here, Approximate Nearest Neighbor Search (ANNS) can be performed by any method such as local sensitive hashing, KD tree, ball tree, random projection tree, etc. Before performing the approximate nearest neighbor search, several keywords can be vectorized to obtain keyword vectors; each paragraph can be vectorized to obtain a paragraph vector. Then through the keyword vector and the paragraph vector. Here, the vectorization process can be performed by a text embedding model. Exemplarily, the text embedding model can be text-embedding-ada-002.

[0083] Step 350: Based on several related paragraphs corresponding to each paragraph, construct a second input parameter for each question of the paragraph, and process the second input parameter through the specified large model to obtain the answer corresponding to the question; wherein the second input parameter indicates the generation of an answer for the question.

[0084] For any paragraph, after determining several related paragraphs using several keywords of the paragraph, a second input parameter corresponding to each question of the paragraph can be constructed, and the second input parameter is submitted to the designated large model so that the designated large model can generate an answer to the question based on the second input parameter. In this case, answers to all questions can be obtained.

[0085] Step 360: construct each question corresponding to the text to be processed and the answer corresponding to the question into a question-answer pair to obtain multiple question-answer pairs.

[0086] After processing each paragraph of the text to be processed, all questions in the text to be processed and their corresponding answers can be obtained, and then the questions and answers can be constructed into question-answer pairs to obtain multiple question-answer pairs.

[0087] Through the above measures, multiple texts in the target knowledge base can be effectively preprocessed, and then questions and answers can be effectively generated with the help of a specified large model to build high-quality question-answer pairs for the target knowledge field.

[0088] In one embodiment, see Figure 5 , is a flow chart of a paragraph division method provided in an embodiment of the present application, such as Figure 5 As shown, the method may include the following steps 510 to 540.

[0089] Step 510: Segment the text to be processed according to text symbols to obtain multiple text units.

[0090] Here, the text symbol may be a combination of one or more punctuation marks (such as a period), special symbols (such as a line break), etc.

[0091] The text to be processed can be segmented minimally through text symbols to obtain multiple text units.

[0092] Step 520: Perform vectorization processing on each text unit to obtain a unit vector.

[0093] By vectorizing each text unit through a text embedding model (for example, text-embedding-ada-002), the unit vector corresponding to the text unit can be obtained.

[0094] Step 530: Perform an approximate nearest neighbor search on the unit vector of each text unit in the logical word vector library, and determine a similarity score based on the similarity between the unit vector and the most similar logical word vector.

[0095] Among them, the logical word vector library includes the logical word vectors corresponding to each logical word in the logical word list. In one embodiment, before executing step 530, each logical word in the preset logical word list can be vectorized to obtain the logical word vector corresponding to the logical word. Furthermore, the logical word vector library can be constructed based on multiple logical word necklaces. It should be noted that the text embedding model used for vectorization processing in each link of this solution should be the same text embedding model.

[0096] By performing an approximate nearest neighbor search on the unit vector of any text unit in the logical word vector library, the logical word vector that is most similar to the unit vector of the text unit can be obtained. Furthermore, the similarity between the unit vector and the most similar logical word vector can be used to determine the similarity score. Exemplarily, the similarity can be used directly as the similarity score. Alternatively, the similarity can be multiplied by a preset coefficient and then added to a preset constant to obtain a similarity score. The similarity score can characterize the degree of similarity between the unit vector and its most similar logical word vector.

[0097] Step 540: Compare the similarity score of each text unit with the similarity score of the adjacent text unit. If the difference in similarity score is less than a preset difference threshold, merge the text unit with the text unit of the vector. Multiple text units are merged to obtain multiple paragraphs.

[0098] After calculating the similarity score for each text unit, each text unit can be compared with its adjacent text units (e.g., the previous text unit, the next text unit) according to the order of arrangement of the text units in the text to be processed. During the comparison process, it can be determined whether the difference between the two similarities is less than a difference threshold. Here, the difference threshold can be configured based on experience.

[0099] On the one hand, if the difference between the similarities of the two text units is not less than the difference threshold, no processing is required. On the other hand, if the difference between the similarities of the two text units is less than the difference threshold, the two text units can be merged. After the merging process, multiple paragraphs can be obtained. Each paragraph can include at least two text units.

[0100] Through the above measures, multiple text units with similarity can be merged into one paragraph based on the highest similarity between the text unit and the logical word. In this case, text units with similar text logic can be merged together to facilitate subsequent processing.

[0101] In one embodiment, when executing step 330 to construct the first input parameter, each paragraph and its corresponding keywords can be respectively filled into a preset first prompt word template to obtain the first input parameter. The first prompt word template indicates the generation question.

[0102] An example paragraph is: The insurance amount is determined by negotiation between the policyholder and the insurer and is specified in this contract. The insurance premium is calculated and determined by the insurer based on the applicable insurance premium rate when underwriting.

[0103] Keywords: insurance amount

[0104] The first prompt template is:

[0105] You are an expert in question generation and you will generate questions based on {keywords} keywords and according to the following paragraph data.

[0106] ###Paragraph data

[0107] {Paragraph data}

[0108] Fill the keywords and paragraphs into the first prompt word template, and you can get the first input parameter as follows:

[0109] You are a question generation expert, and you will generate questions based on the keyword "insurance amount fee" and the data in the following paragraph.

[0110] ###Paragraph data

[0111] The insurance amount shall be determined by negotiation between the policyholder and the insurer and shall be specified in this contract. The insurance premium shall be calculated and determined by the insurer based on the applicable insurance premium rate when underwriting the insurance.

[0112] In one embodiment, when executing step 350 to construct the first input parameter, for each question in the paragraph, the question, a number of keywords corresponding to the paragraph, a number of related paragraphs corresponding to the keywords, and a full text summary of the text to be processed can be filled into the second prompt word template to obtain the second input parameter. The second prompt word template indicates generating an answer.

[0113] By way of example, the question is: How is the insurance amount determined?

[0114] Some key words are: insurance amount fee

[0115] Some relevant paragraphs are: The insurance amount is determined by negotiation between the policyholder and the insurer and is specified in this contract. The insurance premium is calculated and determined by the insurer based on the applicable insurance premium rate when underwriting.

[0116] The full text summary is: This insurance contract consists of multiple documents and must be in written form. The insured is between 30 and 65 years old, in good health, and able to work or live normally; those aged 65 to 70 must meet specific conditions. The insured should have full civil capacity, and the beneficiary is the insured himself by default. The insurance amount is determined by negotiation between the two parties, and the insurance premium is determined by the insurer based on the rate.

[0117] The second prompt word template is:

[0118] You are an answer generation expert. First, you need to generate the answer to the question based on {question} and {keywords} keywords and the full text abstract {abstract} and the following paragraph data.

[0119] ###Paragraph data

[0120] {paragraph}

[0121] Fill the question, several keywords, several relevant paragraphs corresponding to the keywords, and the full text summary of the text to be processed into the second prompt word template to obtain the second input parameter:

[0122] You are an answer generation expert. First, you need to generate the answer based on the question "How is the insurance amount determined?" and the keywords "insurance amount premium" and the full text summary "This insurance contract consists of multiple documents and must be in written form. The insured is between 30 and 65 years old, in good health, and able to work or live normally; those between 65 and 70 years old must meet specific conditions. The insured should have full civil capacity, and the beneficiary is the insured himself by default. The insurance amount is determined by negotiation between the two parties, and the premium is determined by the insurer based on the rate calculation." and the following paragraph data.

[0123] ###Paragraph data

[0124] The insurance amount shall be determined by negotiation between the policyholder and the insurer and shall be specified in this contract. The insurance premium shall be calculated and determined by the insurer based on the applicable insurance premium rate when underwriting the insurance.

[0125] Figure 6 is a block diagram of a device for generating question-answer pairs according to an embodiment of the present invention. Figure 6 As shown, the device may include:

[0126] A division module 610 is used to divide the text to be processed in the target knowledge base according to a preset division strategy to obtain multiple paragraphs;

[0127] An extraction module 620, for extracting a number of keywords for each paragraph based on a preset extraction strategy;

[0128] The first generating module 630 is used to construct a first input parameter for each paragraph and its corresponding keywords, and process the first input parameter by specifying a large model to obtain questions corresponding to the paragraph; wherein the first input parameter indicates generating questions related to the keywords for the paragraph;

[0129] A recall module 640 is used to recall a number of related paragraphs corresponding to the keywords from the plurality of paragraphs through an approximate nearest neighbor search according to the keywords corresponding to each paragraph;

[0130] A second generating module 650 is used to construct a second input parameter for each question of each paragraph based on a number of related paragraphs corresponding to each paragraph, and process the second input parameter through the specified large model to obtain an answer corresponding to the question; wherein the second input parameter indicates generating an answer for the question;

[0131] The construction module 660 is used to construct each question corresponding to the text to be processed and the answer corresponding to the question into a question-answer pair to obtain multiple question-answer pairs.

[0132] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned question-answer pair generation method, which will not be repeated here.

[0133] In several embodiments provided in the present application, the disclosed devices and methods may also be implemented in other ways. The device embodiments described above are merely schematic, for example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram may represent a module, a program segment or a part of a code, and a module, a program segment or a part of a code contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they may sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0134] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0135] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program codes.

Claims

1. A method for generating question-answer pairs, characterized in that: include: For the text to be processed in the target knowledge base, divide it according to the preset division strategy to obtain multiple paragraphs; For each paragraph, extract several keywords based on the preset extraction strategy; Constructing a first input parameter for each paragraph and its corresponding keywords, and processing the first input parameter by specifying a large model to obtain questions corresponding to the paragraph; wherein the first input parameter indicates generating questions related to the keywords for the paragraph; According to the keywords corresponding to each paragraph, recalling the relevant paragraphs corresponding to the keywords from the multiple paragraphs through approximate nearest neighbor search; Based on a number of related paragraphs corresponding to each paragraph, construct a second input parameter for each question of the paragraph, and process the second input parameter through the specified large model to obtain an answer corresponding to the question; wherein the second input parameter indicates generating an answer for the question; Each question corresponding to the text to be processed and the answer corresponding to the question are constructed as a question-answer pair to obtain multiple question-answer pairs.

2. The method according to claim 1, characterized in that The text to be processed in the target knowledge base is divided according to a preset division strategy to obtain multiple paragraphs, including: Segmenting the text to be processed according to text symbols to obtain multiple text units; Perform vectorization on each text unit to obtain a unit vector; Perform an approximate nearest neighbor search on the unit vector of each text unit in the logical word vector library, and determine a similarity score based on the similarity between the unit vector and the most similar logical word vector; The similarity score of each text unit is compared with the similarity score of the adjacent text unit. If the difference in the similarity scores is less than a preset difference threshold, the text unit is merged with the text unit of the vector, and the multiple text units are merged to obtain multiple paragraphs.

3. The method according to claim 2, characterized in that Before performing an approximate nearest neighbor search on the unit vector of each text unit in the logical word vector library, the method further includes: Perform vectorization processing on each logical word in the preset logical word list to obtain the logical word vector corresponding to the logical word; Build a logical word vector library based on multiple logical word vectors.

4. The method according to claim 1, characterized in that: For each paragraph, a number of keywords are extracted based on a preset extraction strategy, including: For each paragraph, perform word segmentation on the paragraph to obtain multiple word segmentation results; Specific characters and stop words are filtered out from the multiple word segmentation results, and the remaining word segmentation results are used as keywords for the paragraph; wherein the specific characters are characters that are not Chinese characters, English letters, numbers, or blank characters.

5. The method according to claim 1, characterized in that For each paragraph, a number of keywords are extracted based on a preset extraction strategy, including: For each paragraph, each logical word in the preset logical word list is matched in the paragraph. If any logical word is matched in the paragraph, it is expanded according to the position index of the logical word in the paragraph to obtain the text interval corresponding to the logical word; Perform word segmentation processing on the text in each text interval to obtain multiple word segmentation results corresponding to the text interval; Filter out specific characters and stop words from the multiple word segmentation results of each text interval, and use the remaining word segmentation results as keywords of the text interval; wherein the specific characters are characters that are not Chinese characters, English letters, numbers, or blank characters; The keywords in several text intervals in each paragraph are deduplicated to obtain the keywords of the paragraph.

6. The method according to claim 1, characterized in that The first input parameter is constructed for each paragraph and its corresponding keywords, including: Each paragraph and its corresponding keywords are respectively filled into a preset first prompt word template to obtain a first input parameter; wherein the first prompt word template indicates a generated question.

7. The method according to claim 1, characterized in that The second input parameter is constructed for each question of each paragraph based on a number of related paragraphs corresponding to each paragraph, including: For each question in the paragraph, the question, several keywords corresponding to the paragraph, several related paragraphs corresponding to the keywords, and the full-text summary of the text to be processed are filled into a preset second prompt word template to obtain a second input parameter; wherein the second prompt word template indicates the generation of an answer.

8. The method according to claim 7, characterized in that The step of obtaining the full text abstract of the text to be processed includes: The text to be processed and the summary prompt word are submitted to the designated large model to obtain a full-text summary output by the designated large model; wherein the summary prompt word indicates the generation of a summary.

9. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing processor-executable instructions; The processor is configured to execute the question-answer pair generation method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program can be executed by a processor to complete the method for generating question-answer pairs as described in any one of claims 1-8.

11. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the method for generating question-answer pairs according to any one of claims 1 to 8 is implemented.

Images