A test set generation method, device, storage medium and electronic device
By obtaining the target keywords and question words in the target field, forming seed problems, searching and semantic analysis of candidate test problems, the problem of low test set generation efficiency and coverage in the existing technology is solved, and efficient and comprehensive test set generation is achieved .
Patent Information
- Application Number
- CN202111094233.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-17
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2041-09-17
AI Technical Summary
In the prior art, the generation efficiency of question-answer test sets is low and the coverage rate is low, resulting in low efficiency in question-and-answer system evaluation and improvement.
By obtaining the target keywords in the target field, forming seed problems with the splicing of question words, searching for candidate test problems, and semantic analysis of them, generating test sets based on preset conditions to improve generation efficiency and coverage.
A large number of candidate test questions are generated without manual sampling surveys, which improves test set generation efficiency and coverage, and ensures the quality and coverage of the test set.
Smart Images
Figure CN114090421B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a test set generation method, device, storage medium and electronic device. Background Art
[0002] With the development of the Internet, the information on the Internet is becoming more and more abundant, and the relevant information retrieved through search engines is becoming more and more. In addition, automatic question answering has solved the problems caused by information explosion. Therefore, designing a good question answering test set is very important for the evaluation and improvement of the question answering system.
[0003] In the prior art, test sets are usually generated using questions and answers obtained from manual sampling surveys, resulting in low generation efficiency and low coverage. Summary of the invention
[0004] In view of this, the embodiments of the present application propose a test set generation method, device, storage medium and electronic device to solve the problems of low test set generation efficiency and low coverage in the prior art.
[0005] In a first aspect, an embodiment of the present application provides a test set generation method, which includes:
[0006] Get target keywords in the target field;
[0007] collecting candidate test questions based on the target keywords;
[0008] A test set is generated using the candidate test questions that meet preset conditions.
[0009] In a possible implementation, the collecting candidate test questions based on the target keyword includes:
[0010] Concatenate the target keyword with the question word to obtain a seed question;
[0011] The candidate test questions within the target domain are searched based on the seed questions.
[0012] In a possible implementation manner, the generating a test set using the candidate test questions that meet preset conditions includes:
[0013] Performing semantic analysis on the candidate test question to determine whether the candidate test question meets a preset condition; wherein the preset condition is that the semantic clarity is greater than a threshold;
[0014] A test set is generated using the candidate test questions that meet preset conditions.
[0015] In a possible implementation manner, the generating a test set using the candidate test questions that meet preset conditions includes:
[0016] If the candidate test question does not meet the preset condition, then delete it;
[0017] If the candidate test question meets the preset condition, the candidate test question is used as the target test question;
[0018] All of the target test questions are combined to form the test set.
[0019] In a second aspect, an embodiment of the present application further provides a test set generating device, which includes:
[0020] An acquisition module configured to acquire target keywords in a target domain;
[0021] a collection module configured to collect candidate test questions based on the target keyword;
[0022] A generating module is configured to generate a test set using the candidate test questions that meet preset conditions.
[0023] In a possible implementation manner, the collection module is specifically configured as follows:
[0024] Concatenate the target keyword with the question word to obtain a seed question;
[0025] The candidate test questions within the target domain are searched based on the seed questions.
[0026] In a possible implementation, the generation module is specifically configured as follows:
[0027] Performing semantic analysis on the candidate test question to determine whether the candidate test question meets a preset condition; wherein the preset condition is that the semantic clarity is greater than a threshold;
[0028] A test set is generated using the candidate test questions that meet preset conditions.
[0029] In a possible implementation manner, the generating module is further specifically configured as follows:
[0030] If the candidate test question does not meet the preset condition, then delete it;
[0031] If the candidate test question meets the preset condition, the candidate test question is used as the target test question;
[0032] All of the target test questions are combined to form the test set.
[0033] In a third aspect, an embodiment of the present application further provides a storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the following steps are performed:
[0034] Get target keywords in the target field;
[0035] collecting candidate test questions based on the target keywords;
[0036] A test set is generated using the candidate test questions that meet preset conditions.
[0037] In a fourth aspect, an embodiment of the present application further provides an electronic device, comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through a bus, and when the machine-readable instructions are executed by the processor, the following steps are performed:
[0038] Get target keywords in the target field;
[0039] collecting candidate test questions based on the target keywords;
[0040] A test set is generated using the candidate test questions that meet preset conditions.
[0041] The embodiment of the present application utilizes the target keyword to collect candidate test questions within the target field to which it belongs, and determines whether the candidate test questions meet the preset conditions, and then generates a test set based on the candidate test questions that meet the preset conditions. A large number of candidate test questions can be obtained without manual sampling investigation, which greatly improves the efficiency of test set generation; and, the candidate test questions are derived from the search records existing in the network, so that the coverage of the generated test set is relatively high. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0043] Figure 1 A flow chart of a test set generation method provided by the present application is shown;
[0044] Figure 2 A flowchart of collecting candidate test questions based on target keywords in a test set generation method provided by the present application is shown;
[0045] Figure 3A flowchart of generating a test set using candidate test questions that meet preset conditions in a test set generation method provided by the present application is shown;
[0046] Figure 4 A schematic diagram of the structure of a test set generation device provided by the present application is shown;
[0047] Figure 5 A schematic structural diagram of an electronic device provided by the present application is shown. DETAILED DESCRIPTION
[0048] In order to make the purpose, technical solution and advantages of the embodiment of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described in conjunction with the drawings of the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, not all of the embodiments. Based on the described embodiment of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0049] Unless otherwise defined, the technical terms or scientific terms used in this application should be understood by people with ordinary skills in the field to which this application belongs. "First", "second" and similar words used in this application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.
[0050] In order to keep the following description of the embodiments of the present application clear and concise, the present application omits detailed descriptions of known functions and known components.
[0051] like Figure 1 As shown, it is a flowchart of the test set generation method provided in the first aspect of the present application, wherein the specific steps include S101-S103.
[0052] S101, obtaining target keywords in a target field.
[0053] In a specific implementation, the target domain is a specific domain, such as beauty, food, mobile phones, etc., and the smaller the target domain is, the more accurate the question-answering system tested by the test set generated by the method of the present application will be.
[0054] Furthermore, the target keyword in the target field may be the name of each product, the name of each accessory, the name of each food, etc., and may also be determined based on the frequency and number of times it appears in other search engines, etc.
[0055] S102, collecting candidate test questions based on target keywords.
[0056] After obtaining the target keywords in the target field, candidate test questions are collected based on the target keywords. The candidate test questions are a large number of questions that already exist in other search engines.
[0057] Specifically, please refer to Figure 2 The method flowchart shown in FIG. 1 is used to collect candidate test questions based on target keywords, wherein specific steps include S201 and S202.
[0058] S201, concatenate the target keyword with the question word to obtain a seed question.
[0059] S202, searching for candidate test questions in the target domain based on the seed questions.
[0060] In the specific implementation, the target keyword is first concatenated with the question word to obtain the seed question. For example, cosmetics A and what are concatenated to obtain the seed question "cosmetics A what". Of course, question words also include "how", "how", "how much", etc. Each target keyword can be concatenated with all the question words to improve the accuracy of the candidate test questions.
[0061] After obtaining the seed questions, we search for candidate test questions in the target field based on the seed questions, that is, we input the seed questions into other search engines and use the questions recommended and displayed by other search engines as candidate test questions. These candidate test questions are all questions that users have searched on the Internet and are more authentic.
[0062] For example, after entering "What is Cosmetics A", other search engines recommend and display questions such as "What grade of product is Cosmetics A", "What product of Cosmetics A is easy to use", "What country is Cosmetics A from", "What series of Cosmetics A is suitable for what age group", "What country's brand of Cosmetics A", "What series of Cosmetics A is easy to use", "When was Cosmetics A established", "When was Cosmetics A launched on the market", "What is easy to use in Cosmetics A", "When did Cosmetics A enter the Chinese market", etc. All of the above questions are used as candidate test questions.
[0063] S103: Generate a test set using candidate test questions that meet preset conditions.
[0064] In a specific implementation, not all candidate test questions can be used as samples in a test set. Therefore, after obtaining a candidate test question, it is necessary to further determine whether the candidate test question meets preset conditions, and generate a test set using the candidate test questions that meet the preset conditions.
[0065] Specifically, Figure 3 A flow chart of a method for generating a test set using candidate test questions that meet preset conditions is shown, wherein specific steps include S301-S304.
[0066] S301, performing semantic analysis on the candidate test questions to determine whether the candidate test questions meet a preset condition; wherein the preset condition is that the semantic clarity is greater than a threshold.
[0067] S302: If the candidate test question does not meet the preset conditions, it is deleted.
[0068] S303: If the candidate test question meets the preset conditions, the candidate test question is used as the target test question.
[0069] S304, all target test questions are combined to form a test set.
[0070] In a specific implementation, a pre-trained semantic model can be used to perform semantic analysis on candidate test questions to determine whether the candidate test questions meet preset conditions. Of course, other analysis methods can also be used to perform semantic analysis on candidate test questions, as long as the semantics of the candidate test questions can be analyzed. The embodiments of the present application do not make specific limitations on this.
[0071] Among them, the preset condition is that the semantic clarity is greater than the threshold, and the semantic clarity represents whether the sentence can form a complete question, whether there is ambiguity, etc. The candidate test question is semantically analyzed to determine whether the candidate test question meets the preset condition, that is, the candidate test question is semantically analyzed to obtain the semantic clarity of the candidate test question, and the semantic clarity is compared with the threshold. If the semantic clarity is greater than the threshold, it is determined that the candidate test question meets the preset condition; if the semantic clarity is less than or equal to the threshold, it is determined that the candidate test question does not meet the preset condition.
[0072] If the candidate test question does not meet the preset conditions, it will be deleted. For example, the semantic clarity of "What is the good use of cosmetic A" is less than the threshold, indicating that it cannot form a complete question, so it is directly deleted; if the candidate test question meets the preset conditions, the candidate test question is used as the target test question, and all target test questions are combined to form a test set. The test set is used to train the question-answering system without manually setting questions, which greatly improves the efficiency of test set generation and the test efficiency of the question-answering system.
[0073] The embodiment of the present application utilizes the target keyword to collect candidate test questions within the target field to which it belongs, and determines whether the candidate test questions meet the preset conditions, and then generates a test set based on the candidate test questions that meet the preset conditions. A large number of candidate test questions can be obtained without manual sampling investigation, which greatly improves the efficiency of test set generation; and, the candidate test questions are derived from the search records existing in the network, so that the coverage of the generated test set is relatively high.
[0074] Based on the same inventive concept, the second aspect of the present application also provides a test set generation device corresponding to a test set generation method. Since the principle of solving the problem by the test set generation device in the present application is similar to the above-mentioned test set generation method of the present application, the implementation of the test set generation device can refer to the implementation of the method, and the repeated parts will not be repeated.
[0075] Figure 4 A schematic diagram of a test set generation device provided in an embodiment of the present application is shown, which specifically includes:
[0076] An acquisition module 401 is configured to acquire target keywords in a target field;
[0077] A collection module 402 configured to collect candidate test questions based on the target keyword;
[0078] The generating module 403 is configured to generate a test set using the candidate test questions that meet preset conditions.
[0079] In yet another embodiment, the collection module 402 is specifically configured as follows:
[0080] Concatenate the target keyword with the question word to obtain a seed question;
[0081] The candidate test questions within the target domain are searched based on the seed questions.
[0082] In yet another embodiment, the generating module 403 is specifically configured as follows:
[0083] Performing semantic analysis on the candidate test question to determine whether the candidate test question meets a preset condition; wherein the preset condition is that the semantic clarity is greater than a threshold;
[0084] A test set is generated using the candidate test questions that meet preset conditions.
[0085] In yet another embodiment, the generating module 403 is further specifically configured as follows:
[0086] If the candidate test question does not meet the preset condition, then delete it;
[0087] If the candidate test question meets the preset condition, the candidate test question is used as the target test question;
[0088] All of the target test questions are combined to form the test set.
[0089] The embodiment of the present application utilizes the target keyword to collect candidate test questions within the target field to which it belongs, and determines whether the candidate test questions meet the preset conditions, and then generates a test set based on the candidate test questions that meet the preset conditions. A large number of candidate test questions can be obtained without manual sampling investigation, which greatly improves the efficiency of test set generation; and, the candidate test questions are derived from the search records existing in the network, so that the coverage of the generated test set is relatively high.
[0090] An embodiment of the present application provides a storage medium, which is a computer-readable medium and stores a computer program. When the computer program is executed by a processor, the method provided by any embodiment of the present application is implemented, including the following steps S11 to S13:
[0091] S11, obtaining target keywords in the target field;
[0092] S12, collecting candidate test questions based on the target keywords;
[0093] S13, generating a test set using the candidate test questions that meet preset conditions.
[0094] When the computer program is executed by the processor to collect candidate test questions based on the target keyword, the processor specifically executes the following steps: concatenating the target keyword with the question word to obtain a seed question; and searching for the candidate test question in the target field based on the seed question.
[0095] When the computer program is executed by the processor to generate a test set using the candidate test questions that meet the preset conditions, the processor also executes the following steps: performing semantic analysis on the candidate test questions to determine whether the candidate test questions meet the preset conditions; wherein the preset conditions are that the semantic clarity is greater than a threshold; and generating a test set using the candidate test questions that meet the preset conditions.
[0096] When the computer program is executed by the processor to generate a test set using the candidate test questions that meet the preset conditions, the processor also executes the following steps: if the candidate test question does not meet the preset conditions, it is deleted; if the candidate test question meets the preset conditions, the candidate test question is used as the target test question; all the target test questions are combined to form the test set.
[0097] The embodiment of the present application utilizes the target keyword to collect candidate test questions within the target field to which it belongs, and determines whether the candidate test questions meet the preset conditions, and then generates a test set based on the candidate test questions that meet the preset conditions. A large number of candidate test questions can be obtained without manual sampling investigation, which greatly improves the efficiency of test set generation; and, the candidate test questions are derived from the search records existing in the network, so that the coverage of the generated test set is relatively high.
[0098] An embodiment of the present application provides an electronic device. The structural diagram of the electronic device can be as follows: Figure 5 As shown, at least a memory 501 and a processor 502 are included, the memory 501 stores a computer program, and the processor 502 implements the method provided by any embodiment of the present application when executing the computer program on the memory 501. Exemplarily, the steps of the electronic device computer program are as follows S21 to S23:
[0099] S21, obtaining target keywords in the target field;
[0100] S22, collecting candidate test questions based on the target keywords;
[0101] S23, generating a test set using the candidate test questions that meet preset conditions.
[0102] When collecting candidate test questions based on the target keyword stored in the execution memory, the processor also executes the following computer program: concatenating the target keyword with the question word to obtain a seed question; and searching for the candidate test questions in the target field based on the seed question.
[0103] When the processor generates a test set using the candidate test questions that meet the preset conditions stored in the execution memory, it also executes the following computer program: performing semantic analysis on the candidate test questions to determine whether the candidate test questions meet the preset conditions; wherein the preset conditions are that the semantic clarity is greater than a threshold; and generating a test set using the candidate test questions that meet the preset conditions.
[0104] When the processor generates a test set using the candidate test questions that meet the preset conditions stored in the execution memory, it also executes the following computer program: if the candidate test question does not meet the preset conditions, it is deleted; if the candidate test question meets the preset conditions, the candidate test question is used as the target test question; all the target test questions are combined to form the test set.
[0105] The embodiment of the present application utilizes the target keyword to collect candidate test questions within the target field to which it belongs, and determines whether the candidate test questions meet the preset conditions, and then generates a test set based on the candidate test questions that meet the preset conditions. A large number of candidate test questions can be obtained without manual sampling investigation, which greatly improves the efficiency of test set generation; and, the candidate test questions are derived from the search records existing in the network, so that the coverage of the generated test set is relatively high.
[0106] Optionally, in this embodiment, the above-mentioned storage medium may include, but is not limited to, various media that can store program codes, such as a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk. Optionally, in this embodiment, the processor executes the method steps recorded in the above-mentioned embodiment according to the program code stored in the storage medium. Optionally, the specific examples in this embodiment can refer to the examples described in the above-mentioned embodiment and the optional implementation mode, and this embodiment is not repeated here. Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be implemented with a general computing device, which can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices. Optionally, they can be implemented with executable program codes of computing devices, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from here, or they can be made into individual integrated circuit modules respectively, or multiple modules or steps in them can be made into a single integrated circuit module to implement. In this way, the present application is not limited to any specific combination of hardware and software.
[0107] In addition, although exemplary embodiments have been described herein, the scope includes any and all embodiments based on the present application with equivalent elements, modifications, omissions, combinations (e.g., various embodiments intersecting schemes), adaptations or changes. The elements in the claims will be interpreted broadly based on the language adopted in the claims, and are not limited to the examples described in this specification or during the implementation of the application, and the examples will be interpreted as non-exclusive. Therefore, this specification and examples are intended to be considered as examples only, and the true scope and spirit are indicated by the following claims and the full scope of their equivalents.
[0108] The above description is intended to be illustrative rather than restrictive. For example, the above examples (or one or more of them) can be used in combination with each other. For example, those of ordinary skill in the art can use other embodiments when reading the above description. In addition, in the above-mentioned specific embodiments, various features can be grouped together to simplify the application. This should not be interpreted as an intention that a disclosed feature that does not require protection is necessary for any claim. On the contrary, the subject matter of the present application may be less than all the features of a specific disclosed embodiment. Thus, the following claims are incorporated into the specific embodiments as examples or embodiments, wherein each claim is independently used as a separate embodiment, and it is considered that these embodiments can be combined with each other in various combinations or arrangements. The scope of the present application should be determined with reference to the attached claims and the full scope of equivalent forms granted by these claims.
[0109] Multiple embodiments of the present application are described in detail above, but the present application is not limited to these specific embodiments. Those skilled in the art can make various variations and modifications to the embodiments based on the concept of the present application, and these variations and modifications should all fall within the scope of protection required by the present application.
Claims
1. A test set generation method, characterized in that: include: Get target keywords in the target field; Collecting candidate test questions based on the target keyword includes: concatenating the target keyword with a question word to obtain a seed question; searching for the candidate test questions in the target field based on the seed question; Generate a test set using the candidate test questions that meet the preset conditions, including: performing semantic analysis on the candidate test questions to determine whether the candidate test questions meet the preset conditions; wherein the preset conditions are that the semantic clarity is greater than a threshold; if the candidate test questions do not meet the preset conditions, delete them; if the candidate test questions meet the preset conditions, use the candidate test questions as the target test questions; and all the target test questions are combined to form the test set.
2. A test set generation device, characterized in that: include: An acquisition module configured to acquire target keywords in a target domain; A collection module configured to collect candidate test questions based on the target keyword, wherein the collection module is specifically configured to: concatenate the target keyword with the question word to obtain a seed question; searching the candidate test questions in the target domain based on the seed questions; A generation module is configured to generate a test set using the candidate test questions that meet preset conditions, and the generation module is specifically configured to: perform semantic analysis on the candidate test questions to determine whether the candidate test questions meet the preset conditions; wherein the preset conditions are that the semantic clarity is greater than a threshold; if the candidate test questions do not meet the preset conditions, delete them; if the candidate test questions meet the preset conditions, use the candidate test questions as the target test questions; and all the target test questions are combined to form the test set.
3. A computer-readable storage medium, characterized in that: The computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the following steps: Get target keywords in the target field; Collecting candidate test questions based on the target keyword includes: concatenating the target keyword with a question word to obtain a seed question; searching for the candidate test questions in the target field based on the seed question; Generate a test set using the candidate test questions that meet the preset conditions, including: performing semantic analysis on the candidate test questions to determine whether the candidate test questions meet the preset conditions; wherein the preset conditions are that the semantic clarity is greater than a threshold; if the candidate test questions do not meet the preset conditions, delete them; if the candidate test questions meet the preset conditions, use the candidate test questions as the target test questions; and all the target test questions are combined to form the test set.
4. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via a bus. When the machine-readable instructions are executed by the processor, the following steps are performed: Get target keywords in the target field; Collecting candidate test questions based on the target keyword includes: concatenating the target keyword with a question word to obtain a seed question; searching for the candidate test questions in the target field based on the seed question; Generate a test set using the candidate test questions that meet the preset conditions, including: performing semantic analysis on the candidate test questions to determine whether the candidate test questions meet the preset conditions; wherein the preset conditions are that the semantic clarity is greater than a threshold; if the candidate test questions do not meet the preset conditions, delete them; if the candidate test questions meet the preset conditions, use the candidate test questions as the target test questions; and all the target test questions are combined to form the test set.