Question sentence corpus generation method and device, equipment and computer readable storage medium

Abstract

This application provides a method, apparatus, device, and computer-readable storage medium for generating question corpora. The method includes: obtaining a keyword group; the keyword group includes at least two keywords; obtaining at least one target question expression corresponding to the keyword group; the target question expression is used to connect at least two keywords according to at least one connecting symbol; and generating at least one corresponding question corpus based on the keyword group and at least one target question expression. This application can improve the efficiency of question corpus generation and expand the question phrasing coverage of the question corpus.

Description

Technical Field

[0001] This application relates to data processing technology, and more particularly to a method, apparatus, device, and computer-readable storage medium for generating question corpora. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. Intelligent question-answering systems are an important branch of AI. To build an ideal intelligent question-answering system, a large corpus of questions is needed as a test set for testing.

[0003] When building a test set, it is necessary to manually compile a corpus of questions, which is not only time-consuming and laborious, but also inefficient. Since manual compilation relies heavily on personal prior knowledge, it often gets stuck in one's own knowledge system and cannot compile a corpus of questions with complete question types and broad coverage. Summary of the Invention

[0004] This application provides a method, apparatus, device, and computer-readable storage medium for generating question corpora, which can improve the generation efficiency of question corpora and expand the question types covered by the question corpora.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a method for generating interrogative sentence corpora, including:

[0007] Obtain keyword phrases, which must include at least two keywords;

[0008] Obtain at least one target question expression corresponding to the keyword group; the target question expression is used to connect at least two keywords based on at least one connector symbol.

[0009] Generate at least one corresponding question corpus based on the keyword group and at least one target question expression.

[0010] In some embodiments, the keyword category of the keyword includes at least one of the following: subject category, relation category, and object category.

[0011] In some embodiments, obtaining at least one target question expression corresponding to a keyword group includes: obtaining phrase information of the keyword group; the phrase information includes at least one of the following: keyword information and connector information; the keyword information includes the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category; the connector information includes the connector category corresponding to at least one keyword category in the keyword group; obtaining at least one target question expression from a preset question expression set based on the phrase information; the question expression set includes multiple question expressions.

[0012] In some embodiments, when the phrase information includes keyword information, obtaining the phrase information of the keyword group includes: obtaining the keyword category of each keyword in the keyword group; determining the number of keyword categories and the number of keywords corresponding to each keyword category based on the keyword category of each keyword.

[0013] In some embodiments, when the phrase information includes connection symbol information, obtaining the phrase information of the keyword phrase includes: when the number of keywords corresponding to the subject category is at least two, determining the connection symbol category corresponding to the subject category based on the attribute information of the keywords corresponding to the subject category; the connection symbol category corresponding to the subject category includes at least one of the following: parallel category and modifying category; when the number of keywords corresponding to the relation category is at least two, determining the connection symbol category corresponding to the relation category based on the attribute information of the keywords corresponding to the relation category; the connection symbol category corresponding to the relation category includes at least one of the following: parallel category and modifying category; when the number of keywords corresponding to the object category is at least one, determining the connection symbol category corresponding to the object category based on the attribute information of the keywords corresponding to the object category; the connection symbol category corresponding to the object category includes at least one of the following: single element category, set element category, and domain category.

[0014] In some embodiments, obtaining at least one target question expression from a preset set of question expressions based on phrase information includes: when the phrase information includes keyword information, determining the expression information corresponding to each question expression among multiple question expressions; the expression information includes the number of keyword categories in the question expression and the number of keywords corresponding to each keyword category; matching the expression information corresponding to each question expression based on the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category to obtain at least one target question expression.

[0015] In some embodiments, obtaining at least one target question expression from a preset set of question expressions based on phrase information includes: when the phrase information includes keyword information and connector information, determining the expression information corresponding to each question expression among multiple question expressions; the expression information includes the number of keyword categories in the question expression, the number of keywords corresponding to each keyword category, and the connector category corresponding to at least one keyword category; matching the expression information corresponding to each question expression based on the number of keyword categories in the keyword group, the number of keywords corresponding to each keyword category, and the connector category corresponding to at least one keyword category to obtain at least one target question expression.

[0016] In some embodiments, the method further includes: adding annotations to each question corpus; the annotations include at least one of the following: keyword groups corresponding to the question corpus, target question expression corresponding to the question corpus, and keyword tags corresponding to each keyword in the question corpus; in the process of generating a corpus test set based on the annotated question corpus, removing test corpora with the same annotations; the corpus test set includes multiple annotated test corpora.

[0017] In some embodiments, the method further includes: testing the question-answering system based on multiple annotated test corpora in a corpus test set; obtaining at least one target test corpus with abnormal test results; and determining the optimization direction of the question-answering system based on the annotations corresponding to the at least one target test corpus.

[0018] In some embodiments, the keyword tags include a first keyword tag corresponding to a keyword belonging to the subject category, a second keyword tag corresponding to a keyword belonging to the relation category, and a third keyword tag corresponding to a keyword belonging to the object category; the first keyword tag includes at least one of the following: a polysemous tag, an alias tag, a misspelling tag, and a missing word tag; the second keyword tag includes at least one of the following: a polysemous tag, an alias tag, and an implicit tag; the third keyword tag includes at least one of the following: a length tag, an amount tag, a time tag, a temperature tag, a volume tag, and a character tag.

[0019] This application provides a question corpus generation apparatus, the apparatus comprising:

[0020] The first acquisition module is used to acquire keyword groups, which include at least two keywords.

[0021] The second acquisition module is used to acquire at least one target question expression corresponding to the keyword group; the target question expression is used to connect at least two keywords based on at least one connector symbol.

[0022] The generation module is used to generate at least one corresponding question corpus based on keyword groups and at least one target question expression.

[0023] This application provides an apparatus for generating interrogative sentence corpora, including:

[0024] Memory, used to store executable instructions;

[0025] The processor, when executing executable instructions stored in the memory, implements the question corpus generation method provided in the embodiments of this application.

[0026] This application provides a computer-readable storage medium storing executable instructions, which, when executed by a processor, implement the question corpus generation method provided in this application.

[0027] The embodiments of this application have the following beneficial effects:

[0028] This application embodiment obtains a keyword group and at least one target question expression corresponding to the keyword group; generates at least one corresponding question corpus based on the keyword group and at least one target question expression, which can automatically generate multiple question corpora corresponding to the keyword group, thereby improving the generation efficiency of question corpora. Furthermore, since different question types are represented by target question expressions, the question types covered by question corpora generated by different target question expressions are wider. Attached Figure Description

[0029] Figure 1 This is an optional architecture diagram of the question corpus generation system provided in the embodiments of this application;

[0030] Figure 2 This is a schematic diagram of the structure of the question corpus generation device provided in the embodiments of this application;

[0031] Figure 3A This is an optional flowchart illustrating the question corpus generation method provided in the embodiments of this application;

[0032] Figure 3B This is an optional flowchart illustrating the question corpus generation method provided in the embodiments of this application;

[0033] Figure 3C This is an optional flowchart illustrating the question corpus generation method provided in the embodiments of this application;

[0034] Figure 3D This is an optional flowchart illustrating the question corpus generation method provided in the embodiments of this application;

[0035] Figure 3E This is an optional flowchart illustrating the question corpus generation method provided in the embodiments of this application;

[0036] Figure 3FThis is an optional flowchart illustrating the question corpus generation method provided in the embodiments of this application;

[0037] Figure 4 This is an optional symbol meaning diagram provided in an embodiment of this application;

[0038] Figure 5 This is an optional question corpus generation diagram provided in an embodiment of this application. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0040] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0041] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described in this embodiment can be implemented in an order other than that illustrated or described in this embodiment.

[0042] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0043] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0044] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0045] (1) Knowledge base question answering (KBQA): Intelligent question answering that combines natural language understanding and knowledge graph technology.

[0046] (2) Grammatical attributes: including at least one of the following: the named entity attribute of the entity word (such as personal name, institution name and university name, etc.), the part of speech of the entity word (such as noun, verb and preposition, etc.), and the grammatical structure of the entity word in the user's question (such as subject, predicate and object, etc.).

[0047] (3) Knowledge graph: It is used to describe various entities and concepts that exist in the real world, as well as the relationships between various entities and concepts. It is usually built using structures such as “entity-relationship-entity” and “entity-attribute-attribute value”. In this case, “entity-relationship-entity” is equivalent to a piece of knowledge in the knowledge graph, and “entity-attribute-attribute value” is similar.

[0048] (4) Entity words: also known as named entities, refer to entities with specific meanings in a statement, such as personal names, place names, organization names, and proper nouns. Entity words in a statement are usually identified through Named Entity Recognition (NER) technology.

[0049] (5) Question corpus: User questions used as language materials.

[0050] (6) Semantic parsing: The process of converting natural language into a logical form that machines can understand.

[0051] (7) Triple (SPO, Subject-Predicate-Object): A knowledge representation in a knowledge graph.

[0052] KBQA primarily combines natural language understanding and knowledge graph technologies for intelligent question answering. Natural language understanding involves semantic understanding, semantic parsing, and semantic representation, while knowledge graphs are used for knowledge retrieval, reasoning, and answer generation. The semantic parsing in question answering involving knowledge graphs typically requires modeling the question as a triple, utilizing the structure of the knowledge graph database for complex retrieval, such as single-hop or multi-hop queries. For example, the triple for "What is Zhang San's height?" is (Subject: Zhang San, Predicate: height, Object: x), abbreviated as SPO.

[0053] If it is necessary to evaluate the knowledge-based question-answering capabilities of a question-answering system, it is necessary to establish a question set with broad question coverage, that is, to generate question corpora with different question formats. Among the related technologies, the methods for mining question corpora include: (1) Crowdsourcing: brainstorming with people of different identities to manually compile question corpora for knowledge-based question-answering. (2) Online data sampling: finding an online system with massive amounts of data and sampling question corpora from the online system. (3) Website crawling: finding a question-answering website and crawling relevant question corpora from the website.

[0054] The related technical solutions have the following problems: (1) Manual compilation relies heavily on each person's prior knowledge, often getting stuck in one's own knowledge system. The compiled question corpus relies heavily on personal prior knowledge, with a single question format and narrow coverage. (2) The current comprehension ability of knowledge-based question answering is relatively simple, and the covered question formats are also relatively few. Many truly complex question corpora cannot be covered. (3) The question corpus crawled from websites also has the problem of small coverage.

[0055] This application provides a method, apparatus, device, and computer-readable storage medium for generating question corpora. By acquiring a keyword group and at least one target question expression corresponding to the keyword group, and generating at least one corresponding question corpus based on the keyword group and the at least one target question expression, multiple question corpora corresponding to the keyword group can be automatically generated, improving the efficiency of question corpus generation. Furthermore, since different question types are represented through target question expressions, the question corpora generated through different target question expressions have a wider range of question types covered. The exemplary application of the electronic device provided in this application embodiment is described below.

[0056] See Figure 1 , Figure 1 This is an optional architecture diagram of the question corpus generation system 100 provided in the embodiments of this application. In order to support a question corpus generation application, the terminal 400 (terminal 400-1 and terminal 400-2 are shown as examples) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. Figure 1 It is also shown that server 200 can be a server cluster, which includes servers 200-1 to 200-3. Similarly, servers 200-1 to 200-3 can be physical machines or virtual machines built using virtualization technologies (such as container technology and virtual machine technology). This application embodiment does not limit this. Of course, in this embodiment, a single server can also be used to provide services.

[0057] In some embodiments, server 200 is configured to receive a question corpus generation instruction sent by terminal 400 via network 300. In response to the instruction, server 200 acquires a keyword group, which includes at least two keywords; acquires at least one target question expression corresponding to the keyword group; the target question expression is used to connect at least two keywords using at least one connection symbol; and generates at least one corresponding question corpus based on the keyword group and the at least one target question expression. Server 200 sends the generated question corpus to terminal 400. The keyword group can be pre-stored in server 200, or it can be carried in the question corpus generation instruction and sent by terminal 400 to server 200.

[0058] In some embodiments, the terminal 400 is used to send a question corpus generation instruction to the server 200 via the network 300. This instruction instructs the server 200 to obtain a keyword group, which includes at least two keywords; obtain at least one target question expression corresponding to the keyword group; the target question expression is used to connect at least two keywords using at least one connecting symbol; and generate at least one corresponding question corpus based on the keyword group and the at least one target question expression. The generated question corpus is then sent to the terminal 400. After receiving the question corpus, the terminal 400 can display the question corpus on a graphical interface 410 (graphical interfaces 410-1 and 410-2 are shown exemplarily).

[0059] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the question corpus generation device 500 provided in the embodiments of this application. Figure 2 The question corpus generation device 500 shown includes at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the question corpus generation device 500 are coupled together via a bus system 540. It is understood that the bus system 540 is used to implement communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 540.

[0060] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0061] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0062] Memory 550 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory. Memory 550 may optionally include one or more storage devices physically located away from processor 510.

[0063] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0064] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0065] The network communication module 552 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0066] Display module 553 is configured to enable the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with user interface 530.

[0067] The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0068] In some embodiments, the question corpus generation device provided in this application can be implemented in a combination of hardware and software. As an example, the question corpus generation device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the question corpus generation method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0069] In some embodiments, the question corpus generation apparatus provided in this application can be implemented in software. Figure 2 A question corpus generation device 555 stored in memory 550 is shown. It can be software in the form of programs and plug-ins, including the following software modules: a first acquisition module 5551, a second acquisition module 5552, a generation module 5553, a labeling module 5554, and a testing module 5555. These modules are logically related and can therefore be arbitrarily combined or further split according to the functions they implement.

[0070] The functions of each module will be explained below.

[0071] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the question corpus generation method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0072] The question corpus generation method provided in this application will be described in conjunction with exemplary applications and implementations of the server provided in the embodiments of this application.

[0073] See Figure 3A , Figure 3A This is an optional flowchart illustrating the question corpus generation method provided in this application embodiment, which will be combined with... Figure 3A The steps shown are explained.

[0074] In step 101, a keyword group is obtained; the keyword group includes at least two keywords.

[0075] In some embodiments, the following methods can be used to obtain keyword groups: (1) Select any keyword group that needs to be converted into question corpus in a database that stores multiple keyword groups; (2) Select any knowledge that needs to be converted into question corpus in the knowledge graph, and use the triple information contained in the knowledge as keyword group; (3) parse an existing question corpus, extract the keywords in it, and form the corresponding keyword group according to the grammatical attributes of each keyword; for example, for the question corpus "Is Zhang San's height 170?", the entity words involved in the question corpus can be extracted first through NER, such as "Zhang San" in the question corpus above, and the obtained entity words are used as the keywords corresponding to the subject category. After entity extraction, attribute inference needs to be made by combining the entity words and the meaning of the whole sentence. Here, the keyword corresponding to the relation category is inferred to be "height"; finally, the keyword corresponding to the object category is determined to be "170".

[0076] In some embodiments, keywords have keyword category attributes, and each keyword can be identified as a subject category, a relation category, and an object category. Taking the structure of a triple in a knowledge graph, such as "entity 1-relation-entity 2" or "entity 1-attribute-attribute value", the keyword corresponding to the subject category can be "entity 1", the keyword corresponding to the relation category can be "relation" or "attribute", and the keyword corresponding to the object category can be "entity 2" or "attribute value".

[0077] In some embodiments, a keyword group may include at least two keywords, which may belong to the same keyword category or different keyword categories. This application does not limit this.

[0078] For example, the keyword group can include the subject category keyword "Zhang San" and the relation category keyword "height". The question corpus generated based on the keyword group can be "How tall is Zhang San?". The keyword group can also include two subject category keywords "flower" and "grass". The question corpus generated based on the keyword group can be "What common attributes do flowers and grass have?".

[0079] In step 102, at least one target question expression corresponding to the keyword group is obtained; the target question expression is used to connect at least two keywords based on at least one connecting symbol.

[0080] In some embodiments, the following methods can be used to obtain at least one target question expression corresponding to a keyword group: (1) By analyzing the attribute / category information of at least two keywords in the preset keyword group, at least one question expression matching the keyword group is found among a plurality of preset question expressions as the at least one target question expression; (2) At least two keywords in the keyword group are displayed through the display interface, and at least one connecting symbol is also displayed. The user performs operations on the at least two keywords and at least one connecting symbol through the interactive device (e.g., click operation, drag operation and long press operation) to arrange the at least two keywords and at least one connecting symbol in a certain order. The system generates a target question expression corresponding to the keyword group based on the relative position between the at least two keywords and at least one connecting symbol. Repeating the above operations can yield a plurality of target question expressions; (3) At least one target question expression input by the user for the keyword group is received through the interactive device.

[0081] The target question expression consists of keywords and connectors. In its simplest form, two keywords can be connected by a connector to form a target question expression. For example, as shown in Table 1 below.

[0082] Keyword 1 Connection symbol Keyword 2

[0083] Table 1

[0084] In some embodiments, the target question expression is used to connect at least two keywords using at least one connector. For each pair of keywords, there is at least one connector. Different target question expressions can be formed by connecting the same two keywords using connectors with different meanings; correspondingly, the questions generated from different target question expressions containing connectors with different meanings will have different expected meanings.

[0085] For example, for the same set of keywords "keyword 1: 5G package", "keyword 2: tariff", and "keyword 3: 30 yuan", if " "(Keyword 1 + Keyword 2) > Keyword 3" is used as the corresponding target question expression, which means "Are all 5G package fees greater than 30 yuan?"; if " "(Keyword 1 + Keyword 2) > Keyword 3" as the corresponding target question expression means "Are there 5G data plans with a price greater than 30 yuan?" It can be seen that for different connective symbols... "and" The meanings of the corresponding connecting symbols are different, which in turn affects the meaning of the question corpus.

[0086] In some embodiments, the connector can also be placed before a keyword to modify that keyword. Similarly, the connector can also be placed before a set of multiple keywords to modify the set or to perform operations on the multiple keywords within the set.

[0087] For example 1, the connecting symbol “¬” means “not”, indicating negation. Therefore, when generating the question “How tall is Li Si? He is not that host”, the connecting symbol “¬” can be placed before the keyword “host” to indicate “not that host”. For example 2, the connecting symbol “Σ” means “sum”, indicating calculation of the total. Therefore, when generating the question “How many children does Wang Wu have?”, the connecting symbol “Σ” can be placed before the keywords “Wang Wu” and “children” to indicate “calculation of the number of Wang Wu’s children”.

[0088] In step 103, at least one corresponding question corpus is generated based on the keyword group and at least one target question expression.

[0089] In some embodiments, each keyword in the keyword group can be substituted into the corresponding position in the target question expression, and the meaning of each connecting symbol in the target question expression can be combined to generate the corresponding question corpus.

[0090] For example, taking the keyword group "keyword 1: 5G package", "keyword 2: tariff" and "keyword 3: 30 yuan" as an example, the target question expression corresponding to this keyword group is " "(Keyword 1 + Keyword 2) > Keyword 3", in the process of generating the question corpus, the keywords in the target question expression can be replaced with the keywords in the keyword group, and each connecting symbol can be replaced with its corresponding meaning. In this example, the connecting symbol "()" indicates that the meaning of "Keyword 1 + Keyword 2" should be determined first, where "+" indicates a modification relationship, and the meaning of "Keyword 1 + Keyword 2" is "5G package (of) tariff". The meaning of “” is “all / all / all”, and the meaning of “>” is “greater than”; correspondingly, the generated question corpus can be at least one of the following: “Are all 5G packages more than 30 yuan?”, “Are all 5G packages more than 30 yuan?”, and “Are all 5G packages more than 30 yuan?”

[0091] Through the embodiments of this application, for Figure 3AAs can be seen from the above exemplary implementation, the embodiments of this application can automatically generate multiple question corpora corresponding to the keyword group by obtaining the keyword group and at least one target question expression corresponding to the keyword group; and generate at least one question corpus corresponding to the keyword group based on the keyword group and at least one target question expression, thereby improving the generation efficiency of question corpora. Furthermore, since different question types are represented by target question expressions, the question types covered by question corpora generated by different target question expressions are wider.

[0092] In some embodiments, see Figure 3B , Figure 3B This is an optional flowchart illustrating the question corpus generation method provided in this application embodiment, based on... Figure 3A , Figure 3A The step 102 shown can be updated to steps 201 to 202.

[0093] In step 201, the phrase information of the keyword group is obtained; the phrase information includes at least one of the following: keyword information and connection symbol information; the keyword information includes the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category; the connection symbol information includes the connection symbol category corresponding to at least one keyword category in the keyword group.

[0094] In some embodiments, the keyword categories may include: subject category, relation category, and object category.

[0095] In some embodiments, the phrase information of a keyword group may include keyword information; the phrase information of a keyword group may include connector information; the phrase information of a keyword group may also include both keyword information and connector information. Specifically, the keyword information includes the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category, and the connector information includes the connector category corresponding to at least one keyword category in the keyword group.

[0096] For example 1, taking the keyword group "Keyword 1: 5G package", "Keyword 2: tariff", and "Keyword 3: 30 yuan" as an example, Keyword 1 belongs to the subject category, Keyword 2 belongs to the relation category, and Keyword 3 belongs to the object category. Therefore, the keyword information corresponding to this keyword group is "3 keyword categories, 1 keyword corresponding to the subject category, 1 keyword corresponding to the relation category, and 1 keyword corresponding to the object category". For example 2, taking the keyword group "Keyword 1: Mount Hua", "Keyword 2: Mount Tai", and "Keyword 3: altitude" as an example, Keyword 1 and Keyword 2 belong to the subject category, and Keyword 3 belongs to the relation category. Therefore, the keyword information corresponding to this keyword group is "2 keyword categories, 2 keywords corresponding to the subject category, and 1 keyword corresponding to the relation category". Since the keywords "Mount Hua" and "Mount Tai" in the subject category are in a parallel relationship, the connection symbol information corresponding to this keyword group is "the connection symbol corresponding to the subject category is in a parallel category".

[0097] In step 202, at least one target question expression is obtained from a preset question expression set based on the phrase information; the question expression set includes multiple question expressions.

[0098] In some embodiments, step 202 may include: searching for a question expression that matches the keyword information in a preset question expression set as the at least one target question expression; step 202 may also include: searching for a question expression that matches the connection symbol information in a preset question expression set as the at least one target question expression; step 202 may also include: simultaneously searching for a question expression that matches both the keyword information and the connection symbol information in a preset question expression set as the at least one target question expression.

[0099] Through the embodiments of this application, for Figure 3B As can be seen from the above exemplary implementation, the embodiments of this application can obtain more comprehensive and wider-ranging target question expressions by acquiring at least one of the keyword information and connection symbol information, and by searching for the target question expression corresponding to the keyword group through the keyword information, thereby improving the question range of the question corpus; at the same time, it can make the target question expression more matched with the keyword group, and the generated question corpus more realistic.

[0100] In some embodiments, see Figure 3C , Figure 3C This is an optional flowchart illustrating the question corpus generation method provided in this application embodiment, based on... Figure 3BStep 201 specifically includes steps 301 and 302.

[0101] In step 301, if the phrase information includes keyword information, the keyword category of each keyword in the keyword group is obtained; the number of keyword categories and the number of keywords corresponding to each keyword category are determined based on the keyword category of each keyword.

[0102] In some embodiments, where phrase information includes keyword information, the keyword category of each keyword can be determined based on the attribute information corresponding to each keyword in the knowledge graph.

[0103] For example, for the keyword group "keyword 1: Huashan", "keyword 2: Taishan" and "keyword 3: height", the knowledge graph can determine that keyword 1 and keyword 2 belong to the subject category, and keyword 3 belongs to the relation category. Therefore, the keyword categories include the subject category and the relation category, with a corresponding number of 2 keyword categories. The number of keywords corresponding to the subject category is 2, and the number of keywords corresponding to the relation category is 1.

[0104] In step 302, when the phrase information includes connection symbol information, if the number of keywords corresponding to the subject category is at least two, the connection symbol category corresponding to the subject category is determined based on the attribute information of the keywords corresponding to the subject category; if the number of keywords corresponding to the relation category is at least two, the connection symbol category corresponding to the predicate category is determined based on the attribute information of the keywords corresponding to the relation category; if the number of keywords corresponding to the object category is at least one, the connection symbol category corresponding to the object category is determined based on the attribute information of the keywords corresponding to the object category.

[0105] In some embodiments, where the phrase information includes connector information, the number of keywords corresponding to each keyword category is first determined, i.e., the number of keywords corresponding to the subject category, the number of keywords corresponding to the relation category, and the number of keywords corresponding to the object category. The connector category corresponding to each keyword category is then determined based on the number of keywords corresponding to each keyword category.

[0106] Where there are at least two keywords corresponding to the subject category, the category of the connecting symbol corresponding to the subject category is determined based on the attribute information of the keywords corresponding to the subject category. The category of the connecting symbol corresponding to the subject category includes at least one of the following: parallel category and modifier category.

[0107] For example, regarding the keywords "Huashan" and "Taishan" corresponding to the subject category, it can be seen that there are two keywords corresponding to the subject category, and the corresponding connecting symbol category is the parallel category; regarding the keywords "host" and "Zhang San" corresponding to the subject category, it can be seen that there are two keywords corresponding to the subject category, and the corresponding connecting symbol category is the modifier category, where "host" modifies "Zhang San," meaning "Zhang San as the host."

[0108] In cases where there are at least two keywords corresponding to a relation category, the connection symbol category corresponding to the relation category is determined based on the attribute information of the keywords. The connection symbol category corresponding to the relation category includes at least one of the following: parallel category and modifying category. For example, for the keywords "classmate" and "father" corresponding to the relation category, it can be seen that there are two keywords corresponding to the relation category, and its corresponding connection symbol category is the modifying category; "classmate" modifies "Zhang San's father," meaning "classmate's father." For the keywords "size" and "weight" corresponding to the relation category, it can be seen that there are two keywords corresponding to the relation category, and its corresponding connection symbol category is the parallel category.

[0109] In cases where there is at least one keyword corresponding to the object category, the connection symbol category corresponding to the object category is determined based on the attribute information of the keyword corresponding to the object category. The connection symbol category corresponding to the object category includes at least one of the following: single element category, set element category, and domain category.

[0110] Through the embodiments of this application, for Figure 3C As can be seen from the above exemplary implementation, this application can obtain target question expressions with more complete question types and a wider coverage, thereby improving the question type range of the question corpus; at the same time, it can make the target question expression match the keyword group better, and the generated question corpus is more realistic.

[0111] In some embodiments, see Figure 3D , Figure 3D This is an optional flowchart illustrating the question corpus generation method provided in this application embodiment, based on... Figure 3B Step 202 specifically includes steps 401 and 402.

[0112] In step 401, if the phrase information includes keyword information, the expression information corresponding to each question expression in the multiple question expressions is determined; the expression information corresponding to each question expression is matched according to the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category to obtain at least one target question expression.

[0113] In this embodiment, when the phrase information includes keyword information, the number of keyword categories corresponding to each question expression and the number of keywords corresponding to each keyword category are obtained. Question expressions that match the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category are selected as at least one target question expression.

[0114] In step 402, when the phrase information includes keyword information and connector information, the expression information corresponding to each question expression in the multiple question expressions is determined; the expression information corresponding to each question expression is matched according to the number of keyword categories in the keyword group, the number of keywords corresponding to each keyword category, and the connector category corresponding to at least one keyword category, so as to obtain at least one target question expression.

[0115] In this embodiment, when the phrase information includes keyword information and connector information, the number of keyword categories corresponding to each question expression, the number of keywords corresponding to each keyword category, and the connector information are obtained. Question expressions that match the number of keyword categories in the keyword group, match the number of keywords corresponding to each keyword category, and also match the connector information are selected as at least one target question expression.

[0116] Through the embodiments of this application, for Figure 3D As can be seen from the above exemplary implementation, the embodiments of this application can obtain target question expressions with more complete question types and a wider coverage, thereby improving the question type range of the question corpus; at the same time, it can make the target question expression match the keyword group better, and the generated question corpus is more realistic.

[0117] In some embodiments, see Figure 3E , Figure 3E This is an optional flowchart illustrating the question corpus generation method provided in this application embodiment, based on... Figure 3A After step 103, step 501 may also be included.

[0118] In step 501, annotations are added to each question corpus; the annotations include at least one of the following: keyword groups corresponding to the question corpus, target question expression corresponding to the question corpus, and keyword tags corresponding to each keyword in the question corpus; in the process of generating a corpus test set based on the annotated question corpus, test corpus with the same annotations are removed; the corpus test set includes multiple annotated test corpus.

[0119] In this embodiment, each question corpus is annotated simultaneously with its generation. That is, the generated question corpus carries corresponding annotations. These annotations include at least one of the following: keyword groups corresponding to the question corpus, the target question expression corresponding to the question corpus, and keyword tags corresponding to each keyword in the question corpus.

[0120] In some embodiments, different annotation information can be selected when adding annotations, depending on the evaluation dimension. When it is necessary to evaluate the question phrasing dimension, the target question expression corresponding to the question corpus can be added as the annotation of the question corpus. When it is necessary to evaluate the coverage dimension, the keyword group corresponding to the question corpus can be added as the annotation of the question corpus. When it is necessary to evaluate the question error tolerance, the keyword tag corresponding to each keyword in the question corpus can be added as the annotation of the question corpus.

[0121] In some embodiments, the keyword tags include a first keyword tag corresponding to a keyword belonging to the subject category, a second keyword tag corresponding to a keyword belonging to the relation category, and a third keyword tag corresponding to a keyword belonging to the object category; the first keyword tag includes at least one of the following: a polysemous tag, an alias tag, a misspelling tag, and a missing word tag; the second keyword tag includes at least one of the following: a polysemous tag, an alias tag, and an implicit tag; the third keyword tag includes at least one of the following: a length tag, an amount tag, a time tag, a temperature tag, a volume tag, and a character tag.

[0122] In some embodiments, for multiple annotated question corpora generated from the keyword groups, the number of corresponding question corpora will increase as the number of keyword groups increases. The question corpora generated from a large number of keyword groups will be used as a test set to test the question-answering system. Specifically, if different question corpora in the test set have the same annotations, it indicates that the question corpora with the same annotations are completely equivalent, and only one of them needs to be retained.

[0123] Through the embodiments of this application, for Figure 3E As can be seen from the above exemplary implementation, the embodiments of this application, by annotating each question corpus, can delete question corpora with the same annotations when generating the corpus evaluation set, thereby enabling the obtained corpus evaluation set to cover a wider range of question types with a minimum number of question corpora. This improves the testing capability of the corpus test set for question-answering systems.

[0124] In some embodiments, see Figure 3F , Figure 3F This is an optional flowchart illustrating the question corpus generation method provided in this application embodiment, based on... Figure 3EStep 601 may also be included after step 501.

[0125] In step 601, the question-answering system is tested based on multiple annotated test corpora in the corpus test set; at least one target test corpus with abnormal test results is obtained; and the optimization direction of the question-answering system is determined based on the annotations corresponding to at least one target test corpus.

[0126] In some embodiments, when testing the question-answering system based on the corpus test set, the question corpus whose question-answering results are abnormal can be used as target test corpus based on the system's response to each question corpus, thereby obtaining at least one target test corpus.

[0127] In some embodiments, since each question corpus in the test corpus contains corresponding annotations, the annotations include at least one of the following: keyword groups corresponding to the question corpus, target question expressions corresponding to the question corpus, and keyword tags corresponding to each keyword in the question corpus. Therefore, at least one keyword group corresponding to the target test corpus, the target question expression, and the keyword tags corresponding to each keyword can be statistically analyzed, and the keyword groups, target question expressions, and keyword tags corresponding to each keyword with the highest frequency can be used as the optimization direction for the question answering system.

[0128] For example, in at least one keyword group corresponding to the target test corpus, if keyword group 1 appears 5 times, keyword group 2 appears 3 times, and all other keyword groups appear 1 time, then keyword groups 1 and 2 are considered logical vulnerabilities in the question-answering system, and keyword groups 1 and 2 are output as optimization directions for the question-answering system. In the target question expression corresponding to at least one target test corpus, " The number of occurrences of "(S1+P1)>O1" is 5, and the number of occurrences of "(O1+O2+O3)" is also 5. The number of occurrences of "(S1+P1)" is 4, and the number of occurrences of the other target question expressions is 1. Therefore, it is determined that " (S1+P1)>O1” and (O1+O2+O3) (S1+P1)” is a logical flaw in this question-and-answer system. (S1+P1)>O1” and (O1+O2+O3) (S1+P1)” is output as the optimization direction of the question-answering system. If the number of misspelled tags is 15 and the number of missing tags is 10 in the keyword tags corresponding to at least one target test corpus, and the number of other target question expressions is less than 5, then the misspelled tags and missing tags are judged as logical vulnerabilities of the question-answering system, and the misspelled tags and missing tags are output as the optimization direction of the question-answering system.

[0129] Through the embodiments of this application, for Figure 3F As can be seen from the above exemplary implementation, the corpus test set provided in this application embodiment can identify logical vulnerabilities of the question-answering system in three dimensions: keyword groups, target question expressions, and keyword tags during the testing process, thereby improving the testing efficiency of the corpus test set.

[0130] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0131] This application proposes a method for constructing a knowledge question corpus (question corpus) evaluation set by combining SPOs (Special Points of Interest) with a finite set of symbolic descriptions. The question corpus obtained through this method is more comprehensive, and the corresponding evaluation set can more scientifically measure the differences in capabilities across different dimensions of a knowledge question-answering system. Specifically, a finite set of connection symbols is derived from the combinations of SPOs in the question corpus, and the question corpus is represented through different combinations of these symbols.

[0132] KBQA (knowledge-based question answering) is an intelligent question answering system that combines natural language understanding and knowledge graph technologies. Natural language understanding involves semantic understanding, semantic parsing, and semantic representation, while knowledge graphs are used for knowledge retrieval, reasoning, and answer generation. The semantic parsing involved in knowledge graph-based question answering typically requires modeling the question as a triple, utilizing the structure of the knowledge graph database for complex retrieval, such as single-hop or multi-hop queries. For example, the triple for "What is Zhang San's height?" is (Subject: Zhang San, Predicate: height, Object: x), abbreviated as SPO. More complex questions exist, such as "Which is taller, Mount Hua or Mount Tai?" or "Is there a province in China with an area greater than 400,000 square kilometers?"

[0133] The technical problem this application aims to solve is: how to organize a comprehensive set of questions to evaluate the knowledge-based question-answering (KBQA) capabilities of a system. For example, how to develop a better approach to question modeling? Without a reasonable ability to model a question evaluation set, the evaluation set may only ever contain a small portion of the KBQA question world. Knowledge is infinite, and the knowledge-based questions involved in KBQA seem infinite as well. Faced with this endless array of questions, the evaluation set cannot be infinite, and the final evaluation set often deviates from the real world because the concepts humans can understand and the channels through which evaluation sets are formed are limited. Nevertheless, we hope to grasp the infinite with the finite, to contain infinite facts with a few rules, and to capture the infinitely rich unknown from finite deductions. We hope to use a formal language, through a finite set of descriptions, to generate infinitely possible questions.

[0134] In related technologies, methods for mining question corpora may include: (1) Crowdsourcing: brainstorming with people of different backgrounds to manually compile question corpora for knowledge-based Q&A. (2) Online data sampling: finding an online system with massive amounts of data and sampling question corpora from the online system. (3) Website crawling: finding a question-and-answer website and crawling relevant question corpora from the website.

[0135] The related technical solutions have the following problems: (1) Manual writing relies heavily on each person's prior knowledge and often gets stuck in one's own knowledge system, making it impossible to compile a comprehensive collection of question corpora with all question types and wide coverage; (2) The current understanding of knowledge Q&A is relatively simple, and the number of question types covered is also relatively small, so many truly complex question corpora cannot be covered; (3) The question corpora crawled from websites also have a small coverage and cannot determine which question corpora have not been covered.

[0136] This application's embodiments decompose the question into SPOs (Single Point of Interest), resulting in multiple keywords. The relationships between these keywords are then represented using connecting symbols. By summarizing a finite set of expressions, an infinite corpus of question sentences is mined. This not only generates a comprehensive and wide-ranging evaluation set but also provides directional guidance for manually compiled question sentences. Furthermore, the evaluation set can be used to determine which dimensions of the question-answering system have missing data.

[0137] This application can be applied to various projects and products, including smart speakers, smart TV boxes, online voice interaction systems, smart voice assistants, and in-vehicle smart voice devices. It can build a reasonable and high-quality knowledge question-and-answer evaluation set to evaluate these products and competitors, while simultaneously driving the optimization and improvement of one's own smart products.

[0138] In typical application scenarios, such as smart speakers, the question corpus generated by this application can reveal which dimensions of question-answering capabilities the smart speaker lacks. For example, if it is found that the smart speaker cannot support questions involving "union," this can guide the product to conduct in-depth optimization in "union" question answering. For instance, questions containing "equals," such as "Is Zhang San's height 1.7 meters?", can not be answered through evaluation. This reveals that the question-answering process combined with the knowledge graph lacks the "equals" judgment logic, thus driving further product optimization.

[0139] The embodiments of this application include the following aspects:

[0140] (1) Constructing the SPO connection method;

[0141] (2) Construct a labeling system for each SPO;

[0142] (3) Corpus writing based on expressions;

[0143] (4) Output the evaluation set.

[0144] Among them, for (1) constructing the SPO connection method, the embodiments of this application include:

[0145] In one embodiment, the SPO connection method can include the connection between Sbj (keywords of the subject category) and Pred (keywords of the relation category). In one embodiment, KBQA uses SPO for semantic representation. Sbj can be one or more, such as "Who are Zhang San's and Li Si's wives respectively?" or "How tall is host Wang Wu?" Similarly, Pred can also be one or more, such as "What are the names of Zhang San's wife and child?" or "What is the name of Zhang San's wife's mother?"

[0146] In one embodiment, there are two types of relationships between multiple Sbj: one is modification (category), and the other is parallel (category). The relationships between multiple Pred can also be summarized into these two types. For modification, the connection method can be represented by the "*" symbol; for parallel, the connection method can be represented by the "+" symbol. Parallel Sbj and / or Pred can also be combined to form Obj.

[0147] In one embodiment, the relationship between Sbj and Pred can be represented by the "+" symbol.

[0148] In one embodiment, in addition to the relationship between Sbj and Pred, each Sbj or Pred can be negativeed by adding the symbol "¬". For example, the question corpus "What is the height of Li Si (S1: Li Si) (P1: height) and not that host (S2: host)" is represented as: (S1*¬S2)+P1.

[0149] Therefore, the connection between Sbj (keyword of the subject category) and Pred (keyword of the relation category) can be represented by a combination of "*, +, ¬".

[0150] In one embodiment, the SPO connection method may further include the Obj connection method. The Obj connection method may include: the connection method for single-element category Obj, the connection method for collection element category Obj, and the connection method for domain category Obj.

[0151] In one embodiment, for the concatenation of a single-element category Obj, when Obj (the answer) is a single element, the symbol "" can be used. "" indicates the connection method. For example, "Are the 5G package (S1: 5G package) tariff (P1: tariff) all greater than 30 yuan (O1: 30 yuan)?", the corresponding expression is: "(S1+P1)>O1".

[0152] In one embodiment, for the concatenation method of the set element category Obj, when Obj (the answer) is a set, the symbol "" can be used. "f(x)" represents the connection method. For example, "What foods (S1: food) can be both cooked (P1: cooking) and eaten as fruit (P2: fruit)?" corresponds to the expression "(S1+P1)U(S1+P2)". In one embodiment, f(x) is a function symbol. Common function symbols include: max, min, topk, etc. For example, "Which is higher, Mount Hua (S1: Mount Hua) or Mount Tai (S2: Mount Tai) (P1: height)?" is expressed as max(S1+P1, S2+P1).

[0153] In one embodiment, for the connection method of the domain category Obj, when Obj (the answer) is a certain domain, the symbol " , , , "Indicates the connection method. For example, for the questions 'Is the fee of the 5G package (S1: 5G package) (P1: fee) greater than 30 yuan (O1: 30 yuan)?' and 'Is there any fee of the 5G package (S1: 5G package) (P1: fee) greater than 30 yuan (O1: 30 yuan)?', it is unreasonable to use '(S1 + P1) > O1' for both. The former refers to whether all fees are greater, while the latter only refers to whether there exists one. Therefore, using the symbols corresponding to the above domain categories, the expression corresponding to 'Is the fee of the 5G package (S1: 5G package) (P1: fee) greater than 30 yuan (O1: 30 yuan)?' can be set as '(S1 + P1) > 01'; and the expression corresponding to 'Is there any fee of the 5G package (S1: 5G package) (P1: fee) greater than 30 yuan (O1: 30 yuan)?' can be set as '(S1 + P1) > O1'.

[0154] In one embodiment, for (2) constructing the respective label systems for SPO. Since the extraction of spo depends on different capabilities (Sbj mainly depends on NER extraction, Pred mainly depends on attribute inference, etc., and Obj more depends on the knowledge graph), and the types of different spo also determine the capabilities of the entire question and answer, therefore, this application defines labels to distinguish the types of each spo.

[0155] For example, for the question corpus 'What is Zhang San's height?', the processing logic of a question and answer system is to first identify and extract the entity words involved in the question, such as 'Zhang San' in the above question corpus, and use the obtained entity words as the keywords corresponding to the subject category. Here, entity extraction is NER extraction; after entity extraction, it is necessary to combine the entity words and the meaning of the whole sentence for attribute inference. Here, the keyword corresponding to the relationship category inferred is 'height'; finally, combine the entity and the attribute to construct a query statement for the knowledge graph and query out 173 cm.

[0156] In one embodiment, for the keywords corresponding to the subject category, the corresponding labels may include polysemy labels, alias labels, misspelling labels, and missing character labels, etc. Among them, the polysemy label indicates that there are multiple keywords corresponding to this subject category on the knowledge graph. For example, for 'Zhang San', there may be a host named 'Zhang San' and an actor named 'Zhang San'; the alias label indicates that the keyword corresponding to the subject category is an alias of its corresponding entity. For the entity 'Li Si', the keywords with alias labels corresponding to it are 'Lao Li', 'Li Laosi', etc.; the misspelling label indicates that the keyword corresponding to the subject category is a misspelled form of its corresponding entity. For the entity 'furniture', the keyword with a misspelling label corresponding to it is 'jiāju'; the missing character label indicates that the keyword corresponding to the subject category is a form with a missing character of its corresponding entity. For the entity 'thermos cup', the keyword with a missing character label corresponding to it is 'wēnbēi'.

[0157] In one embodiment, for keywords corresponding to a relationship category, the corresponding tags may include polysemous tags, alias tags, and implicit tags. Alias ​​tags indicate that the keyword corresponding to this relationship category has multiple meanings in the graph; for example, in "What is Zhang San's height?", the keyword "height" can be an alias for "body height". Polysemous tags indicate that the keyword corresponding to this relationship category has multiple meanings; for example, in "What works has Zhang San created?", the keyword "works" can mean either "film and television works" or "musical works". Implicit tags indicate that the keyword corresponding to this relationship category does not explicitly state an attribute relationship; for example, in "Is Zhang San 1.7 meters tall?", although the attribute "height" is not explicitly asked, the idea is implicitly expressed.

[0158] In one embodiment, for keywords corresponding to object categories, the corresponding tags may include length tags, amount tags, time tags, temperature tags, volume tags, and character tags, etc.

[0159] In one embodiment, (3) is corpus writing based on expressions.

[0160] Please refer to Figure 4 The illustrated embodiment of this application provides an optional symbol meaning diagram, which includes the symbol meaning of the connecting symbols. For example, 1, in Figure 4 The conjunction “×” in the sentence structure represents multiplication, indicating modification. Therefore, when generating the question corpus “Who is Zhang San’s wife?”, the conjunction “×” can be used to connect the keywords “Zhang San” and “wife” to represent “Zhang San’s wife”. For example, in 2… Figure 4 The connecting symbol “¬” in the text means “not”, indicating negation. Therefore, when generating the question corpus “How tall is Li Si? He is not that host”, the connecting symbol “¬” can be placed before the keyword “host” to indicate “not that host”. For example, in 3, Figure 4 The connecting symbol “Σ” in the text means “summation”, which means to calculate the total number. Therefore, when generating the question corpus “How many children does Wang Wu have?”, the connecting symbol “Σ” can be placed before the keywords “Wang Wu” and “children” to indicate “the calculation of the number of Wang Wu’s children”.

[0161] Please refer to Figure 5 The illustrated embodiment of this application provides an optional question corpus generation diagram, which exemplarily shows the correspondence between different keyword groups combined with different target expressions and the question corpus. For example, in 1, Figure 5The keyword phrase "the keyword for the subject category is China, the keyword for the relation category is the capital, and the keyword for the object category is Beijing" is expressed as "If the target question expression is used, it would be " (S1+P1=O1)”, where “ The meaning of "is" is "existence", and the generated question corpus could be "Is Beijing the capital of China?". For example, in 2, Figure 5 In the keyword phrase "the subject category keyword is food, and the relation category keywords are cooking and fruit", if the target question expression is "(S1+P1)∪(S1+P2)", where "∪" means "union", the generated question corpus could be "What foods can be both cooked and eaten as fruit?". For example, in 3, Figure 5 The keyword phrase "the subject category keyword is Sichuan Province, the relation category keyword is tourist attractions, and the object category keyword is Jiuzhaigou, Emei Mountain, and Daocheng Yading" can be expressed as "(O1+O2+O3)" in the target question form. (S1+P1)”, where “ The meaning of "belongs to" is "belongs to". The generated question corpus can be "Are Jiuzhaigou, Emei Mountain and Daocheng Yading all tourist attractions in Sichuan Province?"

[0162] The embodiments of this application achieve the following technical effects: In traditional KBQA evaluation, the evaluation set for KBQA is either crawled from the internet or mined from online resources. However, neither method can act as a guide to improve the ability to answer higher-order questions in KBQA. This application, starting from the logic of formal languages, models the KBQA question corpus as a representation problem of triples, and summarizes the limited connection symbols and methods between elements of triples. In this way, higher-order questions in KBQA can be constructed from another dimension.

[0163] The following continues to describe the exemplary structure of the question corpus generation device 555 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software module stored in the question corpus generation device 555 in the memory 550 may include:

[0164] The first acquisition module 5551 is used to acquire keyword groups, which include at least two keywords.

[0165] The second acquisition module 5552 is used to acquire at least one target question expression corresponding to the keyword group; the target question expression is used to connect at least two keywords according to at least one connector symbol;

[0166] The generation module 5553 is used to generate at least one corresponding question corpus based on the keyword group and at least one target question expression.

[0167] In some embodiments, the keyword category includes at least one of the following: subject category, relation category, and object category.

[0168] In some embodiments, the second acquisition module 5552 is further configured to acquire phrase information of the keyword group; the phrase information includes at least one of the following: keyword information and connection symbol information; the keyword information includes the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category; the connection symbol information includes the connection symbol category corresponding to at least one keyword category in the keyword group; and acquire at least one target question expression from a preset question expression set based on the phrase information; the question expression set includes multiple question expressions.

[0169] In some embodiments, the second acquisition module 5552 is further configured to acquire the keyword category of each keyword in the keyword group; and determine the number of keyword categories and the number of keywords corresponding to each keyword category based on the keyword category of each keyword.

[0170] In some embodiments, the second acquisition module 5552 is further configured to determine the connection symbol category corresponding to the subject category based on the attribute information of the keywords corresponding to the subject category when the number of keywords corresponding to the subject category is at least two; the connection symbol category corresponding to the subject category includes at least one of the following: parallel category and modifier category; when the number of keywords corresponding to the relation category is at least two, determine the connection symbol category corresponding to the relation category based on the attribute information of the keywords corresponding to the relation category; the connection symbol category corresponding to the relation category includes at least one of the following: parallel category and modifier category; when the number of keywords corresponding to the object category is at least one, determine the connection symbol category corresponding to the object category based on the attribute information of the keywords corresponding to the object category; the connection symbol category corresponding to the object category includes at least one of the following: single element category, set element category, and domain category.

[0171] In some embodiments, the second acquisition module 5552 is further configured to determine the expression information corresponding to each question expression among a plurality of question expressions when the phrase information includes keyword information; the expression information includes the number of keyword categories in the question expression and the number of keywords corresponding to each keyword category; and to match the expression information corresponding to each question expression according to the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category to obtain at least one target question expression.

[0172] In some embodiments, the second acquisition module 5552 is further configured to determine the expression information corresponding to each question expression among a plurality of question expressions when the phrase information includes keyword information and connection symbol information; the expression information includes the number of keyword categories in the question expression, the number of keywords corresponding to each keyword category, and the connection symbol category corresponding to at least one keyword category; and to perform matching in the expression information corresponding to each question expression according to the number of keyword categories in the keyword group, the number of keywords corresponding to each keyword category, and the connection symbol category corresponding to at least one keyword category, so as to obtain at least one target question expression.

[0173] In some embodiments, the question corpus generation apparatus 555 further includes an annotation module 5554, which is used to add annotations to each question corpus; the annotations include at least one of the following: keyword groups corresponding to the question corpus, target question expressions corresponding to the question corpus, and keyword tags corresponding to each keyword in the question corpus; in the process of generating a corpus test set based on the annotated question corpus, test corpus with the same annotations are removed; the corpus test set includes multiple annotated test corpus.

[0174] In some embodiments, the question corpus generation apparatus 555 further includes a testing module 5555, which is used to test the question answering system based on multiple annotated test corpora in the corpus test set; obtain at least one target test corpus with abnormal test results; and determine the optimization direction of the question answering system based on the annotations corresponding to at least one target test corpus.

[0175] In some embodiments, the keyword tags include a first keyword tag corresponding to a keyword belonging to the subject category, a second keyword tag corresponding to a keyword belonging to the relation category, and a third keyword tag corresponding to a keyword belonging to the object category; the first keyword tag includes at least one of the following: a polysemous tag, an alias tag, a misspelling tag, and a missing word tag; the second keyword tag includes at least one of the following: a polysemous tag, an alias tag, and an implicit tag; the third keyword tag includes at least one of the following: a length tag, an amount tag, a time tag, a temperature tag, a volume tag, and a character tag.

[0176] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the question corpus generation method provided in this application. For example, ... Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E or Figure 3F The method shown.

[0177] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0178] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0179] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0180] As an example, executable instructions can be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network.

[0181] In summary, the following technical effects can be achieved through the embodiments of this application:

[0182] (1) By obtaining the keyword group and at least one target question expression corresponding to the keyword group; generating at least one question corpus corresponding to the keyword group and at least one target question expression, multiple question corpus corresponding to the keyword group can be automatically generated, which improves the generation efficiency of question corpus. Furthermore, since different questioning methods are represented by the target question expression, the questioning methods of the question corpus generated by different target question expressions have a wider range of questioning coverage.

[0183] (2) By obtaining at least one of the word group information, including keyword information and connection symbol information, and by using the word group information to find the target question expression corresponding to the keyword group, a more complete and wider range of question expressions can be obtained, thereby improving the question range of the question corpus; at the same time, the target question expression can be better matched with the keyword group, and the generated question corpus can be more realistic.

[0184] (3) By annotating each question corpus, question corpus with the same annotation can be deleted when generating the corpus evaluation set, thereby enabling the obtained corpus evaluation set to cover a wider range of question types with the fewest number of question corpus. This improves the corpus test set's ability to test question-answering systems.

[0185] (4) The corpus test set provided in this application can identify logical vulnerabilities of the question-answering system in three dimensions: keyword groups, target question expressions, and keyword tags during the testing process, thereby improving the testing efficiency of the corpus test set.

[0186] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for generating question corpus for evaluating intelligent question-answering systems, characterized in that, include: The server receives a command from the terminal to generate a question corpus sent over the network, and in response to the command, the server obtains a keyword group. The keyword group includes at least two keywords; wherein, the keyword categories of the keywords include: subject category, relation category, and object category; The acquisition of keyword groups includes: selecting any keyword group that needs to be converted into question corpus from a database storing multiple keyword groups; or selecting any knowledge that needs to be converted into question corpus from a knowledge graph and using the triple information contained in the knowledge as keyword groups; or parsing an existing question corpus, extracting keywords from it, and forming corresponding keyword groups based on the grammatical attributes of each keyword. The server obtains the phrase information of the keyword group; wherein, the phrase information includes: keyword information and connection symbol information; the keyword information includes the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category; the connection symbol information includes the connection symbol category corresponding to at least one keyword category in the keyword group; Determine the expression information corresponding to each question expression in the preset question expression set; the expression information includes the number of keyword categories in the question expression, the number of keywords corresponding to each keyword category, and the connection symbol category corresponding to at least one keyword category; Matching is performed on the expression information corresponding to each question expression based on the number of keyword categories in the keyword group, the number of keywords corresponding to each keyword category, and the category of the connecting symbol corresponding to at least one keyword category, to obtain at least one target question expression; wherein, the target question expression is used to connect the at least two keywords according to at least one connecting symbol; Generate at least one corresponding question corpus based on the keyword group and the at least one target question expression; The server sends the generated question corpus to the terminal.

2. The method according to claim 1, characterized in that, The set of question expressions includes multiple question expressions.

3. The method according to claim 2, characterized in that, The step of obtaining the phrase information of the keyword group includes: Obtain the keyword category for each keyword in the keyword group; The number of keyword categories and the number of keywords corresponding to each keyword category are determined based on the keyword category of each keyword.

4. The method according to claim 3, characterized in that, The step of obtaining the phrase information of the keyword group includes: When there are at least two keywords corresponding to the subject category, the connection symbol category corresponding to the subject category is determined based on the attribute information of the keywords corresponding to the subject category; the connection symbol category corresponding to the subject category includes at least one of the following: parallel category and modifying category; When the number of keywords corresponding to the relation category is at least two, the connection symbol category corresponding to the relation category is determined based on the attribute information of the keywords corresponding to the relation category; the connection symbol category corresponding to the relation category includes at least one of the following: parallel category and modifier category; When the number of keywords corresponding to the object category is at least one, the connection symbol category corresponding to the object category is determined according to the attribute information of the keywords corresponding to the object category; the connection symbol category corresponding to the object category includes at least one of the following: single element category, set element category, and domain category.

5. The method according to any one of claims 2 to 4, characterized in that, Obtaining the at least one target question expression includes: Determine the expression information corresponding to each of the plurality of question expressions; the expression information includes the number of keyword categories in the question expression and the number of keywords corresponding to each keyword category; Matching is performed on the expression information corresponding to each question expression based on the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category, so as to obtain the at least one target question expression.

6. A question corpus generation device for evaluating intelligent question-answering systems, wherein the device is used as a server, characterized in that, The server receives a question corpus generation instruction sent by the terminal via the network. In response to the instruction, the server acquires keyword groups. Acquiring keyword groups includes: selecting any keyword group to be converted into question corpus from a database storing multiple keyword groups; or selecting any knowledge from a knowledge graph to be converted into question corpus, and using the triplet information contained in the knowledge as keyword groups; or parsing an existing question corpus, extracting keywords, and forming corresponding keyword groups based on the grammatical attributes of each keyword. The server generates question corpus and sends the generated question corpus to the terminal. The device includes: The first acquisition module is used to acquire keyword groups, wherein the keyword groups include at least two keywords; wherein the keyword categories of the keywords include: subject category, relation category, and object category; The second acquisition module is used to acquire the phrase information of the keyword group; wherein, the phrase information includes: keyword information and connection symbol information; the keyword information includes the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category; the connection symbol information includes the connection symbol category corresponding to at least one keyword category in the keyword group; The second acquisition module is further configured to determine the expression information corresponding to each question expression in the preset question expression set; the expression information includes the number of keyword categories in the question expression, the number of keywords corresponding to each keyword category, and the connection symbol category corresponding to at least one keyword category; The second acquisition module is further configured to match the expression information corresponding to each question expression according to the number of keyword categories in the keyword group, the number of keywords corresponding to each keyword category, and the connection symbol category corresponding to at least one keyword category, so as to obtain at least one target question expression; wherein, the target question expression is used to connect the at least two keywords according to at least one connection symbol; The generation module is used to generate at least one corresponding question corpus based on the keyword group and the at least one target question expression.

7. The apparatus according to claim 6, characterized in that, The set of question expressions includes multiple question expressions.

8. The apparatus according to claim 7, characterized in that, The second acquisition module is further configured to acquire the keyword category of each keyword in the keyword group; The number of keyword categories and the number of keywords corresponding to each keyword category are determined based on the keyword category of each keyword.

9. The apparatus according to claim 8, characterized in that, The second acquisition module is further configured to determine the connection symbol category corresponding to the subject category based on the attribute information of the keywords corresponding to the subject category when the number of keywords corresponding to the subject category is at least two; the connection symbol category corresponding to the subject category includes at least one of the following: parallel category and modifier category; When the number of keywords corresponding to the relation category is at least two, the connection symbol category corresponding to the relation category is determined based on the attribute information of the keywords corresponding to the relation category; The connection symbol category corresponding to the relation category includes at least one of the following: parallel category and modifier category; When the number of keywords corresponding to the object category is at least one, the connection symbol category corresponding to the object category is determined according to the attribute information of the keywords corresponding to the object category; the connection symbol category corresponding to the object category includes at least one of the following: single element category, set element category, and domain category.

10. The apparatus according to any one of claims 7 to 9, characterized in that, The second acquisition module is further configured to determine the expression information corresponding to each of the plurality of question expressions; the expression information includes the number of keyword categories in the question expression and the number of keywords corresponding to each keyword category; Matching is performed on the expression information corresponding to each question expression based on the number of keyword categories in the keyword group and the number of keywords corresponding to each keyword category, so as to obtain the at least one target question expression.

11. A question corpus generation device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, It stores executable instructions for implementing the method of any one of claims 1 to 5 when executed by a processor.

Images